AI Search

Most Irish websites have no AI crawler policy at all

We read the robots.txt of 107 of Ireland's best-known websites to see what they tell AI crawlers. Of the 92 that gave us a file, 71 say nothing about AI at all.

Ten block the crawlers that feed AI answers. Every single one of those ten is a news publisher. Not one Irish shop, bank, utility, university or airline in the sample has taken a position either way.

This follows on from the sitemap audit. Same idea: check the boring file nobody opens twice.

Method

We started with 113 domains across state bodies, transport, media, retail, banking, utilities, tourism, universities, property and Irish tech. For each one we fetched https://domain/robots.txt, parsed it into user-agent groups, and worked out whether each AI crawler is blocked from the whole site, either by name or by inheriting a User-agent: * rule.

Before any of that we fetched every homepage and read its <title>, which is a step we nearly skipped. It caught six domains that are not who we assumed. eason.com is a domain registrar, not the bookshop (that's easons.com). uisce.ie is an Irish-language college in Mayo, not Uisce Éireann (that's water.ie). University of Galway has moved off nuigalway.ie. We had also listed four domains that no longer resolve or now redirect into gov.ie. Correcting those is why the sample is 107 and not 113.

We ran the survey twice, on 5 October 2026, from Dublin. Once with a crawler user agent that identifies itself as ours, and once pretending to be Chrome. That second run turned out to matter.

We ran it with a small script we keep off this website. It's the same one we use when we run an audit.

Which crawlers we checked, and why the names matter

Most of the confusion here comes from treating "AI bots" as one thing. They are two things, and mixing them up is how a site ends up invisible.

Training crawlers collect text to train models. Blocking them is a licensing decision and it has no effect on whether you appear in search:

Answer crawlers build the indexes behind AI search, or fetch a page live when somebody asks a question. Blocking these removes you from the answers:

One of these gets repeated wrongly a lot, so: blocking Google-Extended does not remove you from Google and it does not remove you from AI Overviews. It stops your content being used to improve Gemini and Vertex AI, and nothing else. AI Overviews are built from the ordinary Google index, so Googlebot is what decides that.

Finding 1: almost nobody has a policy

Of the 92 sites that served us a robots.txt, 21 name any AI crawler. The other 71 have never touched the subject.

That includes nearly every retailer, all the utilities, most of the universities, the property portals and the airlines. The default is "everything allowed", so those 71 are training every model and appearing in every answer engine. For a lot of businesses that is the right outcome. It should still be a decision rather than an accident.

Of the 21 that do have rules, GPTBot is the most blocked at 12 sites. Then ClaudeBot at 8, with CCBot, Google-Extended and meta-externalagent at 7 each. Only 15 block a training crawler at all. The other 6 name one and then allow it.

Finding 2: 19 sites would not show us the file

This is the part we didn't expect. Nineteen of the 107 gave us nothing on the crawler run. Running it again as Chrome split them into three groups.

Four serve robots.txt to a browser and return 403 to a crawler: gov.ie, education.ie, ucd.ie and aviva.ie.

gov.ie is the one worth looking at. It has a careful AI policy, blocking GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, Meta-ExternalAgent and Amazonbot, with a comment above each group explaining who it belongs to. Somebody put real thought into it. Then the server was configured to refuse crawlers, so the only things that can read the rules are the things not bound by them.

Eight refused both identities: Tesco, AIB, PTSB, Currys, Harvey Norman, Very, Dublin Bus and the CRO. That is bot protection in front of everything, which is defensible for a bank. It does mean no crawler can read a policy either.

Seven have no robots.txt at all: Citizens Information, the CSO, Irish Ferries, Vodafone, the National Museum, University of Galway and the CAO. A missing file is not a disaster. It means no rules and no sitemap pointer.

Two went the other way and refused Chrome while serving our crawler: Aldi and IrishJobs. A bot wall that lets a self-declared bot through and turns a browser away is not protecting anything in particular.

Finding 3: the publishers are deliberate, and the rest are absent

All ten sites blocking answer crawlers are news publishers. Independent.ie blocks all five, by name, in a long hand-maintained list. The Business Post blocks four. For newspapers negotiating content licensing that is a strategy, and we're not going to call it a mistake.

The smaller case is more interesting. RTÉ, the Irish Times, TheJournal.ie and the Irish Examiner all block GPTBot and ChatGPT-User while leaving OAI-SearchBot alone.

Read that carefully. They stay in ChatGPT's search index, so ChatGPT can cite them. But when a reader pastes one of their article links into ChatGPT and asks what it says, the fetch is refused. The citation works and the reader doesn't.

We don't know whether that asymmetry is intended. It is the kind of thing that happens when a blocklist gets copied without checking what each name does.

Finding 4: the most careful site still got one wrong

BreakingNews.ie has the best robots.txt in the sample. It blocks all seven training crawlers and documents the reasoning in a comment:

# ChatGPT is allowed: OAI-SearchBot (search citations) and ChatGPT-User
# (live user fetches) fall under the wildcard group above and are permitted.

breakingnews.ie/robots.txt

Somebody there understood the distinction and wrote it down for whoever edits the file next. Then they put PerplexityBot in the training block list, next to GPTBot and CCBot. PerplexityBot is the index crawler, so the site is deliberately citable in ChatGPT and accidentally absent from Perplexity.

If the most careful site out of 107 gets a crawler name wrong, the names are too confusing. That's on the AI companies, not on the publishers.

What this data cannot tell you

robots.txt is a request, not a wall. Well-behaved crawlers obey it and badly-behaved ones ignore it, so a blocked bot here has not necessarily stopped visiting. Blocking at the CDN is the version with teeth, and we can't see that from outside.

We checked whole-site blocks only. A site that disallows /products/ for GPTBot and allows everything else shows up as allowed, which is correct but blunt.

It's also a single point in time, and publishers edit these files often. And none of it measures whether a site actually appears in AI answers. Access is necessary, it isn't sufficient.

What to do about your own site

Open yoursite.ie/robots.txt and read it. Then:

  1. Decide the training question on its own. "No" is fine and costs you nothing in search.
  2. Leave OAI-SearchBot, PerplexityBot, Claude-SearchBot and the two -User agents alone, unless you want to be absent from AI answers.
  3. Check the file is readable by something that isn't a browser. If your host or CDN returns 403 to crawlers, your rules are decoration.
  4. If a plugin, a host or a CDN toggle wrote your AI rules, read what it actually blocked. The managed lists are aggressive and they update without telling you.
  5. Confirm the file names your sitemap.

Mine is at wpdublin.com/robots.txt if you want to see the shape of it. We've gone a different way and declared the policy once, with a Content-Signal line that says ai-train=no, search=yes, ai-input=yes, instead of maintaining a list of bot names. Given that the names are the part everyone gets wrong, we'd rather state the terms and let the crawlers read them. We publish an llms.txt as well.

That is a weaker mechanism than a named Disallow, and we know it. It's a stated preference rather than a block, so it depends on crawlers choosing to honour it. The trade-off is that it doesn't go stale every time somebody ships a new agent.

Two years ago this file only decided whether Google could see you. Now it decides whether you exist in the thing a lot of people ask instead of Google. Most Irish websites haven't looked at it since.

WordPress PluginsWooCommerce Plugins WordPress & WooCommerce Plugins Partner 🎯 SEO & PPC Partner