| Crawler | Whose | State |
|---|
In short
- What gets read
- Your robots.txt at root level, plus whether an llms.txt sits beside it. Nothing else is fetched.
- The three outcomes
- Explicitly allowed, not blocked, or blocked. That middle one is not a half state but the normal one.
- What it does not say
- Whether you get named. Access is a condition, not an outcome — that is a different measurement.
What do the three outcomes mean?
Explicitly allowed means your robots.txt names that crawler and lets it in. Not blocked means it is named nowhere and may therefore come in — that is the normal state of a site that never configured anything, and there is nothing wrong with it.
Blocked by a catch-all rule is the outcome that most often comes as a surprise: there is no rule about that crawler, but there is one shutting out everything not named. Read without context, it looks like a deliberate choice.
How does a site accidentally end up closed?
Almost always by one of three routes: a security extension blocking everything it does not recognise, a robots.txt closed during a rebuild and never reopened, or a hosting setting nobody ever looked at.
All three are fixable in minutes and all three cost months if nobody looks. That is why this check is free: it finds the cheapest problem this craft has.
What does this check not say about my visibility?
Everything that comes after access. A site standing wide open and appearing in no answer at all exists — and then the cause lies outside your site, not in this file.
| What this tool does | What it does not do |
|---|---|
| Reads your robots.txt the way a crawler does | Looks only at root level — rules deeper in your site are not included |
| Says whether an llms.txt exists on your domain | Does not judge the contents of that file |
| Shows the current state per crawler | Does not measure whether that crawler actually visits, let alone whether you are named |
Frequently asked questions
I have no robots.txt. Is that bad?
No. No file means everything may come in, and for visibility that is the most favourable state. Creating one only makes sense if you want to exclude something or to record explicitly what is allowed — the latter is tidier and changes practically nothing.
Why do I have to enter an email address here?
Because this check runs on our server and not in your browser: an address is fetched, and that costs something. The email ties the outcome to you, and it is the only tool here where it is asked because something happens outside your browser. The robots generator and the self-test ask for nothing, because there is nothing to fetch.
Read on
Free, no account and no card.