6 October 2026 · Fedor Dyatlov
I had both of those backwards, and I shipped it with a docstring claiming compliance.
Here is the rule, from RFC 9309 — the 2022 standard that finally wrote down what robots.txt means:
So a server answering 403 Forbidden on /robots.txt
is telling you to go ahead, and a server answering
500 Internal Server Error is telling you to stop. If that feels
inverted, it did to me too, and my code followed my intuition rather than the
text.
Intuition says forbidden means forbidden. You asked for a file, the server said no, clearly this place does not want you.
But robots.txt is not a resource you need permission to read. It is a statement of rules, and a 404 or a 403 means there are no rules here. A site that wants to refuse you says so inside the file. Its absence is not a refusal; it is silence, and silence is permission in this protocol. That is the whole point of a file you have to go looking for.
The 5xx case has the real reasoning behind it. A server returning 500 is a server in trouble. It may well have rules — rules that say no — and it simply cannot tell you right now. Assuming "no rules" at the exact moment a site is falling over means a crawler piles on precisely when it hurts most. So the spec picks the cautious reading: a server that cannot answer has not given you permission.
"There is no sign on the door" is not the same as "I can't reach the door."
Not by reading the spec. By a storage bucket.
Object stores and CDNs frequently answer 403 for
/robots.txt — not because anyone decided anything, but because the
file was never uploaded and the bucket policy denies unknown keys instead of
returning a 404. My crawler read that 403 as "this entire origin is off
limits" and refused to fetch the images it served.
Nobody had forbidden anything. A file that did not exist, served by a storage system with an opinion about missing keys, stopped a job fetching images that nobody had protected. I went to the RFC expecting to confirm my reading and found both status-code rules exactly inverted.
The docstring above the function said it followed RFC 9309. I wrote that docstring. It had been wrong since the day I wrote it, and no test caught it, because every test I had written asked the question the way I already believed the answer.
"Must assume complete disallow" raises an obvious question: for how long?
Forever is wrong. One stray 503 during a long crawl would lock you out of a site that is fine again a minute later. The RFC allows downgrading an unreachable verdict to unavailable after "a reasonably long period", which is a phrase doing a great deal of work.
I cache the disallow for five minutes and then ask again. Re-asking costs one request; being wrong in either direction costs much more. That is a judgement call rather than a rule, and it is worth making deliberately — which, until I looked, is how I had made none of the other decisions in that file.
I failed the first two.
page.goto. Rendering is
still fetching. If the rules live in your HTTP helper, everything that does
not use that helper is outside them."We obey robots.txt" is the sort of thing that goes in a README and then quietly isn't true, because nobody tests the embarrassing paths and the failure mode is invisible: you fetch things you should not have, and nothing anywhere says so.
The correctness here is not legal cover. It is that a crawler is a guest in someone else's house, the house has left instructions, and reading them wrong in the polite direction costs you a few pages — while reading them wrong in the other direction makes you the thing everyone blocks.