Notebook · Ed. 07 · October 15, 2026

None of the Georgia Fortune 500 robots.txt files we could read block an AI crawler.
Three would not let us read the rule.

Sixteen Fortune 500 companies are headquartered in Georgia. On 15 October 2026 we fetched each one’s robots.txt and checked it against seven AI crawlers. Thirteen gave a readable answer (twelve files and one 404), and not one blocks any of the seven. The other three, Home Depot, UPS and Asbury Automotive, answered with a firewall denial or a bot challenge instead of rules.

Get my free citability check See how the audit works

Finding 01

0 / 13

readable robots.txt answers block any of the seven AI crawlers.

Do Georgia’s Fortune 500 companies block AI crawlers?

On the answers we could read, no. We tested GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended and CCBot, and counted a bot as blocked only when the rule group that applies to it contains Disallow: /. Twelve companies serve a robots.txt that does that for none of the seven, and a thirteenth, Genuine Parts, serves a 404, which means no rules at all. Delta, the largest company we could read, disallows booking, trip, file-type and non-US locale paths, and none of those rules is a site-wide block.

Eleven of the 16 sit in the first bar with no rule aimed at an AI crawler. The last bar is the three we could not read. No company blocked an AI crawler in anything we could see.

Read the zero as a count of robots.txt rules. A robots.txt file is a request that crawlers choose to honor, and a firewall can still refuse them. The next finding covers the three where we could not see the file at all.

Finding 02

3 / 16

returned a denial or challenge instead of a robots.txt.

Why could we not read three of the sixteen?

Their servers refused our automated request for the file. Home Depot returned HTTP 403 with an “Access Denied” page to both of our user agents. UPS returned HTTP 200, but the body was an “Access Denied” page, not rules. Asbury Automotive returned HTTP 429 with a Vercel bot-challenge page. We did not try to get around any of them, so for these three we publish no verdict on which crawlers they allow.

Two more companies only served the file on the second try. Delta returned HTTP 444 with an Access Denied page to a plain curl request and served its rules to a desktop Safari user-agent string. PulteGroup returned a 403 to the plain request and its rules to Safari. So five of the sixteen refused or challenged a plain curl request. Two served the file once we sent a Safari user-agent string, and three refused that as well. We did not test a real browser.

What that means for an AI crawler is not settled by this data. Under RFC 9309, the robots.txt standard, a 4xx response means the file is unavailable and a crawler may access any resource, while a 5xx means the crawler must assume everything is disallowed. None of our three returned a 5xx. Home Depot’s 403 and Asbury’s 429 are 4xx responses, and UPS’s 200 carries an HTML denial page with no rules in it, so at the robots.txt layer a standards-following crawler would treat all three as unrestricted. That is the rule for a polite crawler reading a file. It says nothing about whether the same firewall that refused our script would pass GPTBot or ClaudeBot on a page request. We did not test that, and it is the more important question.

“Of 16 Georgia Fortune 500 headquarters fetched on 15 Oct 2026, 13 gave a readable robots.txt answer (12 files, one 404) and none blocked any of seven AI crawlers. Three returned a firewall denial or challenge instead.”

Finding 03

One company
names the bots.

Does any company write rules for specific AI crawlers?

One of the 13: Norfolk Southern. Its file opens with the comment “allow all verified crawlers” and then gives GPTBot, PerplexityBot and Claude-Web their own groups, each with Allow: /. Claude-Web is not ClaudeBot, the token this census tests, so ClaudeBot falls through to the wildcard group, which also allows everything. The other twelve readable files have no group for any AI crawler at all.

That makes this a different result from the East Cobb med spa index, where one business blocked the training crawlers and allowed the answer-time ones. Among Georgia’s Fortune 500 headquarters, none of the 13 readable files blocks either kind, so the split between training bots and answer-time bots, which OpenAI documents as separate settings (OAI-SearchBot for ChatGPT search, GPTBot for training), has nothing to separate here. Whether the three firewalled sites would let the answer-time bots through is the open question, and the reason the next edition fetches pages as well as the rule file.

The Census

All 16,
gaps included.

Ordered by Fortune 500 rank. Each row links to the live robots.txt, which may have changed since we fetched it. “Unreadable” means we do not know what the file says, not that the company blocks anything.

RANK	COMPANY	HTTP	RESULT	DETAIL
25	The Home Depotwww.homedepot.com	403	Unreadable	Access Denied page to both user agents
48	United Parcel Servicewww.ups.com	200	Unreadable	HTTP 200, but the body is an Access Denied page, not rules
71	Delta Air Lineswww.delta.com	200	Readable, nothing blocked	Only a * group, many path disallows; 444 Access Denied to the default agent; read with a Safari user agent
98	The Coca-Cola Companywww.coca-colacompany.com	200	Readable, nothing blocked	Only a * group, no disallow rules
152	Southern Companywww.southerncompany.com	200	Readable, nothing blocked	Only a * group, two path disallows
188	Genuine Partswww.genpt.com	404	No robots.txt, nothing blocked	No robots.txt, so no rules
246	Asbury Automotive Groupwww.asburyauto.com	429	Unreadable	Bot challenge page to both user agents
256	PulteGroupwww.pultegroup.com	200	Readable, nothing blocked	A * group plus three non-AI crawler groups, all Allow: /; 403 to the default agent; read with a Safari user agent
261	Aflacwww.aflac.com	200	Readable, nothing blocked	Only a * group, path disallows
345	Assurantwww.assurant.com	200	Readable, nothing blocked	Only a * group, Allow: /
347	Intercontinental Exchangewww.ice.com	200	Readable, nothing blocked	Only a * group, path disallows
359	Norfolk Southernwww.norfolksouthern.com	200	Readable, nothing blocked	Allows GPTBot, PerplexityBot and Claude-Web by name
404	Mohawk Industriesmohawkind.com	200	Readable, nothing blocked	Only a * group, empty Disallow
415	Global Paymentswww.globalpayments.com	200	Readable, nothing blocked	Only a * group, path disallows
417	AGCOwww.agcocorp.com	200	Readable, nothing blocked	Only a * group, one path disallow
460	Graphic Packaging Holdingwww.graphicpkg.com	200	Readable, nothing blocked	Only a * group, Allow: /

Fetched 15 Oct 2026, about 13:23 to 13:25 UTC. Rank is the June 2026 Fortune 500 as compiled by Select Georgia. Bots tested: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended, CCBot.

What it means

The readable files are open.
The firewall is the question.

If robots.txt is open, what should a company check next?

The check is whether the bots can actually fetch a page. A robots.txt that allows everything does nothing if a firewall answers the crawler with a 403 or a challenge, and from the outside you cannot tell the two apart without testing both. That is the same lesson as in the llms.txt note: the cheap file at the root is the visible setting, and the thing that decides the outcome is usually somewhere less visible.

Our AI crawler configuration guide lists which bots to allow and why, and the AI Citation Index shows what the engines then do with the sites that let them in. If you want that check run against your own site, including the firewall layer this census could not see, the audit process is public.

Method

How we
measured this.

Which companies counted as Georgia Fortune 500?

The 16 companies headquartered in Georgia on the June 2026 Fortune 500, as compiled in Select Georgia’s Fortune 500 list (Georgia Power Economic Development, August 2026). That compilation counts 16 Georgia headquarters, 14 in metro Atlanta plus Aflac and Mohawk Industries; older state pages cite a different count from an earlier list. We took the list from it because Fortune’s own page timed out in our fetch tool on 15 October, so a company the compilation missed would be missing here too.

What exactly was fetched, and how was it scored?

For each company we requested https://<primary host>/robots.txt on 15 October 2026, first with curl’s default user agent and, if that was not an HTTP 200 or 404, again with a desktop Safari user agent. A bot counted as blocked only if the group matching its token, else the * group, contained Disallow: /exactly. A 404 counted as no rules. An HTML body or any other status counted as unreadable. The host is the main web address a visitor reaches for the company name. Other hosts can differ: Home Depot’s corporate subdomain, corporate.homedepot.com, served a readable file with no site-wide block when we checked it on the same day, while its main host did not. We scored the main host only.

The limits are these. One day, one network, up to three attempts per host. A robots.txt can change tomorrow, and a firewall can treat a real AI crawler differently from our request, which we did not test. Three companies are unscored, so the headline is 0 of 13. A company can send a correction and we will log it on this page. The raw files and the scoring script are kept so the census can be rerun quarterly.

Can the bots actually reach your site?

Free citability check.
24-hour turnaround.

Get my free check See a full audit

Filed by Jake Pereira · Founder, Citably · October 15, 2026 · Atlanta GA · ET

None of the Georgia Fortune 500 robots.txt files we could read block an AI crawler.Three would not let us read the rule.