Every request that verified AI and search bots made to 14 small-business websites we host, over 30 days (October 20, 2026 to November 18, 2026): who came, what they read, and who asked for llms.txt.
Meta's bots made 41.5% of all 412,610 requests, more than any other company. Its link-preview fetcher (facebookexternalhit) alone made 80,680.
OpenAI's search crawler, OAI-SearchBot, reached 14 of the 14 websites with 22,841 requests, about 3 times as many as GPTBot (7,202), its training crawler.
No AI company's bot asked for llms.txt or llms-full.txt. All 161 requests for those files came from SEO and speed tools: semrush-site-audit (140), chrome-lighthouse (20), ahrefsbot (1).
Fetches made because a person asked an AI assistant to open a page (user-triggered fetches) were rare: 1,028 of 412,610 requests.
The websites
14 websites that ROEM Digital builds and hosts, one per industry. One of them, the marketing agency, is our own. They are shown only by industry, never by name or address.
Websites in the data, by industry
Industry
Bot requests
Data from
Medical aesthetics (in-home)
99,900
Oct 20
Day spa and esthetics
66,856
Oct 20
Painting contractor
60,647
Oct 20
Private elementary school
53,290
Oct 20
Church
39,606
Oct 20
Marketing agency
32,958
Oct 20
Auction company
30,646
Nov 11
Musician
12,543
Oct 20
IT services
5,269
Oct 20
Booking software
4,988
Oct 20
Church youth event
2,731
Nov 11
Life coaching
1,872
Oct 20
Local business directory
663
Oct 20
Education software
641
Oct 23
3 websites have data for part of the window only, from the date shown, because they moved to Vercel or launched during it.
Requests by company
Requests by bot operator, Oct 20 to Nov 18
Company
Requests
Share
Meta
171,177
41.5%
Other companies
98,479
23.9%
Google
87,341
21.2%
OpenAI
30,668
7.4%
Microsoft
7,743
1.9%
Amazon
7,299
1.8%
Anthropic
4,826
1.2%
Perplexity
2,347
0.6%
Apple
1,779
0.4%
DuckDuckGo
734
0.2%
Common Crawl
217
0.1%
Requests by bot
Bots with 100 or more requests
Bot
Company
What it is for
Requests
Websites visited
facebookexternalhit
Meta
other
80,680
14 of 14
semrush-site-audit
Other
other
61,282
12 of 14
meta-webindexer
Meta
search index
44,713
11 of 14
googlebot
Google
search engine
38,827
14 of 14
meta-externalagent
Meta
model training
36,473
13 of 14
oai-searchbot
OpenAI
search index
22,841
14 of 14
googleother
Google
other
19,098
12 of 14
google-adsbot
Google
other
13,335
8 of 14
meta-externalads
Meta
other
9,311
7 of 14
ahrefssiteaudit
Other
other
8,840
5 of 14
ahrefsbot
Other
other
8,554
14 of 14
google-inspectiontool
Google
other
7,919
4 of 14
bingbot
Microsoft
search engine
7,683
13 of 14
amazonbot
Amazon
model training
7,297
14 of 14
gptbot
OpenAI
model training
7,202
14 of 14
claudebot
Anthropic
model training
4,690
11 of 14
exasearchbot
Other
search index
4,213
12 of 14
petalbot
Other
search engine
2,940
10 of 14
chrome-lighthouse
Google
other
2,578
4 of 14
semrush
Other
other
2,353
13 of 14
perplexitybot
Perplexity
search index
2,347
11 of 14
shapbot
Other
search index
2,280
10 of 14
adsense
Google
other
2,072
3 of 14
applebot
Apple
search engine
1,779
14 of 14
google-storebot
Google
other
1,772
6 of 14
baiduspider
Other
search engine
1,536
8 of 14
seranking-backlinks
Other
other
1,353
12 of 14
barkrowler
Other
other
1,240
10 of 14
stripebot
Other
other
1,068
2 of 14
sansec-security-monitor
Other
other
1,054
2 of 14
yandexbot
Other
search engine
846
7 of 14
chrome-privacy-preserving-prefetch-proxy
Google
other
639
12 of 14
google-businesslink-verification
Google
other
546
1 of 14
duckduckbot
DuckDuckGo
search engine
538
12 of 14
gpt-actions
OpenAI
user-triggered fetch
412
10 of 14
seznambot
Other
search engine
380
10 of 14
google-adwords-instant
Google
other
261
5 of 14
ccbot
Common Crawl
other
217
4 of 14
duckassistbot
DuckDuckGo
search index
196
9 of 14
google-agent
Google
user-triggered fetch
177
1 of 14
chatgpt-user
OpenAI
user-triggered fetch
161
8 of 14
naverbot
Other
search engine
158
1 of 14
pinterest-bot
Other
other
158
4 of 14
claude-user
Anthropic
user-triggered fetch
136
6 of 14
18 more bots with fewer than 100 requests each are in the CSV. SEO and site-audit tools (for example semrush-site-audit, ahrefssiteaudit, chrome-lighthouse and google-inspectiontool) are in the data too, and part of their traffic is our own audits of these websites.
What they asked for
Requests by kind of page, all bots
Kind of page
Requests
Share
Other pages
130,986
31.7%
Images
104,310
25.3%
Scripts, styles, fonts
71,354
17.3%
Homepage
70,804
17.2%
Blog posts
14,554
3.5%
robots.txt
14,038
3.4%
Sitemaps
3,506
0.8%
Blog index
2,876
0.7%
llms.txt or llms-full.txt
161
0.0%
Feeds
21
0.0%
Who asked for llms.txt
Requests for llms.txt or llms-full.txt, by bot
Bot
Requests
semrush-site-audit
140
chrome-lighthouse
20
ahrefsbot
1
Columns in the CSV
CSV columns
Column
Meaning
date
Day, UTC (YYYY-MM-DD).
site_id
An anonymous website number (site-01 to site-14). The same number is the same website on every row.
industry
What the business does. Websites are named only by industry, never by name or address.
bot_name
The bot as Vercel names it after verifying it (lowercase, as Vercel writes it).
bot_operator
The company that runs the bot.
bot_purpose
What the operator says the bot is for: search index, model training, user-triggered fetch, search engine, or other.
page_type
What was fetched, grouped by fixed rules: homepage, blog post, blog index, robots.txt, sitemap, llms.txt, images, scripts and styles, feeds, other pages.
requests
Requests that day from that bot to that kind of page on that website.
How it was measured, and its limits
Source: Vercel's request metrics for each website's production custom domains, counted per day, bot and path. Vercel's own preview and .vercel.app addresses are left out.
Verified bots only. Vercel gives a request a bot name only after verifying the bot, for example by its operator's published IP ranges or reverse DNS. A bot that only claims a name in its user agent, such as ByteDance's Bytespider, gets no name, so it is not in the data. Neither are people.
Every request the CDN answered counts, including cached pages and redirects (for example from the bare domain to www), so one page fetch can count twice.
Paths are grouped into kinds of page by fixed rules (anything under /blog/ is a blog post, and so on), then dropped. No URLs are published.
On the booking platform, only its public marketing pages count, not the signed-in app.
Each bot's purpose comes from its operator's published crawler documentation, or from Vercel's verified bot directory where the operator publishes none.
Use and credit
Free to use under CC BY 4.0. Credit "ROEM Digital, AI crawler visits to small-business websites, version 1.0" with a link to this page. Questions about the data: contact us.