Skip to content
Free data · CSV

AI crawler visits to small-business websites

Every request that verified AI and search bots made to 14 small-business websites we host, over 30 days (October 20, 2026 to November 18, 2026): who came, what they read, and who asked for llms.txt.

Download the CSV (1,251 KB)

What it shows

  • Meta's bots made 41.5% of all 412,610 requests, more than any other company. Its link-preview fetcher (facebookexternalhit) alone made 80,680.
  • OpenAI's search crawler, OAI-SearchBot, reached 14 of the 14 websites with 22,841 requests, about 3 times as many as GPTBot (7,202), its training crawler.
  • No AI company's bot asked for llms.txt or llms-full.txt. All 161 requests for those files came from SEO and speed tools: semrush-site-audit (140), chrome-lighthouse (20), ahrefsbot (1).
  • Fetches made because a person asked an AI assistant to open a page (user-triggered fetches) were rare: 1,028 of 412,610 requests.

The websites

14 websites that ROEM Digital builds and hosts, one per industry. One of them, the marketing agency, is our own. They are shown only by industry, never by name or address.

Websites in the data, by industry
IndustryBot requestsData from
Medical aesthetics (in-home)99,900Oct 20
Day spa and esthetics66,856Oct 20
Painting contractor60,647Oct 20
Private elementary school53,290Oct 20
Church39,606Oct 20
Marketing agency32,958Oct 20
Auction company30,646Nov 11
Musician12,543Oct 20
IT services5,269Oct 20
Booking software4,988Oct 20
Church youth event2,731Nov 11
Life coaching1,872Oct 20
Local business directory663Oct 20
Education software641Oct 23

3 websites have data for part of the window only, from the date shown, because they moved to Vercel or launched during it.

Requests by company

Requests by bot operator, Oct 20 to Nov 18
CompanyRequestsShare
Meta171,17741.5%
Other companies98,47923.9%
Google87,34121.2%
OpenAI30,6687.4%
Microsoft7,7431.9%
Amazon7,2991.8%
Anthropic4,8261.2%
Perplexity2,3470.6%
Apple1,7790.4%
DuckDuckGo7340.2%
Common Crawl2170.1%

Requests by bot

Bots with 100 or more requests
BotCompanyWhat it is forRequestsWebsites visited
facebookexternalhitMetaother80,68014 of 14
semrush-site-auditOtherother61,28212 of 14
meta-webindexerMetasearch index44,71311 of 14
googlebotGooglesearch engine38,82714 of 14
meta-externalagentMetamodel training36,47313 of 14
oai-searchbotOpenAIsearch index22,84114 of 14
googleotherGoogleother19,09812 of 14
google-adsbotGoogleother13,3358 of 14
meta-externaladsMetaother9,3117 of 14
ahrefssiteauditOtherother8,8405 of 14
ahrefsbotOtherother8,55414 of 14
google-inspectiontoolGoogleother7,9194 of 14
bingbotMicrosoftsearch engine7,68313 of 14
amazonbotAmazonmodel training7,29714 of 14
gptbotOpenAImodel training7,20214 of 14
claudebotAnthropicmodel training4,69011 of 14
exasearchbotOthersearch index4,21312 of 14
petalbotOthersearch engine2,94010 of 14
chrome-lighthouseGoogleother2,5784 of 14
semrushOtherother2,35313 of 14
perplexitybotPerplexitysearch index2,34711 of 14
shapbotOthersearch index2,28010 of 14
adsenseGoogleother2,0723 of 14
applebotApplesearch engine1,77914 of 14
google-storebotGoogleother1,7726 of 14
baiduspiderOthersearch engine1,5368 of 14
seranking-backlinksOtherother1,35312 of 14
barkrowlerOtherother1,24010 of 14
stripebotOtherother1,0682 of 14
sansec-security-monitorOtherother1,0542 of 14
yandexbotOthersearch engine8467 of 14
chrome-privacy-preserving-prefetch-proxyGoogleother63912 of 14
google-businesslink-verificationGoogleother5461 of 14
duckduckbotDuckDuckGosearch engine53812 of 14
gpt-actionsOpenAIuser-triggered fetch41210 of 14
seznambotOthersearch engine38010 of 14
google-adwords-instantGoogleother2615 of 14
ccbotCommon Crawlother2174 of 14
duckassistbotDuckDuckGosearch index1969 of 14
google-agentGoogleuser-triggered fetch1771 of 14
chatgpt-userOpenAIuser-triggered fetch1618 of 14
naverbotOthersearch engine1581 of 14
pinterest-botOtherother1584 of 14
claude-userAnthropicuser-triggered fetch1366 of 14

18 more bots with fewer than 100 requests each are in the CSV. SEO and site-audit tools (for example semrush-site-audit, ahrefssiteaudit, chrome-lighthouse and google-inspectiontool) are in the data too, and part of their traffic is our own audits of these websites.

What they asked for

Requests by kind of page, all bots
Kind of pageRequestsShare
Other pages130,98631.7%
Images104,31025.3%
Scripts, styles, fonts71,35417.3%
Homepage70,80417.2%
Blog posts14,5543.5%
robots.txt14,0383.4%
Sitemaps3,5060.8%
Blog index2,8760.7%
llms.txt or llms-full.txt1610.0%
Feeds210.0%

Who asked for llms.txt

Requests for llms.txt or llms-full.txt, by bot
BotRequests
semrush-site-audit140
chrome-lighthouse20
ahrefsbot1

Columns in the CSV

CSV columns
ColumnMeaning
dateDay, UTC (YYYY-MM-DD).
site_idAn anonymous website number (site-01 to site-14). The same number is the same website on every row.
industryWhat the business does. Websites are named only by industry, never by name or address.
bot_nameThe bot as Vercel names it after verifying it (lowercase, as Vercel writes it).
bot_operatorThe company that runs the bot.
bot_purposeWhat the operator says the bot is for: search index, model training, user-triggered fetch, search engine, or other.
page_typeWhat was fetched, grouped by fixed rules: homepage, blog post, blog index, robots.txt, sitemap, llms.txt, images, scripts and styles, feeds, other pages.
requestsRequests that day from that bot to that kind of page on that website.

How it was measured, and its limits

  • Source: Vercel's request metrics for each website's production custom domains, counted per day, bot and path. Vercel's own preview and .vercel.app addresses are left out.
  • Verified bots only. Vercel gives a request a bot name only after verifying the bot, for example by its operator's published IP ranges or reverse DNS. A bot that only claims a name in its user agent, such as ByteDance's Bytespider, gets no name, so it is not in the data. Neither are people.
  • Every request the CDN answered counts, including cached pages and redirects (for example from the bare domain to www), so one page fetch can count twice.
  • Paths are grouped into kinds of page by fixed rules (anything under /blog/ is a blog post, and so on), then dropped. No URLs are published.
  • On the booking platform, only its public marketing pages count, not the signed-in app.
  • Each bot's purpose comes from its operator's published crawler documentation, or from Vercel's verified bot directory where the operator publishes none.

Use and credit

Free to use under CC BY 4.0. Credit "ROEM Digital, AI crawler visits to small-business websites, version 1.0" with a link to this page. Questions about the data: contact us.