# SearchGrade data study — "The State of AI Search Readiness"
# Sample: 500 domains. Generated 2026-08-01.
#
# SOURCE
#   Tranco list 5648N          https://tranco-list.eu/list/5648N
#   Generated                  2026-07-31, 30-day average (2026-07-02 .. 2026-07-31)
#   Providers                  crux, farsight, majestic, radar, umbrella
#   Combination                Dowdall; filterPLD on, filterTLD off
#
# ⚠ Tranco ranks by aggregated DNS/resolver and traffic-provider PROMINENCE. It is NOT a
#   ranking of human visits. Do not describe this sample as "the most visited websites" —
#   gtld-servers.net ranks #2, and 469 of the first 1038 entries are not websites at all.
#   Correct phrasing: "500 domains drawn from the Tranco top-1038 (list 5648N, 2026-07-31),
#   after removing non-website domains and collapsing ccTLD mirrors."
#
# SELECTION PROCEDURE
#   1. Take Tranco ranks 1..1038 (1038 domains).
#   2. Classify each by ONE operational test: a domain is a website only if its APEX serves
#      human-readable content a person could plausibly be seeking. Judge the apex, not the
#      company — criteo.com is a website (its apex is a corporate site); doi.org and
#      tinyurl.com are not (pure redirect resolvers), regardless of who owns them.
#        website 528 · infrastructure 369 · adtech 100 · parked/unknown 29 · adult 12
#   3. Collapse EXACT MIRRORS of the same property only (google.co.uk -> google.com): 38 merged.
#      Deliberately NOT deduplicated by owning organization. An earlier draft did that and it
#      was wrong: it deleted youtube.com and instagram.com from a study of major websites, and
#      it assumed same-owner properties are redundant observations. They are not — measured
#      2026-08-01, facebook.com's robots.txt carries 3 AI-bot rules and instagram.com's carries
#      0. Same company, opposite policy on the exact variable being measured.
#   4. Take the first 500 by Tranco rank. Cutoff rank: 1038.
#
# DISCLOSE WITH ANY PUBLISHED RESULT
#   • Classification was done by LLM from domain knowledge WITHOUT visiting the sites. The full
#     label file (every domain, rank, category, org, reason) ships alongside as
#     domains-500-labels.csv so every exclusion is auditable.
#   • 12 adult domains were excluded for publication suitability, NOT on methodological
#     grounds. This removes a category with atypical crawler policies.
#   • 29 domains could not be confidently identified and were excluded as a documented residual.
#   • This is the head of the web, not the web. These are among the best-resourced sites on
#     earth and the earliest adopters of llms.txt and deliberate AI-crawler policy, so every
#     adoption rate here is an UPPER BOUND. Do not generalize to the long tail.
#   • Scope is the homepage plus domain-root files. Domain-level results (llms.txt, robots.txt,
#     DNS, compression) are site-wide facts; page-level results (schema, word count, headings)
#     must be labelled "measured on the homepage only".
#   • Report the reachability funnel: selected / fetched / recovered via JS-render / unreachable.
#     Sites that block auditors are exactly the population of interest; they must not silently
#     drop out of denominators.
#
# One domain per line. Blank lines and # comments are ignored by scripts/run-study.js.

google.com
cloudflare.com
facebook.com
microsoft.com
youtube.com
apple.com
instagram.com
mail.ru
twitter.com
linkedin.com
dzen.ru
office.com
live.com
amazon.com
azure.com
bing.com
wikipedia.org
github.com
netflix.com
wordpress.org
digicert.com
skype.com
gandi.net
pinterest.com
x.com
outlook.com
roblox.com
tiktok.com
whatsapp.com
icloud.com
yahoo.com
msn.com
spotify.com
adobe.com
chatgpt.com
ntp.org
vimeo.com
zoom.us
nginx.org
baidu.com
qq.com
windows.com
opera.com
mozilla.org
nic.ru
samsung.com
yandex.ru
reddit.com
wordpress.com
sentry.io
ui.com
office365.com
europa.eu
criteo.com
snapchat.com
vk.com
apache.org
nih.gov
intuit.com
dropbox.com
miit.gov.cn
kaspersky.com
gravatar.com
archive.org
godaddy.com
xiaomi.com
reg.ru
f5.com
openai.com
nytimes.com
tumblr.com
epicgames.com
discord.com
paypal.com
applovin.com
shopify.com
flickr.com
mts.ru
soundcloud.com
w3.org
webex.com
medium.com
ebay.com
theguardian.com
cnn.com
oracle.com
t-online.de
creativecommons.org
forbes.com
ozon.ru
bbc.com
cpanel.net
gmail.com
twitch.tv
sciencedirect.com
miui.com
mit.edu
nist.gov
researchgate.net
oxylabs.io
weather.com
sourceforge.net
ubuntu.com
android.com
canva.com
comcast.net
stripe.com
wildberries.ru
avast.com
afternic.com
wikimedia.org
telegram.org
cisco.com
hubspot.com
who.int
imdb.com
roku.com
booking.com
meraki.com
linktr.ee
launchpad.net
springer.com
ibm.com
hp.com
reuters.com
telekom.de
salesforce.com
gov.uk
nature.com
alibaba.com
debian.org
weibo.com
googleblog.com
steampowered.com
harvard.edu
autodesk.com
php.net
ok.ru
wiley.com
etsy.com
datadoghq.com
aliyun.com
launchdarkly.com
gnu.org
dnsmadeeasy.com
drom.ru
ea.com
checkpoint.com
weebly.com
un.org
arubanetworks.com
bilibili.com
cdc.gov
issuu.com
zendesk.com
slack.com
opendns.com
naver.com
wsj.com
trustpilot.com
mi.com
washingtonpost.com
walmart.com
bloomberg.com
crpt.ru
tradingview.com
dell.com
nasa.gov
stanford.edu
nvidia.com
selectel.ru
plesk.com
vkontakte.ru
calendly.com
indeed.com
dailymotion.com
tp-link.com
ivi.ru
temu.com
unsplash.com
aliexpress.com
netangels.ru
gitlab.com
163.com
bsky.app
line.me
globo.com
t-mobile.com
huawei.com
taobao.com
brave.com
beian.gov.cn
sina.com.cn
steamcommunity.com
businessinsider.com
espn.com
pixabay.com
intel.com
sberbank.ru
netease.com
yahoo.co.jp
go.com
name.com
2gis.com
statista.com
cnbc.com
substack.com
mikrotik.com
health.mil
ietf.org
goodreads.com
gamepass.com
gosuslugi.ru
sohu.com
quora.com
duckduckgo.com
npr.org
eventbrite.com
claude.ai
mysql.com
scribd.com
hostgator.com
arxiv.org
duolingo.com
paloaltonetworks.com
fandom.com
docker.com
indiatimes.com
shein.com
stackoverflow.com
foxnews.com
behance.net
mcafee.com
ft.com
aol.com
mediatek.com
slideshare.net
blogger.com
360.cn
okta.com
dynatrace.com
wired.com
firefox.com
visualstudio.com
jquery.com
usatoday.com
time.com
tandfonline.com
supercell.com
trendmicro.com
oup.com
capcut.com
patreon.com
cornell.edu
timeweb.ru
speedtest.net
fast.com
deviantart.com
uol.com.br
tripadvisor.com
rakuten.co.jp
eset.com
berkeley.edu
markmonitor.com
telegraph.co.uk
teamviewer.com
aboutads.info
atlassian.com
grammarly.com
hotjar.com
avito.ru
ca.gov
my.com
wix.com
ieee.org
target.com
mdpi.com
dailymail.co.uk
surveymonkey.com
ipinfo.io
free.fr
klaviyo.com
noaa.gov
loc.gov
threads.com
playstation.com
accuweather.com
onetrust.com
wb.ru
jotform.com
ikea.com
unesco.org
newrelic.com
techcrunch.com
britannica.com
bluehost.com
sagepub.com
squarespace.com
fontawesome.com
tencent.com
att.net
nintendo.com
redhat.com
anthropic.com
digitalocean.com
braze.com
dreamhost.com
elpais.com
viber.com
cnet.com
yelp.com
deepl.com
openstreetmap.org
cbsnews.com
chinamobile.com
ups.com
cambridge.org
mercadolivre.com.br
ubi.com
nike.com
wattpad.com
independent.co.uk
shopee.com.br
imgur.com
intercom.io
meta.com
optimizely.com
consultant.ru
fb.com
anydesk.com
g.co
lemonde.fr
wikia.com
mailchimp.com
shop.app
seznam.cz
uber.com
zillow.com
bitrix24.ru
ted.com
dailymail.com
ya.ru
box.com
kick.com
cursor.sh
rt.ru
wps.com
allaboutcookies.org
mediafire.com
mega.co.nz
asus.com
itch.io
ring.com
iso.org
bandcamp.com
shopee.co.id
mckinsey.com
marriott.com
homedepot.com
magnit.ru
python.org
sophos.com
globalsign.com
fda.gov
networkadvertising.org
mlb.com
nikkei.com
bugsnag.com
airbnb.com
fifa.com
prnewswire.com
bestbuy.com
lenovo.com
fiverr.com
apnews.com
livejournal.com
primevideo.com
bild.de
sedo.com
washington.edu
it.com
hbr.org
ameblo.jp
people.com
epam.com
theverge.com
linode.com
wp.pl
disneyplus.com
daum.net
welt.de
blackberry.com
allegro.pl
academia.edu
nbcnews.com
lefigaro.fr
xbox.com
zoho.com
mayoclinic.org
myspace.com
pexels.com
usda.gov
lowes.com
focus.de
discogs.com
vivo.com.cn
disqus.com
agoda.com
princeton.edu
otto.de
figma.com
latimes.com
qualtrics.com
chess.com
heylink.me
synology.com
biblegateway.com
worldbank.org
thenai.org
lg.com
change.org
gov.br
deloitte.com
bankofamerica.com
arcgis.com
investopedia.com
coupang.com
state.gov
youku.com
jetbrains.com
webmd.com
smartthings.com
fidelity.com
epa.gov
healthline.com
kontur.ru
mercadolibre.com.ar
kleinanzeigen.de
hilton.com
genius.com
verisign.com
weforum.org
merriam-webster.com
life360.com
umich.edu
jd.com
nationalgeographic.com
character.ai
agora.io
repubblica.it
hm.com
hostgator.com.br
rutube.ru
freepik.com
corriere.it
odoo.com
webempresa.eu
theatlantic.com
playrix.com
tailscale.com
nexusmods.com
note.com
columbia.edu
perplexity.ai
themeforest.net
caixa.gov.br
kwai.com
whitehouse.gov
onet.pl
elmundo.es
giphy.com
pbs.org
sofascore.com
theconversation.com
pixiv.net
zhihu.com
onesignal.com
kickstarter.com
binance.com
ancestry.com
tesla.com
huffpost.com
poki.com
ox.ac.uk
costco.com
youronlinechoices.com
jimdo.com
commercialrealestate.com.au
docomo.ne.jp
xerox.com
xing.com
garmin.com
goal.com
flipkart.com
ecosia.org
bol.com
nextcloud.com
statcounter.com
curseforge.com
yale.edu
