AI Crawler robots.txt Study

huggingface.cosite

robots.txt policies of the Tranco top 10,000 domains (October 2026), one row per domain with rank band and whether AI training, AI search and user-triggered crawlers are named or blocked. CSV, CC BY 4.0.

from

Data Science

added

2026-10-19

likes

0

similar

1. ▲

   AI Today cognilytica.com

   site topic ai link ↗
   site
2. ▲

   Fetch Robotics fetchrobotics.com

   site link ↗
   
3. ▲

   Fiddler AI fiddler.ai

   site topic ai link ↗
   
4. ▲

   GitHub Repository github.com

   Official AI PR Reviewer repository.

   tool topic ai link ↗
   
5. ▲

   Responsible AI practices ai.google

   ⭐

   site topic ai link ↗
   
6. ▲

   FAQ coderabbit.ai

   Frequently asked questions about CodeRabbit.

   site link ↗
   

Data Science › Fun > Datasets: “robots.txt policies of the Tranco top 10,000 domains (October 2026), one row per domain with rank band and whether AI training, AI search and user-triggered crawlers are named or blocked. CSV, CC BY 4.0.”