Index stats

4,769 pages indexed from 2,566 sites · database 268.9 MB

Biggest sites in the index

SitePages
buckthem.com222
en.wikipedia.org174
developer.mozilla.org110
www.nasa.gov110
www.python.org107
xkcd.com102
www.iana.org100
openlibrary.org84
www.weather.gov84
www.gutenberg.org77
incubator.wikimedia.org50
www.loc.gov10
wikisource.org9
www.hurriyet.com.tr8
sd.wikipedia.org7

Crawl frontier

StateURLs
done5,278
failed918
queued340,846

Recent crawl runs (UTC)

WhenEventDetail
2026-11-16 05:03crawl_endpages=4769 (+1385 new) hosts=2566 stats={'http_error': 63, 'fetched': 1508, 'fetch_error': 11, 'robots_blocked': 66, 'too_thin': 6, 'not_html': 4} top_authority=[('https://www.wikimedia.org/', 480), ('https://wikisource.org/wiki/Wikisource:Main_Page', 395), ('https://species.wikimedia.org/wiki/Main_Page', 393), ('https://commons.wikimedia.org/wiki/Main_Page', 393), ('https://foundation.wikimedia.org/wiki/Home', 393)]
2026-11-16 05:00crawl_startbudget=1500 depth=4 per_host=1500 pages_before=3384
2026-11-16 04:59crawl_endpages=3384 (+4 new) hosts=2529 stats={'http_error': 1, 'fetched': 4} top_authority=[('https://wikisource.org/wiki/Wikisource:Main_Page', 379), ('https://species.wikimedia.org/wiki/Main_Page', 377), ('https://commons.wikimedia.org/wiki/Main_Page', 377), ('https://www.wikidata.org/wiki/Wikidata:Main_Page', 374), ('https://www.mediawiki.org/wiki/MediaWiki', 373)]
2026-11-16 04:59crawl_startbudget=3 depth=0 per_host=1200 pages_before=3380
2026-11-16 04:57crawl_endpages=3380 (+3070 new) hosts=2525 stats={'fetched': 3205, 'fetch_error': 194, 'http_error': 465, 'robots_blocked': 153, 'not_html': 38, 'too_thin': 126} top_authority=[('https://wikisource.org/wiki/Wikisource:Main_Page', 379), ('https://species.wikimedia.org/wiki/Main_Page', 377), ('https://commons.wikimedia.org/wiki/Main_Page', 377), ('https://www.wikidata.org/wiki/Wikidata:Main_Page', 374), ('https://www.mediawiki.org/wiki/MediaWiki', 373)]
2026-11-16 04:47crawl_startbudget=3200 depth=4 per_host=900 pages_before=310
2026-11-16 04:40crawl_startbudget=3500 depth=4 per_host=900 pages_before=0

Most searched (aggregate counts only)

QueryCount
credit card perks tracker2

← Back to search