List of webbots

Download without IP addresses XML format JSON format CSV format

Download including IP addresses XML format JSON format CSV format

Bot name
Last detected
IPs known
 
360Spider
2019-11-12
0
3w24bot/1.2
2019-07-16
0
3w24bot/1.3
2019-07-12
0
3w24bot/1.4
2020-03-29
0
7Siters/1.08
2019-07-10
0
AdAuth/1.0
2020-10-21
1
adidxbot/2.0
2020-06-01
0
admantx-adform/3.1
2020-09-18
24
admantx-eusyncbatch01/3.1
2019-10-02
1
admantx-sap/2.4
2019-07-13
0
admantx-usaspb/3.1
2020-03-15
0
admantx-ussy04/3.1
2020-10-18
2
admantx-ussy04/3.2
2020-10-26
1
AdsBot-Adsrvr
2019-10-29
1
AdsBot-Google
2019-12-27
0
AdsBot-Google-Mobile
2020-01-20
0
Adsbot/3.1
2020-10-20
8
adscanner
2019-09-06
2
adscanner/1.0
2020-11-22
2
adscanner/1.0
2019-10-02
4
adscanner/1.1
2020-03-17
4
AdsTxtCrawler/1.0
2020-11-26
2
AffiliateLabz/1.0
2020-02-05
0
AhrefsBot/5.2
2019-05-26
3
AhrefsBot/6.1
2020-12-01
680
AhrefsBot/7.0
2020-12-01
405
aiHitBot/2.9
2020-10-05
2
AlphaBot/3.2
2019-11-06
0
Amazonbot/0.1
2020-10-02
0
AppEngine-Google
2020-06-12
0
Applebot/0.1
2020-12-01
472
archive.org_bot
2020-05-13
0
archive.org_bot
2019-09-08
2
AteniumBot
2020-11-16
0
AwarioRssBot/1.0
2020-08-14
2
Baidu Japan
2019-05-18
0
Baiduspider
2020-07-18
0
Baiduspider-image+
2019-10-24
42
Baiduspider-render/2.0
2020-12-01
261
Baiduspider/2.0
2020-12-01
515
BananaBot/0.6.1
2020-07-11
0
Barkrowler/0.9
2019-07-04
8
Barkrowler/0.9
2020-12-01
13
bidswitchbot/1.0
2020-11-30
28
Bigsearch.ca/Nutch-1.0-dev
2019-05-18
0
bingbot/2.0
2020-12-01
1611
BingPreview/1.0b
2020-12-01
1384
bitlybot/3.0
2020-01-25
1
BLEXBot/1.0
2020-12-01
44
BorneoBot/0.7.1
2020-05-24
0
BorneoBot/0.9.1
2020-05-26
0
BUbiNG
2020-09-13
0
BuiltWith/1.0
2019-12-14
0
BuiltWith/1.1
2020-04-22
0
bumb.ly/0.01
2019-11-10
0
Bytespider
2020-11-30
268
CATExplorador/1.0beta
2020-11-26
11
CCBot/2.0
2020-11-30
293
CensysInspect/1.1
2020-11-30
35
centuryb.o.t9
2020-11-27
1
CheckMarkNetwork/1.0
2020-11-22
115
Cincraw/1.0
2020-02-18
0
CipaCrawler/3.0
2020-02-08
5
CISPA Webcrawler
2020-07-06
0
Clarabot/1.4
2020-04-18
10
Cliqzbot/2.0
2019-08-13
44
Cliqzbot/3.0
2020-07-12
228
CloudFlare-AlwaysOnline/1.0
2020-02-23
1
CMS Crawler
2020-11-15
1
coccocbot-image/1.0
2020-11-30
18
coccocbot-web/1.0
2020-11-28
23
Cocolyzebot/1.0
2020-12-01
28
colly
2020-05-16
0
contxbot/1.
2019-10-20
0
Crawlson/1.0
2020-11-30
18
CSS Certificate Spider
2019-08-01
0
CZ.NIC DNS crawler
2020-11-26
3
datagnionbot
2020-09-02
1
Dataprovider.com
2020-11-13
70
DataXu/1.0
2020-11-30
6
DF Bot 1.0
2020-04-12
0
Diffbot/0.1
2020-11-06
8
Discordbot/2.0
2020-10-04
4
dns-crawler/1.4.10.dev6
2020-08-20
0
dns-crawler/1.4.10.dev7
2020-08-27
0
dns-crawler/1.4.5
2020-06-25
0
dns-crawler/1.4.6
2020-07-09
0
dns-crawler/1.4.6.dev3
2020-06-25
0
dns-crawler/1.4.8
2020-07-16
0
DNSResearchBot/2.1
2020-11-22
1
DomainCrawler/3.0
2019-08-26
10
Domains Project
2020-01-20
0
Domains Project/1.0.1
2020-02-07
0
Domains Project/1.0.3
2020-03-09
0
Domains Project/1.0.4
2020-04-14
0
Domains Project/1.0.6
2020-04-21
0
Domains Project/1.0.8
2020-06-21
0
Domains Project/1.1.0
2020-08-05
0
Domains Project/1.1.1
2020-08-07
0
Domains Project/1.1.4
2020-08-09
0
Domains Project/1.2.1
2020-09-19
0
Domains Project/1.2.3
2020-10-18
0
Domains Project/1.2.4
2020-11-05
0
DomainSigmaCrawler/0.1
2019-11-24
0
DomainStatsBot/1.0
2020-11-10
6
DomCopBot
2020-08-10
0
DotBot/1.1
2020-12-01
34
DuckDuckBot-Https/1.1
2020-10-04
8
DuckDuckBot/1.0
2020-01-03
0
DuckDuckGo-Favicons-Bot/1.0
2020-12-01
3
E. Orliac, G. Fourestey/1.0
2019-08-27
0
eContext/1.0
2019-10-30
0
Elisabot
2020-11-03
3
evc-batch/2.0
2020-11-29
3
Exabot/3.0
2019-08-04
3
facebookexternalhit/1.0
2020-10-14
9
facebookexternalhit/1.1
2020-12-01
750
facebookscraper/1.0
2020-07-03
0
FastCrawler/4.9
2019-06-05
0
FatBoyKimComBot
2019-11-02
0
Faveeo/1.0
2020-01-25
0
Feedfetcher-Google
2020-05-31
0
Feedly/1.0
2020-11-30
4
FeedlyBot/1.0
2020-01-10
1
FemtosearchBot/1.0
2020-06-23
5
FlipboardProxy/1.2
2020-01-25
0
Gaisbot/3.0
2020-08-25
0
GarlikCrawler/1.2
2020-11-05
1
GigablastOpenSource/1.0
2020-10-15
4
Gigabot
2020-08-12
0
Gigabot/1.1.1.2
2019-10-17
0
Gluten Free Crawler/1.0
2019-12-14
1
go-resty/2.0.0
2019-10-04
1
Google-Adwords-Instant
2020-07-07
3
Google-Read-Aloud
2020-10-06
0
Googlebot-Image/1.0
2020-12-01
493
Googlebot/2.1
2020-12-01
3700
got
2020-11-25
4
Grahambot/0.1
2019-05-18
0
Grammarly/1.0
2019-11-09
0
GrapeshotCrawler/2.0
2020-12-01
49
Graydon bot/1.0
2020-09-05
1
Gregarius/0.5.2
2020-08-27
0
heritrix/3.4.0
2020-07-20
0
HTTP Banner Detection
2020-12-01
4
HuaweiWebCatBot/6.0
2020-10-27
1
HubSpot Connect/2.0
2019-10-19
0
HubSpot Webcrawler
2019-11-28
0
hypestat/1.0
2020-03-31
0
ia_archiver
2020-11-13
13
ia_archiver/1.0
2020-04-08
0
IAS crawler
2020-11-30
4
ias-ir/3.1
2020-12-01
65
ias-or/3.1
2020-11-29
19
ias-va/3.1
2020-12-01
113
Iframely/1.3.1
2019-08-19
0
IndeedBot 1.1
2019-11-12
0
InoopaBot/1.0
2020-11-28
1
istellabot/t.1.13
2019-08-01
3
Jaunt/1.5
2019-11-16
0
JobboerseBot
2020-01-28
7
Jooblebot/2.0
2019-06-28
0
K7MLWCBot/1.0
2020-02-06
0
Keybot Translation-Search-Machine
2020-08-09
0
Khazandebot/2.1
2020-08-02
0
KOCMOHABT
2020-11-29
16
KomodiaBot/1.0
2020-11-08
6
LightspeedSystemsCrawler
2020-11-01
4
Linespider/1.1
2020-11-30
20
Linguee Bot
2020-06-16
4
LinkedInBot/1.0
2019-10-01
2
LinkpadBot/1.12
2019-07-28
1
LinkpadBot/2.3
2019-10-10
0
LinkWalker/3.0
2020-08-16
1
LivelapBot/0.2
2019-06-27
0
looid.com Search Engine/0.1
2019-08-30
1
ltx71
2020-11-29
6
Mail.RU_Bot/2.0
2020-11-30
149
Mail.RU_Bot/Img/2.0
2020-01-20
13
Mail.RU_Bot/Robots/2.0
2020-12-01
17
Mappy/1.0
2020-01-07
1
masscan/1.0
2020-11-30
24
MauiBot
2020-05-21
0
MauiBot
2019-09-09
2
MBCrawler/1.0
2020-11-30
1
Mediapartners-Google
2020-12-01
323
Mediatoolkitbot
2020-09-02
2
meg/0.2
2020-11-20
2
MegaIndex.ru/2.0
2020-11-20
8
MetaInspector/5.6.0
2019-12-05
1
MixnodeCache/1.8
2020-11-30
3
MJ12bot/v1.4.7
2019-07-20
2
MJ12bot/v1.4.8
2020-12-01
526
MojeekBot/0.10
2020-12-01
1
MojeekBot/0.6
2019-09-28
1
MojeekBot/0.7
2020-06-27
1
MojeekBot/0.9
2020-07-01
0
msnbot-media/1.1
2020-09-23
0
msnbot/1.0
2020-08-06
0
msnbot/1.1
2020-04-06
0
msnbot/2.0b
2020-05-01
641
MuckRack/1.0
2020-01-25
0
MxToolbox/Beta7
2020-02-02
0
NameFo
2020-01-12
0
netEstate NE Crawler
2020-11-16
3
NetpeakCheckerBot/3.2
2019-08-29
0
Nimbostratus-Bot/v1.3.2
2020-12-01
48
niocBot
2020-03-16
0
niocBot
2020-04-13
0
Nmap Scripting Engine
0
node-fetch/1.0
2020-12-01
12
oBot/2.3.1
2020-11-30
31
olbicobot/Nutch-1.15-SNAPSHOT
2020-10-26
2
Onespot-ScraperBot/1.0
2020-01-14
0
PagePeeker/3.0
2020-07-09
0
PageThing
2020-11-17
2
Pandalytics/1.0
2020-10-04
30
panscient
2020-09-23
3
Panscient/1.0
2019-12-15
0
PaperLiBot/2.1
2020-06-08
3
PartyflockLinkChecker/0.3
2019-10-30
1
PetalBot
2020-12-01
1449
Picoweb_Bot
2019-11-20
0
pmoz.info ODP link checker
2020-07-03
1
proximic
2020-12-01
954
psbot/0.1
2020-01-17
0
Qwantbot/1.0
2020-10-14
2
Qwantify/2.4w
2020-05-18
67
Qwantify/Bleriot/1.1
2020-02-06
11
Qwantify/Bleriot/1.2.1
2019-10-03
0
Re-re Studio
2019-09-29
2
Researchscan/t12ca
2020-04-14
3
Researchscan/t12sns
2020-06-25
3
Researchscan/t13rl
2020-06-25
3
Riddler
2020-11-18
2
RSSingBot
2019-10-11
0
RyowlEngine/1.0
2020-01-22
0
SafeDNSBot
2020-03-17
3
Scrapy/1.4.0
2020-10-19
7
Scrapy/1.5.0
2019-12-09
2
Scrapy/1.5.1
2020-03-07
2
Scrapy/1.5.2
2019-07-18
1
Scrapy/1.6.0
2019-08-13
3
Scrapy/1.7.2
2020-11-11
4
Scrapy/1.8.0
2020-03-16
2
Scrapy/2.0.1
2020-04-24
0
Screaming Frog SEO Spider/10.1
2020-01-09
0
Screaming Frog SEO Spider/11.2
2020-02-28
1
Screaming Frog SEO Spider/11.3
2019-10-02
0
Screaming Frog SEO Spider/12.2
2019-12-09
0
Screaming Frog SEO Spider/12.3
2019-12-15
0
Screaming Frog SEO Spider/12.6
2020-02-18
0
Screaming Frog SEO Spider/13.2
2020-11-09
1
Screaming Frog SEO Spider/5.1
2020-04-20
0
Screaming Frog SEO Spider/6.2
2019-11-20
0
SecSpider/1.0
2020-11-05
0
Seekport Crawler
2020-12-01
97
Semanticbot/1.0
2020-09-02
0
SemanticScholarBot
2020-09-02
0
SemrushBot
2020-11-30
5
SemrushBot-BA
2020-11-27
23
SemrushBot-SA/0.97
2020-11-30
4
SemrushBot-SI/0.97
2019-09-30
0
SemrushBot/3~bl
2019-08-21
29
SemrushBot/6~bl
2020-11-25
70
SemrushBot/7~bl
2020-12-01
39
SEOkicks
2020-12-01
14
SEOlizer/1.1
2020-08-15
0
serpstatbot/1.0
2020-05-31
26
serpstatbot/2.0 beta
2020-11-29
1
SeznamBot/3.2
2020-11-30
144
SeznamBot/3.2-test1
2019-12-28
3
Sidetrade indexer bot
2020-11-24
1
SiteAnalyzerbot/2.0
2020-06-22
0
SiteCheckerBotCrawler/1.0
2019-10-16
0
SiteExplorer/1.1b
2019-05-20
1
Slackbot 1.0
2020-11-09
17
Slackbot-LinkExpanding 1.0
2020-11-09
37
SMTBot/1.0
2020-11-27
4
SniffRSS/0.5beta
2019-05-18
0
Sogou web spider/4.0
2020-12-01
418
special_archiver/3.1.1
2020-09-19
5
SpiderLing
2019-10-21
0
startmebot/1.0
2020-12-01
883
Startpagina-Linkchecker/1.0
2020-12-01
2
Streamline3Bot/1.0
2019-09-29
1
SurdotlyBot/1.0
2020-11-28
83
TBI-HttpOpenPlugin/0.0.2
2020-08-22
0
TBI-HttpOpenPlugin/0.1.0
2020-11-27
12
TBI-WebScanner/0.0.1
2020-11-10
12
The Knowledge AI
2020-10-19
5
Thither.Direct Bot/0.0.1
2019-06-14
0
TinEye-bot/1.31
2019-12-10
5
TOBBOT
2019-12-19
0
tracemyfile/1.0
2020-03-10
1
trendictionbot/0.5.0
2020-10-29
3
TTD-Content
2020-12-01
24
Turnitin
2020-06-12
0
TurnitinBot
2020-05-16
3
TweetmemeBot/4.0
2019-06-01
2
Twingly Recon
2019-05-18
0
Twitterbot/1.0
2020-11-24
9
Twurly v1.1
2020-01-25
0
Typhoeus
2020-05-27
0
uipbot/1.0
2020-02-22
0
UniversalFeedParser/5.2.1
2020-05-30
0
Uptimebot/1.0
2019-06-27
177
Vagabondo/4.0
2020-06-18
7
Validator.nu/LV
2020-03-21
5
vebidoobot/1.0
2020-11-17
4
VelenPublicWebCrawler/1.0
2020-11-12
20
vuhuvBot/1.0
2020-08-31
3
W3C_Validator/1.3
2020-03-21
4
Wappalyzer
2019-11-03
0
WbSrch/1.1
2019-06-26
1
weborama-fetcher
2020-11-21
4
webtechbot
2020-06-14
0
woorankreview/2.0
2020-11-21
73
Wotbox/2.01
2019-10-06
0
XenForo/2.x
2019-09-24
0
Y!J-ASR/1.0
2020-03-10
0
yacybot
2020-10-26
0
Yahoo Link Preview
2020-08-11
0
Yahoo! Slurp
2020-11-26
18
Yahoo! Slurp/3.0
2020-02-11
0
YaK/1.0
2020-11-20
16
Yandex/1.01.001
2019-05-18
0
YandexAccessibilityBot/3.0
2020-12-01
16
YandexBot/3.0
2020-12-01
402
YandexImages/3.0
2020-11-29
48
YandexMobileBot/3.0
2020-09-10
34
YandexNews/4.0
2020-06-16
0
Yesslebot
2020-11-06
0
YioopBot
2020-08-21
0
YisouSpider
2020-11-24
50
ZoomBot
2020-06-16
2