1Noonbot,Accoona-AI-Agent,Allblog.net,Baiduspider,Blogbeat,Crawler,DigExt,DrecomBot,Exabot,FeedChecker,FeedFetcher,Gigabot,Googlebot,
Googlebot-Image,HMSE_Robot,IP*Works!,IRLbot,Jigsaw,LWP::Simple,Labrador,MJ12bot,MirrorChecking,MissiguaLocator,MSNBot,NG/2.0,NaverBot,
NutchCVS,PEARHTTP_Request,PostFavorites,SBIder,W3C_Validator,WISEbot,Y!J-BSC,Yahoo!Slurp,Yahoo-MMCrawler,ZyBorg,voyager,archiver,carleson,cfetch,sogou,compatible;Eolin,cowbot,empas,favicon,feedfinder,findlinks,
genieBot,ichiro,ia_archiver,kinjabot,larbin,lwp-trivial,msnbot,psbot,urllib/1.15,wisebot,ADSAComponent,Almaden,Almaden c01,Almaden wf224,Almaden wf55,Amfibibot 0.06,Amsterdam 2002,AnswerBus,Antibot 1.1.11,Antibot 1.1.13,Antibot 1.2.0,AOL 3.0,Appie 1.1,Argus 1.1,Art-Online 0.9,Ask Jeeves,Ask Jeeves Teoma,ASPseek 1.2.10,ASPseek 1.2.11,ASPseek 1.2.12,AvantGo 3.2,BaiDuSpider,BaiDuSpider,Battlebot,BDFetch,BDNcentral Crawler 2.3,BDNcentral Crawler 2.3,Big Brother,BlogBot 1.2,Boitho-dc 0.4,Boitho-dc 0.5,Boitho-dc 66,Boitho-robot 1.0,Boitho-robot 1.1,BorderManager 3.0,BrailleBot 1.0,BruinBot,Bumblebee 1.0,CARI Crawler,CARI Crawler,Cd34 0.1,Cerberian Drtrs,Cerberian Drtrs 3.0,Cerberian Drtrs 3.0,CipinetBot,Clushbot 2.1,Clushbot 3.21,Clushbot 3.23,Clushbot 3.24,Clushbot 3.6,Clushbot 3.9,ComMOOnity LambdaMOO 1.8.1,ConveraCrawler 0.1,ConveraCrawler 0.1,ConveraCrawler 0.2,ConveraCrawler 0.5,Cosmos 0.9,Cowbot 0.1,Cowbot 0.1.1,Crawl_Application,CrocCrawler 3.3,CrocCrawler 4.3,Custo 2.0,CydralSpider 1.9,DeepIndex,DeMozulator 1.0,DoCoMo 1.0,Dual Proxy,Dumbot,EARTHCOM.info 1.2,EmailSiphon,Enterprise_Search 00.136,E-SocietyRobot,Exactseek-crawler 2.63,Exactseek-crawler 2.63,Explorer 6,FAST Enterprise Crawler 6,FAST FirstPage retriever,FastBug,FAST-WebCrawler .2,FAST-WebCrawler 3.4,FAST-WebCrawler 3.6,FAST-WebCrawler 3.6,FAST-WebCrawler 3.7,FAST-WebCrawler 3.8,FAST-WebCrawler 3.x,Favicon,FDSE robot,Filangy 0.01,Filangy 1.01,FindLinks 0.71,Findlinks 0.82,Findlinks 0.87,Findlinks 0.89,Firefly 1.0,FlickBot 1.1,FlickBot 2.0,Fluffy the spider,FusionBot,FyberSpider,GAIS Robot 1.1A2,Gaisbot 3.0,GalaxyBot 1.0,Gatherer 0.9,Gazz 5.0,Generic,GeonaBot 1.0,GeonaBot 1.1,GetRight 4.5,Gigabot 1.0,Girafabot,Goldfire Server,Googlebot 2.1,Googlebot Test,Googlebot-Image 1.0,Green Research, Inc.,GregBot,Grub-client,Grub-client 0.3.0,Grub-client 1.0.3,Grub-client 1.0.4,Grub-client 1.0.5,Grub-client 1.0.6,Grub-client 1.0.7,Grub-client 1.07,Grub-client 1.1.1,Grub-client 1.2.1,Grub-client 1.3.1,Grub-ent 1.3.7,Grub-client 1.4.3,Grub-client 1.5.3,Grub-client 2.3,GulperBot 0.2.4,Harvest-NG 1.0.2,Hatena Antenna 0.4,Hget 0.3,Hitwise Spider 1.0,Htdig,Htdig 3.1.5,Htdig 3.1.6,Html Link Validator,Httpcheck 1.0,HTTPConnect,Httpget 5.2.2,HTTrack 3.0x,Ia_archiver,IcabotAccept,Ichiro 1.0,IconSurf 2.0,ICS 1.2.105,Iltrovatore-Setaccio,IlTrovatore-Setaccio 0.03,Iltrovatore-Setaccio 0.3,IlTrovatore-Setaccio 1.2,IlTrovatore-Setaccio 1.2,Imagefetch 0.1,Indy ibrary,InelaBot 0.2,InfoSeek Sidewinder 1.0A,Infoseek SideWinder 1.45,Infoseek SideWinder 2.0B,Ingrid 3.0,Inktomi slurp,InternetLinkAgent 3.1,IPiumBot,IRLbot 1.0,Istarthere,Java 1.4.0,JoBo 1.3,K2spider,KMcrawler,Knowledge 0.2,Knowledge 0.3,Knowledge Engine,Kuloko-bot 0.2,Larbin,Larbin 2.6.2,Larbin 2.6.3,Larbin 2.63,Libwww-MGET 1.0,Libwww-perl,Libwww-perl 5.3.2,Libwww-perl 5.4.0,Libwww-perl 5.48,Libwww-perl 5.50,Libwww-perl 5.51,Libwww-perl 5.52,Libwww-perl 5.53,Libwww-perl 5.63,Libwww-perl 5.64,Libwww-perl 5.65,Libwww-perl 5.66,Libwww-perl 5.68,Libwww-perl 5.69,Libwww-perl 5.74,Libwww-perl 5.75,Libwww-perl 5.76,Libwww-perl 5.800,Libwww-perl 5.801,Libwww-perl 5.802,Libwww-perl 5.803,LimeBot 1.0,Linkbot 3.0,LinkLint-checkonly 2.3.5,Linknzbot,Linknzbot 2004,Links SQL,Lite Bot 0616B,LNSpiderguy,Look,Lwp-request 1.29,Lwp-request 1.35,Lwp-request 1.36,Lwp-request 2.01,Lwp-request 5.48,Lwp-request 5.65,Lycos_Spider,Mediapartners-Google 2.1,Mercator 2.0,Metacarta,MetaGer-LinkChecker,Microsoft URL Control,Microsoft-ATL-Native 7.00,MicrosoftPrototypeCrawler,Moget 1.0,Moget 2.1,MozDex 0.04,MozDex 0.05,Mozilla,MS Data Access,MS FrontPage 4.0,MS FrontPage 5.0,MSIECrawler,Msnbot 0.1,Msnbot 0.11,Msnbot 0.3,Msnbot 1.0,MSProxy 2.0,MSRBot 0.1,MysteryBot,NaverBot 1.0,NaverBot_dloader 1.5,NaverRobot 1.0,NaverRobot 1.5,NetAnts 1.25,NetNoseCrawler 1.0,NetNoseCrawler 2.0,NetResearchServer,NetResearchServer 2.4,NetResearchServer 2.5,NetResearchServer 2.7,NetResearchServer 2.8,NetResearchServer 2.9,NetResearchServer 3.4,NextGenSearchBot 1,NG 1.0,None,NPBot,NuSearch Spider,NutchBot 0.05,NutchBot 0.06,NutchCrawler,NutchOrg 0.03,Obot,Ocelli 1.3,OmniExplorer_Bot 1.07,Openbot 3.0,OrangeBot,Organica 2.24,Overture-WebCrawler 3.8,OWR_Crawler 0.1,Parabot,Patwebbot,Pavuk 0.9,PEERbot,PipeLiner 0.3,Planethosting,Polybot 1.0,Pompos 1.1,Pompos 1.2,Pompos 1.3,Portal Manager 0.7,Potbot 1.0,Program Shareware 1.0.3,ProWebGuide Link Checker,Psbot 0.1,Pverify 1.2,PWS.Kiosk,QPCreep Test Rig,QuepasaCreep,QuepasaCreep 0.9.14,QuepasaCreep v0.9.13,Reifier,Rico 0.1,RixBot,RoboPal,RobotMidareru 0.7,Robots.txt ,alidator,Robozilla 1.0,RPT-HTTPClient 0.3,SafariBookmarkChecker 1.25,SafariBookmarkChecker 1.26,Scooter 1.0,Scooter 1.1,Scooter 3.0,Scooter 3.2,Scooter 3.3,Scrubby 2.1,Scrubby 2.2,Search Agent 1.0,SearchSpider 1.1,Seekbot 1.0,Semanticdiscovery 0.1,Sensis.au Web Crawler,Sherlock 1.3,Sherlock_spider,Sitecheck,Sitescooper 3.1.2,SiteXpert,SlySearch 1.3,Sohu-search,Speedy Spider,SpeedySpider,SpiderKU 0.9,SpiderMonkey 7.04,SpiderMonkey ,.06,SpurlBot 0.2,Sqworm 2.9.85,Star Downloader,Steeler 1.3,SuperCleaner 2.56,SYCLIKControl LinkChecker,Szukacz 1.5,Tarantula,TclSOAP 1.0,TclSOAP 2.3,Teradex ,apper,Teradex_Crawler,TheSuBot 0.1,TheSuBot Beta,Thumbshots-de-Bot,TimboBot 0.9,Tkensaku 0.9,TranSGeniKBot,TranSGeniKBot www.tsgk.net,TulipChain 5.7,TulipChain 5.94,TulipChain 6.01,TulipChain 6.02,TulipChain 6.03,TurnitinBot 1.4,TurnitinBot 1.5,TurnitinBot 2.0,TutorGigBot 1.5,Tutorial Crawler 1.4,UdmSearch 3.1.20,UIowaCrawler 1.0,UIowaCrawler 2.0,Unchaos_crawler 2.0.2,unknown,Updated 0.1,USyd-NLP-Spider,Vagabondo 2.0,Vagabondo 2.1,Vivante Link ,hecker,Void-bot 0.1,VoilaBot 1.6,VSE 1.0,Vspider,W3C_Validator 1.183,W3C_Validator 1.305.2.109,W3C_Validator 1.305.2.12,W3C_Validator 1.305.2.137,W3C_Validator 1.305.2.148,W3C-checklink 2.90,W3C-checklink 3.6.2.3,W3C-checklink 3.9.2,W3C-checklink 4.0,W3C-checklink 4.1,Webbot,Webclipping,Webcollage 1.102,Webcollage 1.104,Webcollage 1.87,Webcollage 1.93,Webcollage 1.94,WebcraftBoot,WebcraftBoot 1.4.1,WebcraftBoot 1.4.2,WebFilter Robot 1.0,Webinator-indexer 2.56,WebRACE 1.1,WebSauger 1.20,WebSearch,WebSearch 2.0.1,WebSearch 3.0.1,WebSearch 3.1.3,WebSearch 3.1.6,Webwombat,Webyield robot,Wget 1.5.2,Wget 1.5.3,Wget 1.5.3.1,Wget 1.6,Wget 1.7,Wget 1.8,Wget 1.8.1,Wget 1.8.2,Wget 1.9,Wget 1.9.1,Willow 2.1,Wotbox ,lpha0.5.1,Www.ciml.co.uk,WWWeasel 1.00,Xenu Link Sleuth 1.1a,Xenu Link Sleuth 1.2b,Xenu Link Sleuth 1.2d,Xenu Link Sleuth 1.2e,Xenu Link Sleuth 1.2f,Yahoo! ,lurp,Yahoo-MMCrawler,YottaCars_Bot 4.12,Zao 0.1,Zao 0.2,Zao-Crawler,Zeus 3140,Zeus 57657,ZipppBot 0.11,ZoomSpider,ZyBorg 1.0