User-agent: * Disallow: /cgi-bin/ User-agent: Mozilla/5.0 (Twiceler-0.9 http://www.cuill.com/twiceler/robot.html) Disallow: / User-agent: Mozilla/5.0 (compatible; Pogodak.co.yu/3.1) Disallow: / User-agent: psbot Disallow: / User-agent: CazoodleBot Disallow: / User-agent: dotbot Disallow: / User-agent: AISearchBot (Email: aisearchbot@gmail.com; If your web site doesn't want to be crawled, please send us a email.) Disallow: / User-agent: Mozilla/5.0 (compatible; Charlotte/1.0b; http://www.searchme.com/support/) Disallow: / User-agent: Mozilla/5.0 (compatible; heritrix/1.14.1 +http://www.iseclab.org) Disallow: / User-agent: WebAlta Crawler/2.0 Disallow: / User-agent: my-heritrix-crawler(+http://mywebsite.com) Disallow: / Sitemap: http://www.uscarriers.net/sitemap.xml