User-Agent: * Disallow: /cgi-bin/ User-Agent: Googlebot Disallow: User-Agent: SpiderMonkey Disallow: User-Agent: Slurp Disallow: User-Agent: Gigabot Disallow: User-Agent: Arachnoidea Disallow: