# See http://www.robotstxt.org/wc/norobots.html for documentation on how to use the robots.txt file User-agent: bingbot Crawl-delay: 5 User-agent: Twiceler-0.9 Disallow: / User-agent: Twiceler Disallow: / User-agent: twiceler Disallow: / User-agent: * Disallow: /search/* Disallow: /search?* Disallow: /features/* Disallow: /-1/* Disallow: /0/* Disallow: /1/* Disallow: /2/* Disallow: /3/* Disallow: /4/* Disallow: /5/* Disallow: /6/* Disallow: /7/* User-agent: * Allow: /public/Greenwire/* Allow: /public/greenwire/* Allow: /public/EEDaily/* Allow: /public/eedaily/* Allow: /public/eenewspm/* Allow: /public/Landletter/* Allow: /public/landletter/* Allow: /public/climatewire/* Allow: /public/energywire/* Allow: /EEDaily/* Allow: /eedaily/* Allow: /Greenwire/* Allow: /greenwire/* Allow: /eenewspm/* Allow: /Landletter/* Allow: /landletter/* Allow: /climatewire/* Allow: /energywire/* Allow: /stories/*