ScoblyBot

ScoblyBot is the crawler used by Scobly, which is in development. Its user agent starts with ScoblyBot and links to this page.

Which sites it fetches

ScoblyBot fetches pages only from sites whose owners have connected them to Scobly. It does not crawl other sites.

For a connected site, some checks also request pages with a search engine crawler's user agent, such as Googlebot's, to see what that crawler is served.

robots.txt

ScoblyBot honors robots.txt for its own crawls.

Contact

If ScoblyBot is causing trouble for your site, or you have a question about it, write to hello@scobly.com. Please include your site's address and roughly when you saw the requests.