Feeder is an RSS service that crawls feeds periodically to check for new updates. Feeder has custom software and servers to do this, and if you are a site owner you might see our User-agent pop up in your logs.
We only ever crawl feeds that are actively subscribed to by our users. As such the Feeder crawler is acting as an agent on behalf of the user and don't check for presence of robots.txt.
User-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6; +feeder.co/crawler) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36
User-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2; +feeder.co/crawler) AppleWebKit/537.31 (KHTML, like Gecko) Chrome/37.0.2062.120 Safari/537.36
If you believe the Feeder crawler is accessing your site too often and wish to set a rate limit we sincerely urge you to contact us at [email protected] We always try to answer and fix issues as fast as possible, so barring holidays or other occasions, expect an answer within 24 hours.