To guarantee the best experience for our mutual clients it might sometimes be required to whitelist our crawlers on your infrastructure. To be able to do that we provide this up-to-date list of DNS entries that will resolve to the server’s IP addresses (IPv4 and IPv6):
- spider-crawler1.do.feeder.co
- spider-crawler2.do.feeder.co
- spider-crawler3.do.feeder.co
- spider-crawler4.do.feeder.co
- spider-crawler5.do.feeder.co
- spider-crawler6.do.feeder.co
- spider-crawler7.do.feeder.co
- spider-crawler8.do.feeder.co
- spider-crawler9.do.feeder.co
- spider-crawler10.do.feeder.co
- spider-worker2.do.feeder.co
- spider-worker3.do.feeder.co
Below is an automatically generate table of the IP addresses for the above servers. We also provide an unauthenticated JSON API for fetching these IPs:
https://feeder.co/1/service-urls/crawlers.json
https://feeder.co/1/service-urls/crawlers.txt
User-agent
Feeder’s user-agent is also something you can add an exception for in your firewall rules. Our user-agent will contain the string “feeder.co“. Example:
Mozilla/5.0 (feeder.co; Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/106.0.0.0 Safari/537.36The exact user-agent may change over time, so we recommend checking for feeder.co.