Amber Crawler
Amber is my personal web archiver. If you found this page from a User-Agent string in your server logs, this is the crawler that visited you.
About
[edit | edit source]Amber keeps versioned, replayable copies of websites I read. The idea is to have my own personal Wayback Machine that I can use to look at historical content without getting slammed by Wayback Machine's onerous rate limits. I also index these pages so that I have a personal search engine to work with.
It identifies itself as:
amber/1.0 (personal archiver; +https://wiki.roshangeorge.dev/w/Amber_Crawler; [email protected])Requests come from 64.71.176.178.
Behavior
[edit | edit source]- It obeys
robots.txt, includingCrawl-delay.- It fetches that file once per crawl and uses the cached copy
- It looks for the
amberUser-Agenttoken before every fetch but if you mark*, it will obey that too.
- It is slow by default: at most about three requests per second per host, with jitter, and typically far less. Crawls are capped in number of pages and bytes.
- It obeys HTTP 429/503 and
Retry-Afterwith exponential backoff. - It is GET only, so won't submit forms etc.
Opting Out
[edit | edit source]Add this to your robots.txt:
User-agent: amber
Disallow: /If you want a hard-ban or have complaints, email me at [email protected]