Rules for well-behaving search engines (e.g. Google or Bing)

Nothing can reliably tell a robot from a person, so every visitor is shown a browser check page before the redirect happens. Someone clicking one link barely notices; a scraper that wants thousands of links has to solve the check thousands of times, which makes harvesting your links at scale extremely slow and computationally expensive.

Robots.txt file