Skip to content

gitlab: decrease impact of the dumbest crawlers

Nicolas Wavrant requested to merge gitlab-bot-protection into master

Use the approach presented here: https://her.esy.fun/posts/0031-how-i-protect-my-forgejo-instance-from-ai-web-crawlers/index.html

to try to reduce the log caused by crawlers.

The idea is make users visiting the website with a JS-enabled browser register a cookie, and then redirect them transparently to the website. If the cookie is set, they can access it. If not, they loop on the simple webpage served by nginx asking them to set the cookie.

There is an exception when the user agent is the git command line.

This approach is completely transparent for the users.

There will probably be other impacts on our infrastructure, for exemple code retrieving the JSON schemas. Let's address this after we can prove this change has any effect on the bot traffic.

Merge request reports