Add User-agent blocking list in varnish
We are struggling with crawlers (again).
root@moma:~# top-known-bot-varnish-hits.sh
----------------------------- -----------------------------
🌐 Web Crawlers 📆 since 25/Sep/2024:00:00:28
----------------------------- -----------------------------
Hits Ratio
-----------------------------
Svix-Webhooks 2361844 35.1143
ClaudeBot 2248407 33.4278
facebookexternalhit 491420 7.3061
meta-externalagent 266058 3.95557
Amazonbot 180120 2.6779
Bytespider 164100 2.43973
fossology 125449 1.86509
AhrefsBot 81814 1.21635
bingbot 59565 0.885572
GoogleOther 37750 0.561241
SemrushBot 22479 0.334202
PetalBot 12126 0.180281
Googlebot 8613 0.128052
Applebot 3218 0.047843
DataForSeoBot 3115 0.0463117
YandexBot 529 0.00786481
YandexRenderResourcesBot 243 0.00361276
Baiduspider 6 8.92039e-05
Turnitin 2 2.97346e-05
ImagesiftBot 0 0
DuckDuckGo 0 0
ChatGPT-User 0 0
----------------------------- -----------------------------
Total requests 6726164 100
Edited by Guillaume Samson