Table of contents

Defense measures against bots

%3 cluster_ab7c4a31_1481_4426_905e_e1b971b38711 Defense measures against bots cluster_7dfa0ade_e106_4714_956f_da4a975c0cc0 Bot blockers cluster_7f19bd94_0a57_4944_b722_e35897c4e942 Crawler tarpits _04656c13_b5d4_4e67_8b8d_e16f644f6c44 Anubis _fd90f926_db74_4369_ac76_e2144ffebfe9 Proof Of Work _04656c13_b5d4_4e67_8b8d_e16f644f6c44->_fd90f926_db74_4369_ac76_e2144ffebfe9 _55c22b96_83c9_492b_9667_f16caceebac8 Nephentes _b8c09743_8f30_4276_8835_63993b733090 LLMs __0:cluster_ab7c4a31_1481_4426_905e_e1b971b38711->_b8c09743_8f30_4276_8835_63993b733090

Some programs used to trap bad behaving crawlers, the most offensive ones are suspected to be used to train LLMs.

Crawler tarpits

These programs create sprawling web mazes used to distract these crawlers away so they don't request more compute-intensive sites.

Bot blockers

Detect bots and block their access.

Anubis

floss

Uses Proof Of Work to detect crawler bots and block at scale their access to a site.