Denial of Service in Protego Python Robots.txt Parser by Scrapy

CVE-2026-55520

What is CVE-2026-55520?

The Protego library, a pure-Python robots.txt parser by Scrapy, is susceptible to a denial of service attack due to improper pattern matching. Prior to version 0.6.2, the library's handling of asterisks in Allow or Disallow directives can lead to exponential backtracking when processing complex directives. This behavior allows an attacker to craft a malicious robots.txt file, causing prolonged resource consumption as the parser attempts to match a near-miss URL. This can effectively deny service to legitimate crawlers. The issue has been addressed and resolved in version 0.6.2.

References