➔ Scrapy This Week: Proxy Auth, Crawler Settings, and scrapy-lint
| 4 minute read
Scrapy landed 17 commits on master over the past week, touching 71 files with a net gain of about 400 lines. Most of the diff is test coverage, but a handful of changes matter if you run spiders behind proxies, pass prebuilt Crawler objects into runners, or rely on rel="nofollow" to stay off certain pages. The headline fix stops Proxy-Authorization from leaking into upstream requests when using the streaming download handlers.