Currently scrapy doesn’t handle 429s properly. So, whenever we get 429 response code, we should update throttling configs and concurrency to adapt to the new rate.

A new middleware/extension that will handle 429 response codes and adjust request rates properly.

175 hours

While using Scrapy, there are certain common issues that are hard to detect. For example, a typo in the name of a setting.

Build a list of common issues in code using Scrapy that could be detected using static code analysis, and build a tool or extend an existing tool to detect those.

175 hours or 350 hours depending on the chosen scope

There are different aspects of cookie handling in Scrapy that we should improve.

Update the handling of cookies by Scrapy to meet modern web standards followed by web browsers, and make it easier for Scrapy users to work with cookies.

175 hours or 350 hours depending on the chosen scope

Scrapy does not support TLS 1.3, and it is important that we do to keep up with servers that drop support for older TLS versions

Update our HTTP 1.1 downloader to support TLS 1.3 connections.

350 hours

Update our HTTP/2 downloader to support TLS 1.3 connections as well.

When you inspect a website element in a web browser, you get a DOM-based HTML tree that is different from the actual, underlying HTML tree. This makes it difficult to translate what you find in a web browser into an XPath or CSS expression that can work in Parsel. More so when the underlying HTML is actually broken.

Extend Parsel to support different HTML parsers, and add support for additional HTML parsers.

175 hours