GHSA-Q7WC-MR9R-GCJH
Link extractors and SitemapSpider follow file://, ftp:// and s3:// URLs from crawled content
Texto original en inglés · Ver en español
CVSS:4.0/AV:N/AC:L/AT:P/PR:N/UI:N/VC:H/VI:N/VA:N/SC:N/SI:N/SA:N
Fuente: GitHub
Ciclo de vida de la vulnerabilidad
| Fecha | Fase | Evento | Detalle |
|---|---|---|---|
| Divulgación | Publicada | GHSA-Q7WC-MR9R-GCJH divulgada · CVSS 8.2 (GitHub) | |
| Remediación | Parche 2.20.0 | Scrapy · 0 d desde divulgación | |
| Monitoreo | Detectada | Ingresó al índice | |
| Monitoreo | Verificada | Última verificación contra las fuentes |
Impact
`LinkExtractor` https://docs.scrapy.org/en/latest/topics/link-extractors.html#scrapy.linkextractors.lxmlhtml.LxmlLinkExtractor, and so `CrawlSpider` https://docs.scrapy.org/en/latest/topics/spiders.html#scrapy.spiders.CrawlSpider rules that use it, and `SitemapSpider` https://docs.scrapy.org/en/latest/topics/spiders.html#scrapy.spiders.SitemapSpider build requests from URLs found in crawled content: links in HTML pages, `<loc>` entries in sitemaps and sitemap indexes, and `Sitemap` lines in `robots.txt` files.
Link extractors accepted `file://` and `ftp://` URLs, and `SitemapSpider` accepted URLs of any scheme. Scrapy downloads those schemes by default, so a crawled website could:
- make Scrapy read local files through `file://` URLs, passing their contents to spider callbacks, and potentially to the spider output,
- make Scrapy log into an FTP server of its choosing through `ftp://` URLs, sending the `FTP_USER`
https://docs.scrapy.org/en/latest/topics/settings.html#ftp-userand `FTP_PASSWORD`https://docs.scrapy.org/en/latest/topics/settings.html#ftp-passwordcredentials, - from a sitemap, make Scrapy send requests signed with the `AWS_ACCESS_KEY_ID`
https://docs.scrapy.org/en/latest/topics/settings.html#aws-access-key-idcredentials to an S3 bucket of its choosing through `s3://` URLs.
Spiders that set `allowed_domains` https://docs.scrapy.org/en/latest/topics/spiders.html#scrapy.Spider.allowed_domains only follow such URLs when their host is one of those domains, which rules out `file://` URLs.
Patches
Upgrade to Scrapy 2.20.0+.
In Scrapy 2.20.0, link extractors and `SitemapSpider` only follow `http://` and `https://` URLs by default, plus URLs that use the scheme of the response they come from. To follow URLs with other schemes, use the `schemes` parameter of `LinkExtractor` or the new `sitemap_schemes` attribute of `SitemapSpider`.
Workarounds
If you cannot upgrade to Scrapy 2.20.0, consider the following mitigations.
- Disable the download handlers you do not need: for example, set `DOWNLOAD_HANDLERS`
https://docs.scrapy.org/en/latest/topics/settings.html#download-handlersto `{"file": None, "ftp": None, "s3": None}`. See also local and non-network resourceshttps://docs.scrapy.org/en/latest/topics/security.html#security-local-resources. - Drop requests with other schemes: Add a spider middleware
https://docs.scrapy.org/en/latest/topics/spider-middleware.htmlthat drops requests from spider callbacks unless they use `http`, `https` or the scheme of the response. For example:
from scrapy import Request
from scrapy.utils.httpobj import urlparse_cached
class SchemeFilterMiddleware:
def _allowed(self, request_or_item, response):
if not isinstance(request_or_item, Request):
return True
schemes = {"http", "https", urlparse_cached(response).scheme}
return urlparse_cached(request_or_item).scheme in schemes
def process_spider_output(self, response, result, spider):
for request_or_item in result:
if self._allowed(request_or_item, response):
yield request_or_item
async def process_spider_output_async(self, response, result, spider):
async for request_or_item in result:
if self._allowed(request_or_item, response):
yield request_or_itemSPIDER_MIDDLEWARES = {
"myproject.middlewares.SchemeFilterMiddleware": 0,
}Desglose de métricas
Rangos afectados
Estado del parche por paquete
| Paquete / producto | Estado | Corregida en | Desde divulgación | Última versión |
|---|---|---|---|---|
pypiScrapy |
Corregida | 2.20.0 | 0 d | 2.20.0 no afectada |
2.20.0Arsenal público
Sin exploits, módulos ni PoC públicos conocidos a la fecha de verificación.
https://github.com/scrapy/scrapy/security/advisories/GHSA-q7wc-mr9r-gcjhLas referencias externas se muestran para copiar: cve.llama.ec no lo redirige a otros sitios.
