GHSA-26PM-PX5V-8C4W
Uncontrolled resource consumption: an unbounded `separators` list on `POST /documents/text` drives synchronous recursive chunking on the asyncio event loop
CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H
Fuente: GitHub
Ciclo de vida de la vulnerabilidad
| Fecha | Fase | Evento | Detalle |
|---|---|---|---|
| Remediación | Parche 1.5.6 | lightrag-hku · -3 d desde divulgación | |
| Divulgación | Publicada | GHSA-26PM-PX5V-8C4W divulgada · CVSS 7.5 (GitHub) | |
| Monitoreo | Detectada | Ingresó al índice | |
| Monitoreo | Verificada | Última verificación contra las fuentes |
Summary
`RecursiveCharacterChunkParams.separators` is a request-supplied `list[str]` with no length bound of any kind. The recursive-character chunker descends one recursion level per remaining separator, and at every level it re-runs `re.search`, `re.finditer` and a whole-text `tokenizer.encode`. Total work is `O(len(separators) x len(text))` with both factors controlled by the request body.
The chunker is then invoked synchronously inside an `async def` (`lightrag/pipeline.py:2317-2328`), with no `asyncio.to_thread`, so the entire burn runs on the asyncio event loop and the HTTP server stops answering anything for its duration.
`POST /documents/text` has no body-size limit at all, so `len(text)` is unbounded: `MAX_UPLOAD_SIZE` is enforced only on the multipart upload route.
Demonstrated impact against a live `lightrag-server` in its default configuration: one 414,221-byte `POST /documents/text` was answered `200 OK` in 0.001609 s, and the next `GET /health` took 63.120969 s against a baseline of about 1.5 ms. The request that causes the freeze completes before the freeze starts.
Preconditions, stated here rather than buried: default configuration, no environment variables set. Every route is behind `Depends(combined_auth)`, which in the shipped default (no `LIGHTRAG_API_KEY`, no `AUTH_ACCOUNTS`) admits everyone, so this is reachable pre-authentication as shipped. That default is an existing published advisory and is not re-reported here; it is only the precondition. With an API key configured this becomes an authenticated denial of service, which is still meaningful because any read-only API-key holder can stop the whole service with a single request.
Severity / CVSS
`CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H`
Score: 7.5 (High).
`PR:N` is scored for the shipped default configuration described above, where `combined_auth` admits every caller. In a deployment with `LIGHTRAG_API_KEY` or `AUTH_ACCOUNTS` configured the correct vector is `PR:L` (6.5, Medium), because any credential holder, including a read-only one, can still trigger it. `AC:L` because the payload is a single JSON POST with no timing, no race and no per-target tuning: the same body works against any deployment. `A:H` because the loss is total rather than partial: the event loop is the single thread serving every route, so `GET /health`, which needs no authentication and touches no storage, is blocked for the whole interval. `C:N`/`I:N` because nothing is read or written beyond the attacker's own submitted document.
CWE-1050 (Excessive Platform Resource Consumption within a Loop), CWE-770 (Allocation of Resources Without Limits or Throttling), CWE-674 (Uncontrolled Recursion).
Affected component
- `lightrag/api/routers/document_routes.py:293-294` `class RecursiveCharacterChunkParams(_OverlapChunkParams)` with `separators: Optional[list[str]] = None`, no `max_length` and no per-element length bound
- `lightrag/api/routers/document_routes.py:1887` `chunk_options[strategy_key].update(chunking.params)`, where the raw list is copied into the resolved snapshot
- `lightrag/api/routers/document_routes.py:1893` `_validate_effective_chunk_overlap` and `:1937` `_validate_effective_semantic_amount`, the only two effective-value validators, neither of which inspects `separators`
- `lightrag/api/routers/document_routes.py:2837` `insert_text`, `:2899` `_resolve_text_chunking`, `:2906-2918` the `_indexing_task` that calls `pipeline_index_texts` (`:1965`)
- `lightrag/api/routers/document_routes.py:457-467` `InsertTextsRequest`, a `list[str]` of texts with `min_length=1` and no `max_length`, sharing one `chunking` config
- `lightrag/pipeline.py:2120` `async def process_single_document`
- `lightrag/pipeline.py:2317-2328` the `R` branch calling `chunking_by_recursive_character` synchronously, no `asyncio.to_thread`
- `lightrag/pipeline.py:2340` the `V` branch, `await chunking_by_semantic_vector(...)`, which is the awaited counterexample in the same `if`/`elif` chain
- `lightrag/chunker/recursive_character.py:344` entry into `_split_text_with_spans`
- `lightrag/chunker/recursive_character.py:210-289` `_split_text_with_spans`, with the self-recursion at `:266`, the per-split `length_function` call at `:247`, and `re.search` at `:230`
- `lightrag/chunker/recursive_character.py:54` `re.finditer` and `:76-95` the `keep_separator` branch that produces the no-op split
- `lightrag/chunker/recursive_character.py:325-326` `length_function`, a full `tokenizer.encode` over the text handed to it
- `lightrag/api/routers/document_routes.py:2694` and `:2766` the only `max_upload_size` checks, both on the multipart route
- `lightrag/api/lightrag_server.py:1389` and `:1409` the complete set of registered middleware (`_RootPathNormalizationMiddleware`, itself conditional on `api_prefix`, and `CORSMiddleware`); there is no request-size middleware
Version tested: 1.5.5, api version 0316, commit `79b54271`.
Root cause
Two facts combine.
- The separator cascade is unbounded and nothing validates it. `RecursiveCharacterChunkParams.separators` (`document_routes.py:294`) is a plain `Optional[list[str]]`. The sibling integer fields in the same model family carry `ge=` bounds; this list carries none, neither on its length nor on the length of its elements. The resolution path copies it through verbatim at `:1887`, and the two effective-value validators that run immediately after, `_validate_effective_chunk_overlap` (`:1893`) and `_validate_effective_semantic_amount` (`:1937`), inspect overlap and breakpoint amounts respectively and never look at `separators`. A list of 100,000 separators is accepted by the real request model and the real resolver, verified below.
- The recursive chunker runs on the event loop. In `process_single_document`, the `R` branch calls `chunking_by_recursive_character(self.tokenizer, content, r_chunk_size, **r_opts)` directly (`pipeline.py:2317-2328`). There is no `asyncio.to_thread` and no executor. The `V` branch eleven lines below (`pipeline.py:2340`) is `await chunking_by_semantic_vector(...)`, so the asymmetry is visible inside a single `if`/`elif` chain. Whatever the recursive chunker spends is spent on the only thread that serves HTTP.
The amplifier, which is the non-obvious part. A long separator list alone would not be expensive if each level actually split the text. It does not.
`RecursiveCharacterTextSplitter`'s `keep_separator` default is `True` (verified below). With `keep_separator` truthy and not `"end"`, `_split_text_with_regex_spans` (`recursive_character.py:76-95`) walks the matches and emits, for the final match, the slice `text[match.start():len(text)]`. When the separator matches exactly once, at offset 0, that final match is also the first: `first.start() > 0` is false so no prefix piece is emitted (`:78-81`), and the loop emits one piece, `text[0:len(text)]`, the entire input unchanged.
So a payload whose first character is the separator, and which contains that separator nowhere else, produces a split that is a no-op. The single piece is still over `chunk_size`, so `:247` sends it down the `else` branch and `:266` recurses with `separators[index+1:]` (computed at `:232`) and the full text. On the next level the next separator matches at offset 0 in the same way, and the descent repeats. Every level re-runs `re.search` (`:230`), `re.finditer` (`:54`) and `length_function` (`:247`), which is a whole-text `tokenizer.encode` (`:325-326`). Total work is `O(len(separators) x len(text))`.
The enabling precondition on size. `len(text)` is unbounded because `POST /documents/text` has no body-size limit. `MAX_UPLOAD_SIZE` (`config.py` default 104857600) is checked in exactly two places, `document_routes.py:2694` (pre-flight on `file.size`) and `:2766` (during the streaming write), both inside the multipart `upload_to_input_dir` handler. The JSON text route never consults it, and no size-limiting middleware exists: the only `add_middleware` calls in `lightrag_server.py` are `_RootPathNormalizationMiddleware` at `:1389` (itself conditional on a non-empty `api_prefix`) and `CORSMiddleware` at `:1409`.
The multiplier. `POST /documents/texts` accepts a `list[str]` of texts (`document_routes.py:457-460`) sharing a single `chunking` config (`:464-467`). The list has `min_length=1` and no `max_length`, so one request can carry many texts, each of which pays the full cascade. [INFERENCE] The multiplication of the measured per-text cost by the list length was not separately measured.
Impact
- A single request freezes the entire API for a minute. Against a live `lightrag-server` (uvicorn, single process, default config), a 414,221-byte `POST /documents/text` was answered `200 OK` in 0.001609 s. The next `GET /health`, a route that requires no authentication and touches no storage, took 63.120969 s. Baseline `/health` latency on the same server was 0.002194 s, 0.001523 s, 0.001288 s. That is roughly 1.5 ms to 63.1 s for a trivial GET.
- The response is not the signal. The POST returns `200` in 1.6 ms and the work happens afterwards in a background task, on the event loop. Nothing in the request/response exchange tells an operator or a rate limiter that a minute of total unavailability has just been queued.
- Cost is linear in both request-controlled knobs. Measured against the real chunker on 204,801 bytes of content: 50 separators 1.51 s, 200 separators 5.92 s, 500 separators 14.91 s, 900 separators 26.67 s. Holding separators at 900 and doubling the text: 409,601 bytes 53.59 s, 819,201 bytes 108.16 s.
- All of that work is pure waste. Every run in the sweep above produced `chunks=1`. The recursion does not split anything; it re-tokenizes the same text once per separator and then emits the input as a single chunk.
- The amplification factor is large against the honest baselines. The identical 409,601-byte text takes 1.09 s with the same `recursive_character` strategy and its default separators, and 0.06 s with the default `fixed_token` chunker. 53.59 / 1.09 is about 49x, and 53.59 / 0.06 is about 890x.
- The ceiling is set by text size, not by the separator list. The separator count is capped in practice by Python's recursion limit: 2000 separators against 8,193 bytes of content raised `RecursionError: maximum recursion depth exceeded` after 1.21 s. An attacker simply stays under it at 900, and scales `len(text)` instead, which nothing bounds. [INFERENCE] 8 MB of text at 900 separators extrapolates from the measured 819,201-byte / 108.16 s point to roughly 18 minutes of frozen API per request; that specific figure was not measured.
Proof of Concept
Environment: LightRAG 1.5.5 at commit `79b54271`, default configuration, no environment variables set, single-process uvicorn.
The request
import json, requests
text = "Q" + ("X " * (400*512)) # ~400 KiB, first char is the separator
requests.post("http://TARGET:9621/documents/text", json={
"text": text,
"file_source": "sep-bomb.txt",
"chunking": {"strategy": "recursive_character",
"params": {"separators": ["Q"] * 900}}, # 900 < recursionlimit 1000
})`"Q"` appears exactly once in the payload, at offset 0. That is what makes every level of the descent a no-op.
The request model and the resolver accept it
Real `InsertTextRequest` and real `_resolve_text_chunking`:
model validated OK; separators len = 900
process_options = R
resolved separators len = 900
resolved size/overlap = 1200 100
100k separators accepted: 100000The splitter defaults that make the split a no-op
keep_separator default = True
is_separator_regex default = FalseReal chunker, cost linear in both knobs, one chunk every time
separators= 50 content= 204801B wall= 1.51s chunks=1
separators= 200 content= 204801B wall= 5.92s chunks=1
separators= 500 content= 204801B wall= 14.91s chunks=1
separators= 900 content= 204801B wall= 26.67s chunks=1
content x2 (400KiB) separators= 900 content= 409601B wall= 53.59s chunks=1
content x4 (800KiB) separators= 900 content= 819201B wall= 108.16s chunks=1
over recursion limit separators=2000 content= 8193B wall= 1.21s RecursionError: maximum recursion depth exceeded`chunks=1` on every row is the point: the entire cost buys nothing.
Baseline for the identical 400 KiB text
default fixed_token: 0.06s chunks=187
recursive default separators: 1.09s chunks=18753.59 / 1.09 is about 49x against the same strategy with default separators, and 53.59 / 0.06 is about 890x against the default chunker. Both baselines produce 187 real chunks; the attack run produces 1.
Live server, total API unavailability
`/health` needs no authentication and touches no storage, so its latency measures the event loop and nothing else.
== baseline /health latency ==
0.002194s
0.001523s
0.001288s
== firing 414KB sep-bomb ==
POST /documents/text -> HTTP 200 in 0.001609s
== /health latency during background chunking ==
attempt 1: 63.120969s (http 200)
attempt 2: 0.001480s (http 200)
...One 414,221-byte POST, answered `200 OK` in 1.6 ms, made an unrelated trivial GET take 63.12 s. Latency returns to baseline once the chunking finishes, so the outage is exactly as long as the burn.
Recommended fix
Two independent defects. Both are worth fixing, and fix 2 is the one that matters most.
- Bound the cascade. In `RecursiveCharacterChunkParams` (`document_routes.py:294`), give the field a length bound and a per-element bound, for example `separators: Optional[list[str]] = Field(default=None, max_length=16)` with a per-element `max_length`. This is the same kind of declarative bound the neighbouring fields already carry (`chunk_overlap_token_size: Optional[int] = Field(default=None, ge=0)` at `:272`), so it needs no new validation machinery. Independently, add a recursion-depth counter to `_split_text_with_spans` (`recursive_character.py:210`) that stops descending past a fixed depth and falls through to `final_chunks.append(split)`, the branch that already exists at `:263` for the exhausted-separator case. That makes the chunker safe against any separator list, including ones assembled internally or supplied through `addon_params` rather than through the HTTP model. A cheaper variant of the same guard: if a recursion level produces a single piece byte-identical to its input, the separator did not split anything, so recursing on it with the same text cannot help.
- Take chunking off the event loop. Move every synchronous chunker call in `process_single_document` behind `await asyncio.to_thread(...)`: `pipeline.py:2301` (`chunking_by_paragraph_semantic`), `:2323` (`chunking_by_recursive_character`), `:2361` (`chunking_by_fixed_token`) and `:2409` (`self.chunking_func`, which is caller-supplied and therefore arbitrary). This is not a new pattern for this codebase, it is a pattern already applied to one sibling and not the others. `GHSA-32jh-39m7-8x84` quotes `await asyncio.to_thread(_semantic_groups_with_spans, splitter, content)`: the semantic strategy is already offloaded, and the same asymmetry is visible in `pipeline.py` itself, where the `V` branch at `:2340` is awaited while the `R`, `P` and `F` branches around it are not. The recursive strategy simply was not given the same treatment.
Fix 2 alone downgrades this from "the whole API is down" to "one ingest worker is busy": the pathological chunking still burns a CPU for a minute, but `/health`, `/query`, `/documents` and every other route keep answering. Fix 1 alone removes this particular payload but leaves the structural hazard, because any future expensive chunking path on those four call sites will stall the loop again.
References
- Version tested: 1.5.5, api version 0316, commit `79b542714732a0396ae2652bc93f93902fd208b2`
- Request model and validation: `lightrag/api/routers/document_routes.py:293-294`, `:457-467`, `:1887`, `:1893`, `:1937`
- Route and task: `lightrag/api/routers/document_routes.py:2837`, `:2899`, `:2906-2918`, `:1965`
- Event loop: `lightrag/pipeline.py:2120`, `:2301`, `:2317-2328`, `:2340`, `:2361`, `:2409`
- Chunker: `lightrag/chunker/recursive_character.py:54`, `:76-95`, `:210-289` (recursion at `:266`, `length_function` at `:247`, `re.search` at `:230`, `re.escape` at `:226` and `:235`), `:325-326`, `:344`
- No body-size limit on the JSON text route: `lightrag/api/routers/document_routes.py:2694`, `:2766` (multipart only), `lightrag/api/lightrag_server.py:1389`, `:1409` (the complete middleware set)
Distinct from `GHSA-32jh-39m7-8x84` on four axes:
- Different field. That advisory concerns `sentence_split_regex`, a `str` field of `SemanticVectorChunkParams`. This concerns `separators`, a `list[str]` field of `RecursiveCharacterChunkParams`.
- Different chunker. That advisory concerns the semantic strategy, fed to LangChain's `SemanticChunker`. This concerns the `recursive_character` strategy and LightRAG's own `_split_text_with_spans`.
- Different mechanism, and specifically not a regex bug. Every separator is passed through `re.escape` before use, at `recursive_character.py:226` and `:235`, because `is_separator_regex` is `False` and is not exposed through the request model. The compiled pattern is a literal and cannot backtrack. The cost here is a missing bound on the length of the separator cascade interacting with a no-op split under `keep_separator=True`, not catastrophic backtracking.
- Different blast radius, from that advisory's own text. `GHSA-32jh-39m7-8x84` quotes `await asyncio.to_thread(_semantic_groups_with_spans, splitter, content)`, i.e. the path it describes is offloaded off the event loop. The path described here is not, which is why a `/health` request that requires no authentication and touches no storage is blocked for 63 s.
The fixes are disjoint: bounding a list length and adding `asyncio.to_thread` at four call sites has no overlap with anything that advisory asks for.
Adjacent advisories that this report does not touch and does not re-claim: `GHSA-mmg5-8x8q-v934` (default `WHITELIST_PATHS` exempting `/api/*`), which is only cited here as the reason the route is reachable pre-authentication in the default profile, and `GHSA-vv3m-f8x4-7377` (IPv6-transition destination bypass in the native markdown image download guard), which concerns destination filtering rather than request size, timeouts or volume.
Incluye detalles técnicos publicados por el investigador. Úselos solo para validar y corregir sus propios sistemas.
Desglose de métricas
Rangos afectados
Estado del parche por paquete
| Paquete / producto | Estado | Corregida en | Desde divulgación | Última versión |
|---|---|---|---|---|
pypilightrag-hku |
Corregida | 1.5.6 | -3 d | 1.5.7 no afectada |
1.5.6Arsenal público
Sin exploits, módulos ni PoC públicos conocidos a la fecha de verificación.
https://github.com/HKUDS/LightRAG/security/advisories/GHSA-26pm-px5v-8c4wLas referencias externas se muestran para copiar: cve.llama.ec no lo redirige a otros sitios.
