Before You Wire an LLM Into GPU Fault Triage: AI-Node-Sentinel's Integration Points and Six Measured Failure Modes

Description and use

The prototype fails to let an LLM influence triage decisions due to deterministic stubs, limited lookup coverage, lack of per‑event isolation, broken output contracts, thin evidence, and fatal provider errors. The six measured failure modes generalize to any system attaching an LLM to an operations pipeline.

Problem

In GPU clusters, hardware failures are first signaled by XID codes in kernel logs, but manual triage is slow and costly. An LLM agent could automate reading telemetry, looking up error codes, and recommending actions, yet the open‑source AI‑Node‑Sentinel prototype fails at several wiring points before the LLM can influence decisions.

Outcome

The prototype fails to let an LLM influence triage decisions due to deterministic stubs, limited lookup coverage, lack of per‑event isolation, broken output contracts, thin evidence, and fatal provider errors. The six measured failure modes generalize to any system attaching an LLM to an operations pipeline.

Workflow

• 1. Ingest telemetry window
• 2. Extract XID events via int(event.payload["xid"])
• 3. Lookup code in static table
• 4. Apply stub triage rule (always NEEDS_HUMAN_REVIEW)
• 5. Render report
• 6. If LLM provider present, build evidence string, call provider, append raw JSON to rationale

Evidence

The triage decision tree is a deliberate stub — every event gets NEEDS_HUMAN_REVIEW with confidence 0.0.

Source

The orchestrator first computes the deterministic verdict; if a provider is present, it builds an evidence string, calls the provider, and appends the returned text to the verdict's rationale with an [LLM] prefix.

Source

The orchestrator extracts the code with int(event.payload["xid"]) — no validation, no per-event isolation.

Source

The code makes the reason plain: the protocol returns a string, the orchestrator only concatenates it, nothing ever parses the JSON, and nothing overwrites the deterministic verdict.

Source