Master's Project · M.S. Computer Science · California State University, Dominguez Hills
Can an AI tell when its evidence isn't enough, and decide what to do about it?
Early sepsis prediction on real ICU stays. The system predicts, orders a lab group, or escalates when it is not confident. Every number below comes from one frozen protocol evaluated once on held-out patients.
Findings
- A calibrated model trained to work with missing labs reaches early-warning utility , in the range of strong PhysioNet 2019 entries.
- Knowing when it is unsure works: escalating of hours captures of all sepsis hours.
- Recorded labs add real value ( utility), but no deployable one-step lab-ordering policy recovers it; even an oracle gains only .
01 · The data
Most of the evidence is missing
PhysioNet / CinC Challenge 2019: hourly ICU records from two hospitals. Vital signs are usually there; laboratory results almost never are.
Share of ICU hours with no value
Patient-level splits
Stratified by hospital and sepsis status, seed 42. No patient appears in two splits.
Sepsis label
The Challenge marks each hour positive from 6 h before clinical onset. A useful alarm fires in the window 12 h before to 3 h after that onset; the official utility score rewards exactly that timing.
02 · Method
A pipeline built to avoid fooling ourselves
Leakage-safe at every step, with all thresholds chosen on validation data and sealed before the held-out test.
1 · Predict
Mask-aware XGBoost (26 labs, randomly hidden during training) with Platt calibration per missingness state.
2 · Decide or escalate
Make a call only when confidence max(p, 1−p) ≥ ; otherwise escalate. Chosen on validation for ≤ 1% risk.
3 · Acquire
For escalated hours, reveal one clinical lab group that was really recorded: fixed order, random, a learned patient-specific selector, or an oracle ceiling.
Two operating points, both frozen on validation
Early-warning alarm when risk ≥ (labs hidden and policies) or ≥ (labs as recorded), each chosen to maximise validation utility. The confidence rule above governs predict vs escalate. They answer different questions and are reported separately.
03 · Prediction quality
Labs help the model rank risk; calibration is good where most patients are
Held-out test, all hourly rows. Same model, with labs hidden versus as recorded.
ROC curve
Calibration (reliability)
Log scales. On the diagonal = predicted risk matches observed rate. Points are fixed probability bins with ≥ 50 rows.
04 · Early warning
Utility : most septic patients are flagged before onset
Official PhysioNet 2019 normalized utility: 0 = never alarm, 1 = perfect timing. The alarm threshold was chosen on validation (left) and applied unchanged to test (right).
Choosing the alarm threshold (validation)
Dots mark the frozen thresholds. A fixed 0.5 cut-off (our first version) scored ≈ 0.02.
Held-out test
05 · Knowing when it is unsure
Quiet nine hours in ten; the tenth holds half the sepsis
Selective prediction with labs hidden. Hover the curve to move along the confidence threshold; the dot is the frozen operating point.
Risk vs coverage
Risk = error rate among hours where the model makes a call. Dashed: predicting “no sepsis” everywhere.
What escalation catches
06 · The twist
The labs matter. Choosing which one to order doesn't, yet.
Each policy orders one clinical lab group for escalated hours. Differences versus “all labs hidden”, with 95% patient-bootstrap intervals.
Δ early-warning utility
Δ AUROC
Reading it
The gap between the oracle (+) and labs as recorded (+) is the open problem: the value of lab information exists, but it is spread across many hours and a single one-step acquisition cannot capture it.
07 · Live demo
Replay a real held-out patient, hour by hour
Outputs of the frozen model on patients it never saw. Pick a case, step through the stay, order a lab group when the AI is unsure, then reveal the true onset.
08 · Integrity & limits
Sealed before we looked, and honest about what it is not
Every opening of the held-out test set
Entries 1–4 were development-phase accesses, disclosed. Entry 5 produced every number in this dashboard.
Frozen protocol
Limitations
- Retrospective: “ordering” a lab reveals a value that was really recorded; we cannot know results of tests that were never ordered.
- One public dataset, two hospitals; random patient split, no external site.
- Calibration drifts above ~20% predicted risk (few rows; over-prediction).
- Lead time is measured from the first alarm in the stay, an optimistic upper bound.
- One-step, one-group acquisition only; timing and multi-step policies are future work.