Technical appendix
The full benchmark, the full audit, and every number behind it.
This page is the complete evaluation record. Everything on it is read at runtime from
assets/site-data.json, which
src/build_site_data.py generates
from the committed artifacts in results/.
01 / AUA 2027 Proposed Research Amendment
Audited & FrozenRepeated nested cross-validation across four feature tiers.
5 outer folds × 5 repetitions (25 evaluations), 3 inner folds for tuning, 9 fixed labels, N = 527 study rows (no patient identifiers in release; unit of analysis is a study row). Outer splits archived prior to fitting. Primary metric is Average Precision (AP); secondary is AUROC. Intervals are 2.5/97.5 percentiles of 2,000 paired study-row bootstrap resamples.
| Model | Inputs | Macro AP [95% CI] | Macro AUROC [95% CI] | Macro F1 (0.5) | Exact match (0.5) | Status |
|---|
Paired differences ΔAP
2,000 resamples| Comparison | Tier | ΔAP [95% CI] |
|---|
Primary: RF Full − Clinical Core (+0.144). Key Secondary: RF Full − Legacy Reduced (+0.097). Exploratory withholding demonstrates reliance on voiding-pressure (+0.185) and storage/compliance (+0.158) feature families.
Guideline rules vs ML (BOO, males)
n = 271| Comparator | Metric | Estimate [95% CI] |
|---|
BOOI score: pdet@Qmax − 2 × Qmax. 99 of 271 adult males have Qmax = 0. Release label definitions and rule derivation are unverified; high agreement is not external validation.
Full-procedure refit bootstrap
200 OOB refits| Quantity | Mean ± SD | 95% OOB Interval |
|---|
Re-training Random Forest pipelines on unique in-bag samples with full nested tuning verifies the feature-set differences under refit uncertainty (ΔAP Full vs Core: +0.155 [0.082, 0.226]).
Calibration & determinism audit
545,445 predictions| Calibration variant | Macro Brier | Macro AP | Macro AUROC |
|---|
Two independent clean runs from raw .mat produced max absolute difference = 0.0 across all predictions.
02 / Historical Baseline
Single-split 80/20 benchmark reproduction.
Audited reproduction of the legacy single-split result (424 train / 103 test patients; seed 42) to within 1e-16 machine precision. Retained as historical sensitivity context, not independent confirmation.
| Model | Metric | Macro F1 | Macro AUROC | Subset acc. | CV macro F1 |
|---|
Global feature importance · mean |SHAP|
Ranked for the winning model in the selected analysis. Variables withheld by the leakage audit are marked where they still appear.
Label distribution · retained annotations
Multilabel and heavily imbalanced. Several labels carry small support, which limits stable inference for the rarer diagnoses.
02 / Leakage audit
Every variable, classified and justified.
The audit runs before modelling and decides what the conservative analysis is permitted to see. Analysis B removes only the clearest label-proximal summaries — it is deliberately conservative rather than maximal.
| Variable | Family | Classification | Analysis A | Analysis B | Rationale |
|---|
03 / Per-label performance
Aggregate scores hide the labels that struggle.
Macro averages flatter a model with imbalanced classes. Every retained label is published individually — including the ones where this prototype performs poorly.
04 / Figures
Every figure regenerates from the pipeline.
Produced by python src/evaluate.py
under a forced headless matplotlib backend with a fixed seed. Nothing is drawn by hand.
05 / Reproducibility
Fixed seed, published protocol, one command.
The notebooks are thin presentation layers over the
src/ modules rather than
notebook-only logic, so results regenerate outside a notebook session.
python3 -m venv .venv source .venv/bin/activate pip install -r requirements.txt # regenerate processed tables, figures and results python src/evaluate.py # rebuild the data layer this site reads python src/build_site_data.py
06 / Limitations
Documented limitations.
Stated in full, because a benchmark without its boundaries is a marketing claim.
- The public release is a tabular feature dataset, not a trace-level waveform dataset.
- Released labels overlap and several are small, which limits stable inference for rarer diagnoses.
- The leakage-reduced analysis is conservative, but is still not a purely primitive-variable or raw-signal benchmark.
- The release exposes no explicit BOOI or BCI columns, so the audit relies on documented proximity rather than direct removal of every derived index.
- No claim is made about prospective clinical performance, workflow utility, or external validity.