Technical appendix

The full benchmark, the full audit, and every number behind it.

This page is the complete evaluation record. Everything on it is read at runtime from assets/site-data.json, which src/build_site_data.py generates from the committed artifacts in results/.

01 / AUA 2027 Proposed Research Amendment

Audited & Frozen

Repeated nested cross-validation across four feature tiers.

5 outer folds × 5 repetitions (25 evaluations), 3 inner folds for tuning, 9 fixed labels, N = 527 study rows (no patient identifiers in release; unit of analysis is a study row). Outer splits archived prior to fitting. Primary metric is Average Precision (AP); secondary is AUROC. Intervals are 2.5/97.5 percentiles of 2,000 paired study-row bootstrap resamples.

Model Inputs Macro AP [95% CI] Macro AUROC [95% CI] Macro F1 (0.5) Exact match (0.5) Status

Paired differences ΔAP

2,000 resamples
Comparison Tier ΔAP [95% CI]

Primary: RF Full − Clinical Core (+0.144). Key Secondary: RF Full − Legacy Reduced (+0.097). Exploratory withholding demonstrates reliance on voiding-pressure (+0.185) and storage/compliance (+0.158) feature families.

Guideline rules vs ML (BOO, males)

n = 271
Comparator Metric Estimate [95% CI]

BOOI score: pdet@Qmax − 2 × Qmax. 99 of 271 adult males have Qmax = 0. Release label definitions and rule derivation are unverified; high agreement is not external validation.

Full-procedure refit bootstrap

200 OOB refits
Quantity Mean ± SD 95% OOB Interval

Re-training Random Forest pipelines on unique in-bag samples with full nested tuning verifies the feature-set differences under refit uncertainty (ΔAP Full vs Core: +0.155 [0.082, 0.226]).

Calibration & determinism audit

545,445 predictions
Calibration variant Macro Brier Macro AP Macro AUROC
Bitwise Determinism: PASSED

Two independent clean runs from raw .mat produced max absolute difference = 0.0 across all predictions.

02 / Historical Baseline

Single-split 80/20 benchmark reproduction.

Audited reproduction of the legacy single-split result (424 train / 103 test patients; seed 42) to within 1e-16 machine precision. Retained as historical sensitivity context, not independent confirmation.

Model Metric Macro F1 Macro AUROC Subset acc. CV macro F1

Global feature importance · mean |SHAP|

Ranked for the winning model in the selected analysis. Variables withheld by the leakage audit are marked where they still appear.

Label distribution · retained annotations

Multilabel and heavily imbalanced. Several labels carry small support, which limits stable inference for the rarer diagnoses.

02 / Leakage audit

Every variable, classified and justified.

The audit runs before modelling and decides what the conservative analysis is permitted to see. Analysis B removes only the clearest label-proximal summaries — it is deliberately conservative rather than maximal.

Download the audit
VariableFamilyClassificationAnalysis AAnalysis BRationale

03 / Per-label performance

Aggregate scores hide the labels that struggle.

Macro averages flatter a model with imbalanced classes. Every retained label is published individually — including the ones where this prototype performs poorly.

04 / Figures

Every figure regenerates from the pipeline.

Produced by python src/evaluate.py under a forced headless matplotlib backend with a fixed seed. Nothing is drawn by hand.

05 / Reproducibility

Fixed seed, published protocol, one command.

The notebooks are thin presentation layers over the src/ modules rather than notebook-only logic, so results regenerate outside a notebook session.

Run it yourself

bash

python3 -m venv .venv source .venv/bin/activate pip install -r requirements.txt # regenerate processed tables, figures and results python src/evaluate.py # rebuild the data layer this site reads python src/build_site_data.py

06 / Limitations

Documented limitations.

Stated in full, because a benchmark without its boundaries is a marketing claim.

  • The public release is a tabular feature dataset, not a trace-level waveform dataset.
  • Released labels overlap and several are small, which limits stable inference for rarer diagnoses.
  • The leakage-reduced analysis is conservative, but is still not a purely primitive-variable or raw-signal benchmark.
  • The release exposes no explicit BOOI or BCI columns, so the audit relies on documented proximity rather than direct removal of every derived index.
  • No claim is made about prospective clinical performance, workflow utility, or external validity.

Artifacts