Novant Health  ·  AI Technical Product Architect II  ·  Round 1 — BACK  ·  2026-05-21  ·  2:00 PM EST

NewsRx and Optinosis are both thriving — I'm not leaving due to failure. What Novant has that neither does is patient outcome stakes at health system scale AND a platform problem I've been solving variants of for years. You're scaling AI through a governed clinical data platform — that's Evernorth's problem, Invistics' problem, and the problem I find most interesting. The Institute has been running since 2019. Mayur built the data platform. The infrastructure is real. That's not common.

A model in production that clinical users trust enough to act on, that performs on the actual population it serves, and that has a feedback loop catching drift before it becomes a patient event. At Invistics: 96% accuracy, 300+ facilities, pharmacists acting on every flag, seven years running. At NewsRx: 83% preference over human-written, zero fabrications in scored output, 21 CFR Part 11 audit trail on every decision. Success is the sustained operational number, not the launch date.

You make them co-designers before they're users. At Invistics I embedded clinical pharmacists in the feedback loop from day one — they labeled edge cases, not just signed off on the final model. That co-ownership is why 300+ facilities trusted it for seven years. At Street Grace I built the survivor advisory council into the AI design process — 13,000+ predators identified in Georgia beta alone, 54,000+ messages exchanged, nine years running — because the people most affected by the system's decisions helped design its behavior. SVPG calls this empowered product teams. I call it the only approach that works in high-stakes regulated environments where trust is earned, not assumed.

At NewsRx: content quality regression when an upstream PubMed schema changed silently — our hallucination detection pipeline flagged an unusual drop in HHEM scores before any customer saw it. Root cause: undocumented field mapping assumption in the ingestion layer. Fix: contract-level validation at every source boundary, not just output scoring. That monitoring-first architecture is why we caught it in production monitoring, not in a customer complaint. The lesson: in complex pipelines, failures are assumptions not made explicit. The discipline is making them explicit before deployment.

Three layers that cannot be skipped. Data governance before model governance — you cannot govern a model built on ungoverned data. At Evernorth that meant 147 reports, a standardized data dictionary, and a governed KPI library before a single model ran. Model governance: every prediction has traceable lineage, statistical assumptions are documented and verified, bias assessment across demographics is required not optional. Deployment governance: human override is architectural, not a toggle. The licensed professional makes the call. At Invistics no drug moved without a pharmacist's eyes on the flag. That's the design.

Starting with data contract from Epic CLARITY — which tables, who owns quality validation, what are the known null-handling issues. Feature engineering in Databricks with PySpark at scale. MLflow for experiment tracking with hyperparameter tuning — I'm using MLflow now at NewsRx and at Optinosis for exactly this. Model registry in MLflow, deployment via Azure ML endpoints. Governance wrapper: drift thresholds set with clinical input before go-live, tied to clinical significance not statistical convenience. SPCC monitoring on prediction distributions in production — I'm a LEAN Six Sigma Black Belt, that's standard tooling. Alerting, logging, error handling from day one, not retrofitted. The pattern you used on Primary Aldosteronism — 30k daily predictions into Epic from CLARITY — is the reference architecture I'd build to.

With cohorts under 2,000 records, calibration beats discrimination. Standard logistic regression often outperforms gradient boosting on small clinical samples because you need the model's confidence to be trustworthy, not just its rank ordering. That's the same conclusion your Primary Aldosteronism model reached — LR outperformed XGBoost and LGBM on your <2k final cohort, with ECE 0.06 indicating good calibration. The principle: start with the simplest model that meets clinical calibration requirements. Complexity earns its way in when you have the data to support it.

SPCC — Statistical Process Control Charts on prediction distributions and performance metrics over time. Six Sigma tooling applied to ML. The drift thresholds have to be clinically meaningful: at Invistics a 2% accuracy drop on diversion detection triggered a pharmacist review queue before any automated retraining — the human-in-the-loop caught two edge cases that pure retraining would have learned wrong. Drift is a clinical safety signal, not a technical metric to optimize silently. The Model Surveillance POC approach using SPCC is the right architecture — I'd extend it with DeepEval CI/CD gates at the pipeline level and Langfuse for production tracing, which is the pattern I'm running at NewsRx.

Performance stratification across sex, race, age demographics before production approval — non-negotiable. At Invistics I validated across facility type and patient population before any hospital went live. The statistical check: not just aggregate AUC but disaggregated AUC by subgroup, and calibration error across each. If the model performs well in aggregate but poorly on a minority subgroup, you have a clinical equity problem regardless of the headline number. At Amgen I embedded bias mitigation controls into the first GxP-validated GenAI for FDA regulatory submissions at any Fortune 500 pharma — CTD Modules 3 through 5. During that engagement, Tarlatamab received FDA accelerated approval for extensive-stage small cell lung cancer (~30K US patients/year; full approval November 2025). When a GenAI error delays a drug approval, patients pay with their lives. Responsible AI in a regulatory context is not a checkbox.