"I'm a pharmacist by training and an AI systems practitioner by the last decade. What makes me unusual: I've built and shipped multi-agent AI in regulated environments where citations must be verifiable — pharma FDA submissions, SaMD platforms, NIH-funded ML at 300+ hospitals.

At NewsRx I built self-hosted LLM infrastructure — vLLM, Qwen3-32B — and a four-blinded multi-agent evaluation harness as sole contributor. At Amgen: first GxP-validated GenAI for FDA regulatory submissions, multi-agent architecture, 400+ user stories, 40% drafting time reduction.

Protege lives at exactly this intersection — domain-critical AI that cannot get citations wrong. I know how to build the system and how to know when it's wrong."

Opening line: "Start with chunking strategy — it determines everything downstream."

Chunking: Semantic > fixed-size for legal docs. Preserve citation context — don't split case citations across chunks.

Retrieval: Hybrid BM25 + dense (maximize recall). Cross-encoder reranker before LLM sees results (boost precision).

Protege angle: "Your knowledge graph traversal post-retrieval for authority ranking is the right pattern — higher-court rulings over lower. I'd want to understand where the graph boundary sits in your retrieval pipeline."

Eval: RAGAS (faithfulness, answer relevancy, context recall). LLM-as-judge + human eval for novel failures.

Multi-agent workflow reliability?
Orchestrator owns state; individual agents stateless + idempotent. Explicit failure contracts — structured result with confidence + failure reason. Reflection agent reviews before delivery (same pattern Protege uses). NewsRx: 4-blinded agents + enrichment agent cross-correlating — extend that here.

Design a verifiable citation system?
Generator and verifier must be separate — same model cannot reliably verify its own output. Citation grounding: every cited case runs through retrieval verification before response finalized. Force structured output (JSON with case_id, jurisdiction, authority_level). Fallback: explicit "could not verify" — never silently omit.

Hallucination in production?
Upstream: verbatim-only constraint for citations/numerics + explicit refusal spec when source unavailable. Architecture: Shepard's pattern — independent citation verification agent. Monitoring: HHEM + BERTScore as pipeline gates, not post-hoc. NewsRx lesson: hallucinations cluster at knowledge gaps — test where data is thin.

Handle agent loops / failures?
Max step budget per agent + explicit termination conditions. Agent returns structured ToolResult with success/failure/reason — orchestrator decides retry vs. escalate vs. fallback. Context window: summarize intermediate results before passing; don't accumulate raw traces.

LLM eval methodology?
Metric set: factual grounding (primary), BERTScore (semantic), HHEM (hallucination), citation accuracy, preference rate. 27-iteration controlled experiments, 1 variable per iteration, full audit trail. RAGAS for RAG-specific (faithfulness, answer relevancy, context recall).

Self-hosted vs. API LLMs?
Self-hosted: data sovereignty, no per-token cost at scale, full control, can fine-tune — GPU infra overhead. API: speed to market, model flexibility (swap GPT-5→Claude 4 without infra change). NewsRx: vLLM + Qwen3-32B on RunPod — eliminated API costs at 950+ doc throughput. LexisNexis hybrid is correct: proprietary legal data on-prem, general AI queries via API.

Production ML end-to-end?
NewsRx: sole contributor, vLLM infra + 4-agent eval harness + PostgreSQL + Streamlit + GxP governance. 4-person staffing plan delivered alone in 8 weeks. Beyond contracted scope: DB, deployed web app, SME review system — became MVP engagement foundation. Quantified: 950+ doc throughput, 18% grounding ↑, 83% human preference, 27 iterations.

ML system drift / monitoring?
Feature drift (input distribution) vs. concept drift (label relationship) — different responses. MLflow for experiment tracking + model registry; shadow deployment before cutover. Invistics: 96% accuracy maintained across 300+ hospitals — EHR diversity was the stress test. Alert on output distribution change — embed quality metrics in inference path.

Domain you don't know?
Step 1: domain expert interviews — what does 'wrong' look like to a lawyer? Step 2: black-box exploration before reading architecture docs (surfaces failures unfiltered). Step 3: failure mode taxonomy. Amgen: told to figure out GxP-validated GenAI with no playbook. Same approach. Shipped.

Ambiguity in ML requirements?
Write expected outputs BEFORE running tests — or you'll rationalize failures as acceptable. Document assumptions explicitly in test plan. Stakeholder walkthrough BEFORE engineering gets the change list.

1 · NEWSRX — self-hosted LLM + multi-agent eval
Sole contributor. vLLM + Qwen3-32B on RunPod. 950+ doc throughput. Built 4-blinded multi-agent eval harness: agents score independently, enrichment agent cross-correlates with HHEM + BERTScore. 27 iterations, 1 variable each. Result: 18% grounding ↑ · 83% human preference · GxP audit trail.
→ Protege bridge: same multi-agent + hallucination mitigation architecture at legal AI scale.

2 · AMGEN — production GenAI in citation-critical domain
First GxP-validated GenAI at Amgen. FDA regulatory submissions — CTD Modules 3, 4, 5. Multi-agent architecture with human signatory accountability. 400+ user stories for AI behavior. Guardrail design + refusal specs. Result: 40% drafting time ↓ · Board expansion after production milestone · Production-grade, not a demo.
→ Protege bridge: legal AI and FDA regulatory AI share the same core problem — citations must be verifiable.

3 · INVISTICS — supervised ML at hospital scale
NIH SBIR $2.1M. Drug diversion ML deployed 300+ hospitals. 96% accuracy. Hybrid feature selection: machine-optimized + domain expert validated. EHR integration across Epic, Cerner, AllScripts, Meditech. Result: 6–8 months faster than manual audits · Maintained accuracy across all EHR diversity.
→ Protege bridge: production ML that ships at scale and stays accurate across data diversity.

Self-hosted LLM in production
vLLM + Qwen3-32B on RunPod. Not just API wrappers — built the infrastructure, owns the stack.

Multi-agent eval harness built from scratch
4-blinded agents + enrichment agent. Designed from requirements, not assembled from a framework.

Hallucination = liability, not a score
Pharma FDA + SaMD — citation accuracy is regulatory sign-off. Operates at the same stakes as legal AI.

96% accuracy at 300+ hospital scale
Invistics: maintained across Epic/Cerner/AllScripts/Meditech data diversity. Scale and accuracy together.

Sole contributor delivery
NewsRx: 4-person staffing plan, 8 weeks, shipped beyond contracted scope. No team. No playbook.

Production, not demos
Amgen: Board expansion triggered by production milestone. Optinosis SaMD: clinical sign-off on schedule.