QA Lead @ Lucid Motors · AI QA Architect
I build the missing QA layer for AI systems — evaluation harnesses, guardrails, and validation frameworks that take LLM apps from "demo works" to "production ready." 14+ years in software QA, now focused on making AI testable, measurable, and shippable.
- 🔍 LLM evaluation — golden datasets, LLM-as-judge scoring, CI release gates, production drift monitoring
- 🤖 Agent validation — trajectory scoring, tool-call correctness, multi-agent orchestration testing
- 🛡️ AI safety — guardrails, adversarial red-teaming, PII protection
- 📊 LLM observability — tracing, cost attribution, prompt-version comparison
- 🔧 Model lifecycle — fine-tuning, prompt optimization, cost-aware routing
Evaluation & quality
- ai-qa-eval-harness — LLM-as-judge harness: golden datasets, rubric scoring, CI release gates
- rag-eval-toolkit — RAG eval: chunking comparisons, retrieval precision/recall, drift detection
- deepeval-starter — DeepEval framework integration with CI threshold gates
- agent-validation-suite — Validate AI agents: tool-call correctness, plan adherence, release gates
Agents
- ai-agent-lab — ReAct agent built from scratch: tools, trajectory logging
- multi-agent-orchestrator — Planner → worker → critic architecture
- langchain-qa-agent — LangChain tool-calling agent with offline eval
- mcp-server-lab — Model Context Protocol: QA-tools server + client
Observability & ops
- langsmith-evalops — Datasets → experiments → regression detection
- langfuse-observability — LLM tracing, scores, cost tracking, dashboard
- llm-router — Cost/latency-aware routing + semantic caching
Safety
- llm-guardrails — PII redaction, jailbreak defense, policy pipeline
- llm-red-teaming — Prompt-injection attack library + defense scoring
Model lifecycle
- llm-finetune-lab — LoRA fine-tuning with before/after evals
- dspy-prompt-optimizer — Metric-driven prompt optimization, versioned registry
Open to AI QA Architect / Senior SDET / QA Lead roles — Bay Area & remote.