Ship the change.
Prove the control.

Three situations, three industries.

A payer’s claims files were failing to load, and reporting did not make the missing population visible. I sampled the records, then extended the investigation. It identified unresolved claims and payments. The absence of a reliable completeness check was itself a finding.

A pharmacy safety workflow raised a tradeoff between current information and responsiveness. I worked on modernizing the supporting capability, but a safety-related product cannot be changed on faith. A new result needs evidence and specialist review; no velocity metric excuses an unsupported decision.

A business-to-business quoting product was not converting, and the team was optimizing the response. My analysis exposed an eligibility problem and a separate unit-of-measure interpretation problem. Shipping more features faster would not have answered either question.

The common thread is not caution. It is that in these systems the cost of being wrong is asymmetric: a confidently wrong result costs more than a slow right one. That asymmetry, not a preference for process, is what the playbook is built around.

The single most useful habit I know for shipping a change to a system that cannot be allowed to fail is this: run the new system on the old system's inputs, and compare the outputs field by field, before any live request touches it.

I call it offline replay. Take a large set of historical transactions — thousands of real conflicts, real claims, real quotes — and pass them through the replacement. Compare every field of the result with what the incumbent produced. The parity rate is the gate: for a safety engine I required a pre-agreed parity threshold. Where the two disagreed, we did not average the disagreement away; we classified it. Some classes were the new engine being right and the old one wrong — those became evidence. Some were defects — those were fixed before cutover. None were shipped as surprises.

For systems that are not deterministic — anything with a language model in the loop — replay becomes shadow mode: the new system runs alongside the old one on live traffic, producing results that are recorded but never acted on. You measure agreement, latency and cost while the incumbent still does the work. A quoting change I shipped ran in shadow with a bounded enterprise pilot before it was allowed to return a price.

Replay and shadow mode are not testing in the conventional sense. They are the evidence you bring to the go/no-go meeting. They turn "we believe it is safe" into "here is the parity table".

Testing is usually described as an engineering discipline, and most of it is. But at every rung there is a decision a product manager must make or the rung does not hold.

Testing responsibilities shown as a ladder; security review is last on the ladder and first on the calendar.

Explore figure labels and responsibilities

1. Unit

   Read coverage on the risky paths — exceptions, not the happy path.

   Read source context
2. Integration

   Negotiate partner sandboxes and test files; own the contract.

   Read source context
3. Regression

   Own the test data — generated, masked, production-shaped, refreshed.

   Read source context
4. Performance

   Set the latency budget (p50 / p90) and throughput target before build.

   Read source context
5. UAT

   Write the operator scenarios; sit in the room; collect the sign-off.

   Read source context
6. Replay / shadow

   Set the parity threshold; classify every disagreement before cutover.

   Read source context
7. Security

   Schedule certification before protected data crosses any boundary.

   Read source context

Unit tests belong to engineers. My job is to read coverage on the paths that carry risk. In a safety system that means the exception branches: what happens when the drug database returns nothing, when the patient record is incomplete, when two checks disagree. Coverage on the happy path is table stakes; coverage on the exceptions is where the defects live.

Integration tests are where partner reality arrives. Every external system — a state reporting endpoint, a procurement platform, a drug-data vendor's API — has a sandbox that behaves slightly differently from production, and a test file format that is almost but not quite the specification. I negotiate those environments early, because they are on the critical path and nobody else will chase them.

Automated regression depends on test data, and test data is a product-management concern as well as a technical one. At a payer we used masked, production-shaped claims to make representative test sets easier to prepare and broaden coverage. That made repeatable regression more practical than manually assembling each run.

Performance testing starts with a number the product manager helps establish before build: the responsiveness or throughput requirement. An interactive workflow and a batch process have different needs. Before one enrollment launch, I worked with engineering on a performance investigation. On a separate claims design, I asked for a proof of concept before committing; its results justified revisiting the design. The lesson is to test the assumption before the implementation makes it expensive to change.

User acceptance testing is where the operator finally touches the product, and it is the rung most often skipped under schedule pressure. I write the scenarios myself — the pharmacist verifying a prescription with a missing allergy record, the claims processor handling a rejected file, the sales advisor onboarding a customer — and I sit in the room. Free exploration follows the script, because operators find what the script did not imagine. Sign-off is written.

Offline replay and shadow mode are described above. They sit near the top of the ladder because they require everything below to be done.

Security and compliance review is last on the ladder and first on the calendar. When protected health information has to cross a boundary — to a state system, to a third-party endpoint — certification takes weeks. I schedule it in the define phase, not after the build.

A launch date is a wish. A gate is a decision with an owner. The five gates I run are the same in every industry I have worked in:

• Discover. Exit when the problem is quantified, has an owner, and has a cost per unit every partner accepts.
• Define. Exit when success metrics are agreed in writing and every partner team has signed the scope — including what is out of scope.
• Design. Exit when every exception path has an owner and a screen, and the security review is scheduled.
• Validate. Exit when the parity table, the performance results and the user acceptance testing (UAT) sign-off are in one report — and every blocked test has either been resolved or converted into a named day-one production validation with an owner.
• Dial up. Exit when 100% of traffic is on the new system and every metric is at or better than the criteria written down in Define.

The go/no-go between validation and introduction is a decision shared with the responsible functions. I have brought safety, compliance, legal and operations into that review, with the evidence and unresolved questions visible. The contribution of the review is to establish readiness for a particular scope, not to turn a date into proof.

Illustrative traffic levels, not an employer rollout schedule. Each step is held until metrics support the next one; rollback returns to the previous step.

Explore figure labels and responsibilities

1. Pilot

   PILOT · one site · day shift. Watch: throughput · errors · latency.

   Read source context
2. 10%

   10% is a sample traffic level in the illustrative staircase. Each step is held until the metrics say the next one is safe.

   Read source context
3. 25%

   25% is a sample traffic level in the illustrative staircase. The diagram shows rollback to the previous step.

   Read source context
4. 100%

   100% is the sample full-traffic step in the illustrative staircase, before its relative deadline marker. It is not an achieved employer rollout result.

   Read source context

The pilot is deliberately bounded. The point is not to substitute a small trial for a performance test, but to learn from real use while the impact of a problem remains limited. On a launch I worked on, we deliberately protected time to inspect feedback rather than maximizing initial volume.

Then expand in steps. At each step watch throughput per operator, error and exception rates, and responsiveness. Did we slow people down, create new work downstream, or exceed the agreed performance limits? Hold expansion while the evidence is uncertain. Reversibility should be designed before introduction, not improvised afterward.

Two-way doors and one-way doors matter here. Dial percentages, feature toggles, prompt scope for a language model, and agent versions deployed as immutable snapshots are all two-way doors: cheap to reverse, so decide fast. Synchronous versus asynchronous architecture, the key a safety task is attached to, cancel versus hold when a check fails — these are one-way doors, and they get the full gate. Knowing which is which is most of the judgement.

A system is not launched when the code is live. It is launched when the people who have to use it can, and the people who have to support it know how.

Training is part of the launch, not a follow-up. For a pharmacy change, I worked with partners on prototype-based walkthroughs, operator guidance and support readiness. The important point was to help people understand ordinary work and exceptions, then examine whether the training prepared them for actual use.

Customer and partner onboarding is a product. When an enterprise integration took too long to become useful, I treated that as a product problem. I worked on self-service guidance and a shared definition of readiness with service partners. Enablement mattered because a partner who cannot explain the experience will struggle to help customers adopt it.

Communication is against the criteria you set before building. Every launch I run ends with a launch announcement that reports the movement — latency, automation rate, manual-review rate, throughput — against the success criteria written down in Define. Not against whatever looks good afterwards. This is the single strongest trust mechanism I know with executives: they learn that the number you promise is the number you report.

Hypercare has runbooks. The on-call engineers get a written runbook for every new failure mode we could imagine, and the product manager owns the ones we could not.

A system can fail at the handoff while every component dashboard remains green. The operating rule is to monitor the seams as carefully as the nodes.

An assistant stopped handing callers to humans. Every component was healthy; every dashboard was green; the handoff between them had broken. The fix was mechanical: monitoring on the transfer itself, and a callback fallback so a failed handoff became a scheduled call rather than a dropped one. The lesson was general. In multi-system architectures — and every AI system is a multi-system architecture — the seams need their own metrics. Handoffs, queues, retries, the boundary where a model's output becomes an action. Instrument those, and alarm on them.

Three more operating habits that follow from the same idea:

• Every automation ships with its defect metric. An automation rate without the downstream error rate it created is half a number. When we moved prescription data entry to higher automation, quality evaluation and operator feedback accompanied the improvement.
• Reconciliation is a control, not a report. The claims investigation showed why received-versus-processed completeness mattered. Better visibility and named ownership turned a one-time investigation into an ongoing product responsibility.
• Correction of Errors, every time. Root cause, the mechanism that prevents recurrence, and the metric that would have caught it. If the answer is "we'll be more careful", the analysis is not finished.

Everything above applies to a system with a language model in it, with three additions.

First, distinguish a proposal from permission to act. I have worked on assistance in which the proposed correction needed a separate confirmation of its effect. Failure of an optional capability should not silently prevent an unrelated essential task. Graceful degradation is a product requirement, not only an infrastructure detail.

Second, evaluators are part of the product. I have paired assistance with evaluation and reviewed the traces with technical partners. The product question is what a score represents, which errors matter and what evidence supports a threshold. An additional model can assist evaluation, but it does not by itself establish correctness or safety.

Third, latency and cost are product metrics. A copilot improved through model and hot-path decisions made in product reviews against p50, p90 and cost per interaction. Those are not engineering side quests. They decide whether the operator uses the thing.

And one constant: a licensed human is the final authority on anything consequential. The agent handles the routine. The pharmacist, the clinician, the compliance officer signs. Every architecture diagram I draw has that person at the top.

• Is the problem quantified in the operator's units and in cost per unit?
• Are the success criteria and non-goals written down and signed before build?
• Is each decision classified as a two-way or one-way door?
• Does every exception path have an owner and a screen?
• Is the latency budget written before build?
• Who owns the test data, and is it production-shaped?
• Have operators run UAT scenarios and free exploration, and signed?
• What is the parity threshold, and how will disagreements be classified?
• Is security certification scheduled, not assumed?
• Does the go/no-go require one approval from each function?
• Is the pilot small, slow and watched? Is rollback rehearsed?
• Are training, standard operating procedures (SOPs) and onboarding in the launch plan with owners?
• Will the launch announcement report against the pre-set criteria?
• Are the seams instrumented, not just the nodes?
• Does every automation ship with its defect metric?

Fast is a result. Safe is a method. In the systems I build, the second is how you get the first — and how you keep it.

A release decision, made concrete.

Read the Go / No-Go sample → Go / No-Go decision