Illustrative worked example: Open slot
Worked example · 44 minutes
Back to How I build · Toolkit and glossary
Illustrative worked example · separate from the source practice and customer interviews
How I work: the illustrative waitlist example
From a problem statement to a measured launch and round again, on one invented example: Open slot, a waitlist that fills clinic appointments that would otherwise go empty, taken from 0 to 1 through the full life cycle. The same method serves any customer; in this example a slot is inventory, the waitlist is demand and a no-show is a lost sale.
Eight stages, one example: a waitlist that fills empty appointment slots goes from a problem statement to a measured launch and back round.
Discover.
Find the number before anything is built: interviews, the baseline and the problem statement, in that order. Open slot: 24 interviews, eight weeks of data, one number: 14% of slots cancelled late.
Decide.
Sort every decision by its door, name who decides, and write the one-way doors down. Open slot: three one-way doors, eight two-way, one escalated and settled in four days.
Prioritize.
Score each feature on patient value, compliance, organization goals and tech debt; set it against effort; the bands set the roadmap. Open slot: thirteen features, five P0.
Plan.
Epics to features to stories, sized in Fibonacci at each level, aligned to program increments, counted back from the launch date. Open slot: 44 stories, two increments, 31 weeks.
Build.
Two-week sprints, a burn-down that shows scope changes, demos every sprint, a tech-debt budget spent by engineering. Open slot: eight build sprints, five points added, the MVP demo on the day planned.
Launch.
One, three, a region, all; each step opens on its gate numbers; SOPs and training first; rollback rehearsed. Open slot: four steps over six weeks, one gate failure fixed before the next.
Measure.
Input, output and defect metrics, defined in discovery and instrumented before the first customer. Open slot: 63% of late-cancelled slots filled at six weeks, from 22% by phone.
Learn.
The post-launch review sends findings back to the roadmap and to the customers who were interviewed; the loop is the method. Open slot: two roadmap items; the paper list retired.
- the loop
- A problem with a number: 14% of slots cancelled late.
- Eleven decisions sorted by door.
- Thirteen features scored; five P0 made the MVP.
- Three epics, 44 stories, counted back from 23 June.
- Eight sprints; five points added, shown.
- One clinic, then three, a region, then all.
- At six weeks, 63% of late-cancelled slots filled.
- Two findings back to the roadmap; the loop closed.
Stage 1 of 8 · Discover
Four weeks of interviews turned a complaint into a number: 14% of booked slots were cancelled late or missed.
Interviews.
One at a time, never a group first: 12 patients, 8 front-desk staff, 4 clinic managers, 30 minutes each, two engineers sitting in on eight of them. The question is what they did last time, not what they want.
Captured.
Every interview becomes verbatims the same day: 310 lines, each with who said it and the situation. Nothing is paraphrased yet. AI drafts the transcript of a recording made with consent; a person checks every line kept.
Tagged.
Each verbatim gets four tags: who, moment, pain, workaround. "I'd take any slot this week" is patient, booking, waiting, phoning daily. Tags are a fixed list so that counts mean something.
Themed.
Tags cluster into nine themes; AI proposes the clusters, I merge and name them by hand. The top three by count and by severity are the candidates. "The slot goes empty while people wait" led both.
Baseline.
Eight weeks of scheduling data from two clinics: 14% of booked slots cancelled late or missed; 22% refilled by phone; 19 days median to the next slot; 40 front-desk minutes a day. No opportunity is written without its number.
Opportunity.
A theme with a number and an owner: fill the slot that opens, for the patient who has waited longest. Three were written; the other two (reminders, online rescheduling) went to the roadmap's parked list.
Roadmap item.
The opportunity becomes a roadmap item with its metric and its milestone, scored like every other item in stage 3. It carries the verbatims that produced it, so the team reads the customer, not a summary.
Back to the customer.
The 24 people interviewed hear what was built and what the number did, in one message after the post-launch review. That is the loop; without it the next interview starts cold.
Front-desk staff kept a paper waitlist and phoned down it when a slot opened, forty minutes a day, while patients waited a median nineteen days. Twenty-four interviews, one at a time, and eight weeks of scheduling data gave the problem its number: 14% of booked slots were cancelled late or missed, and 22% of those were refilled by phone.
The method at this stage
- Interview one person at a time; group sessions come after, never first.
- Capture verbatims the same day with who, moment, pain and workaround as fixed tags.
- Pull the baseline myself from the system of record: the rate, the time, the cost per unit.
- Theme the tags; keep the top three by count and by severity.
- Write an opportunity only with its number and an owner.
- Write the problem statement in one page: the number, who carries it, what it costs, what "better" measures.
- Name the people interviewed and the date they will hear back.
Stage 2 of 8 · Decide
Three decisions were one-way doors and eight were two-way, so most of them took an afternoon and three took a written review.
How it is made.
The test is one question: can we undo it in a sprint? If yes, decide now and revisit with data. If no, slow down once, write it, and never relitigate it in a corridor.
- Two-way: in the room, the same day, one line
- One-way: a one-page review, a decision record
Who decides.
Product questions: the product owner. Build questions: the engineering lead. Regulatory questions: the compliance officer. One name per decision, on the record; the sponsor settles a disagreement, never the loudest room.
- Two-way: the owner of the work
- One-way: one named decider
Who is consulted.
Everyone whose work changes, named before the decision. Each either signs or dissents in writing; silence is not consent. For two-way doors the one line names them.
- Two-way: whoever the line names
- One-way: every partner whose work changes, signed or dissenting
Escalation.
A one-way door with an unresolved disagreement goes to the sponsor within a week, with both options on one page. Open slot: consent storage (a regulatory record or the scheduling system), settled in four days.
- Two-way: none; revisited with data
- One-way: the sponsor within a week
Two-way door.
Reversible in a sprint at a cost the team can carry. Decided in the room by the owner of the work, written in one line with a date and a review date. Open slot: eight of eleven.
- message wording
- ranking order
- offer expiry
- the override in the MVP
- consent wording
- the daily report
- dial-up order
- quiet hours
One-way door.
Not reversible, or reversible only at a cost the team cannot carry. One named decider, a one-page review, the consulted partners signed or dissenting in writing. Open slot: the data model, the messaging contract and consent storage.
- the waitlist data model
- the messaging contract
- consent storage, escalated
The decision record.
Nine lines: date, the decision, door type, decider, consulted, options considered, the reason, the metric that would reverse it, the review date. Eleven records in the detail review.
The PR/FAQ came first: the press release the launch would have to earn, and the questions that would stop it. Then eleven decisions, each sorted by one question: can we undo it in a sprint? Eight could, and took an afternoon. The data model, messaging contract and consent storage could not; each got a decider, a page and a record.
AI stayed out of the ranking and went into reading replies, because a rule did one job and no rule could do the other.
Gate 1.
If a rule I can write down does the job well enough, write the rule. Ranking who gets an offer: the clinic's urgency flag, then time waiting, then distance. Three lines; no model.
- Ranking
Gate 2.
A wrong answer must be cheap to catch: a person in the loop, or a two-way door. Replies with low confidence, or any reply containing a time, go to the front desk.
Gate 3.
It must be measurable before a customer sees it: an offline set of 300 de-identified replies, labelled by two people, that the model never trains on. Target 95% intent accuracy.
Gate 4.
Cost and latency inside the product's budget, stated as numbers: p95 under 2 seconds so the slot is not lost while the reply is read; under a cent a reply.
Gate 5.
The data must be allowed: the reply text only, held 30 days, health detail never echoed back, consent already on file for texting. The compliance officer signs this gate.
- Reply reading
- no
A rule in the product.
Where a rule does the job, the rule ships: readable, testable, explainable to a front desk. Open slot's ranking is three lines of rules and a tie-break on waiting time.
- yes
AI in the product.
Where no rule can do it and the other four gates pass: reply reading, classifying "yes", "no", "yes but" and "something else", behind a human fallback and a per-clinic flag.
AI across the life cycle.
No gate, because a person decides what it produced: interview synthesis and VOC clustering, first drafts of the FAQ and user stories, test cases from acceptance criteria, SQL and error-analysis grouping. AI drafts; I decide.
- Five gates, in order.
- Ranking stopped at gate one: a rule does it.
- Reading replies passed: no rule reads "yes, but".
- Cheap to catch, measured offline, inside the budget.
- Data allowed, so AI ships behind a fallback.
- Everywhere else, AI drafts and I decide.
Who gets the offer was a ranking question, and a rule answered it in three lines: the clinic's urgency flag, then time waiting, then distance. Reading a free-text reply was not: "yes, but can I come at three?" has no rule. That one passed all five gates and shipped behind a human fallback and a flag.
The method at this stage
- Write the press release and the FAQ before anything is scheduled; the FAQ holds the questions that could stop the launch.
- List every decision the scope raises and sort each by one question: reversible in a sprint?
- Two-way doors: decide in the room, one line, a review date.
- One-way doors: a named decider, a one-page review, the consulted partners signed or dissenting, a nine-line record.
- A disagreement on a one-way door goes to the sponsor within a week with both options on one page.
- Put every AI candidate through the five gates with the engineering lead and the compliance officer; what fails becomes a rule.
Stage 3 of 8 · Prioritize
Thirteen features, four factors, one score: five P0 made the MVP, three entered by the compliance gate, two were parked.
Patient value ×4.
0 means no patient notices; 5 removes a harm or a wait for every patient it touches. Rated with the customer's owner in the room: here the clinic lead.
Compliance ×3.
A 5 is the gate: required by law or a regulator by a date, or protects consent or keeps data to the minimum necessary. It makes the feature P0 whatever the score.
Organization goals ×2.
5 moves the metric the plan is judged on this year; 0 serves no stated goal. Rated against the written plan, not a slide.
Tech debt ×1.
5 retires a system or a batch job named in the risk register; 0 adds debt. The engineering lead rates it; nobody else.
P0.
35 to 50, or a gate: the MVP, program increment 1. Open slot: F1, F2, F3, F7, F8, and F4 by decision 7.
F3 Text offer with one-tap accept.
PV 5, C 4, OG 5, TD 2: score 44; effort 13; P0. The highest score on the sheet.
F1 Cancellation detected within 2 minutes.
PV 5, C 3, OG 5, TD 4: score 43; effort 8; P0. Replaces the nightly batch.
F2 Eligibility rules: same clinic type, consent on file, not booked that week.
PV 4, C 5, OG 3, TD 2: score 39; effort 13; P0 by the gate.
F8 Opt-out and quiet hours.
PV 2, C 5, OG 2, TD 1: score 28; effort 3; P0 by the gate.
F7 Audit log of every offer and reply.
PV 1, C 5, OG 2, TD 3: score 26; effort 5; P0 by the gate.
P1.
25 to 34: increment 2, before the dial-up ends. Open slot: F5, F6, F10, and F4 until decision 7 pulled it into the MVP.
F6 Patient self-enrolment on the waitlist.
PV 4, C 4, OG 2, TD 1: score 33; effort 8; P1, built in increment 1 with the capacity left.
F4 Front-desk override and manual offer.
PV 3, C 3, OG 4, TD 1: score 30; effort 5; P1, pulled into the MVP by decision 7 for the dial-up gate.
F5 Reply reading: free-text replies classified, human fallback.
PV 3, C 3, OG 3, TD 0: score 27; effort 8; P1. The one AI feature.
F10 Texts in the patient's language.
PV 4, C 2, OG 2, TD 0: score 26; effort 8; P1, increment 2.
P2.
15 to 24: after launch, re-scored at the quarterly roadmap review. Open slot: F9 the daily report, F11 the reminder.
F11 Reminder the day before the accepted slot.
PV 3, C 1, OG 2, TD 0: score 19; effort 3; P2, increment 2.
F9 Daily report to clinic managers.
PV 1, C 1, OG 4, TD 0: score 15; effort 5; P2, built in increment 2.
Nice-to-have.
0 to 14: parked; re-scored at two quarterly reviews, then dropped. Open slot: F12 app push, F13 the cohort dashboard.
F12 App push instead of text.
PV 1, C 1, OG 1, TD 0: score 9; effort 13; nice-to-have, parked.
F13 Cohort analytics dashboard.
PV 0, C 0, OG 3, TD 0: score 6; effort 8; nice-to-have, parked.
Gate.
Three features entered by rule, not score: eligibility and consent (F2), the audit log (F7), opt-out and quiet hours (F8). The gate is shown so nobody mistakes compliance for a preference.
Effort.
The team's Fibonacci estimate for the feature at PI planning. Value and effort stay on separate axes; they are never multiplied, so a cheap trivial feature never outranks an expensive necessary one.
- M0 problem statementM1 scope signedM2 MVP demoM3 readinessM4 launchM5 review
- F1F2F3F7F8F4 · decision 7
- F5F6F10
- F9F11
- F12F13
- Four factors, weighted four, three, two, one.
- Thirteen features, each scored and sized.
- Three lines make four bands.
- Three features entered by the compliance gate.
- The bands set the roadmap; one P1 was pulled in by a recorded decision.
- Six milestones, from the problem statement to the review.
Each feature was rated 0 to 5 on patient value, compliance, organization goals and tech debt, with the clinic lead, the engineering lead and the compliance partner in the room. Three features entered the MVP by the compliance gate. The front-desk override, a P1, was pulled in because the dial-up gate needed it, and the record says so.
The method at this stage
- Rate every feature 0 to 5 on the four factors with each factor's owner in the room.
- Weight four, three, two, one; a compliance 5 is a gate and makes P0 by rule.
- Size effort in Fibonacci; keep it on its own axis.
- Bands: P0 35 and up, P1 25 to 34, P2 15 to 24, nice-to-have under 15; within a band, score over effort.
- P0 to the MVP, P1 to the next increment, P2 to the next quarter, nice-to-have parked and re-scored twice, then dropped.
- A band change is a recorded decision with its reason.
- Date the milestones from the bands and sign the roadmap.
Stage 4 of 8 · Plan
Three epics became thirteen features and 44 stories, sized in Fibonacci at each level before and after the breakdown.
Roadmap item.
One item, its metric and its milestone, from stage 3. It is broken down only after the scope is signed, so the breakdown answers a fixed question.
E1 Detect and match.
Cancellation detected within two minutes, eligibility rules, the audit log. Estimated 34 at PI planning; its features summed 26; their stories 36. The consent rule was bigger than its feature.
- F1 8 → 10 · PI 1
- F2 13 → 21 · PI 1
- F7 5 → 5 · PI 1
E2 Offer and accept.
The text offer, the override, reply reading, opt-out and quiet hours, language, the reminder. Estimated 55; features 40; stories 50. The largest epic and the one patients touch.
- F3 13 → 15 · PI 1
- F4 5 → 5 · PI 1
- F5 8 → 11 · PI 1
- F8 3 → 3 · PI 1
- F10 8 → 13 · PI 2
- F11 3 → 3 · PI 2
E3 Join and run.
Self-enrolment, the daily report, and two parked features. Estimated 34; features 34; the two built summed 19 at story level. The parked two are never broken down.
- F6 8 → 13 · PI 1
- F9 5 → 6 · PI 2
- F12 13 · Parked
- F13 8 · Parked
Features.
Each feature gets a Fibonacci estimate at PI planning, 1 to 21, and an increment. A feature over 13 is split before it is planned. Thirteen here, eleven built.
PI 1.
Sprints 1 to 4, 83 points: the MVP features plus reply reading and self-enrolment, against four sprints of 26 less 20% for tech debt. The system demo closes it.
PI 2.
Sprints 6 to 8, 43 points: language, the reminder, the report, the AI evaluation work, the dial-up tooling and the rollback switch; 19 points of stated slack for what the dial-up finds.
- Parked
Stories.
Written to one template, who, what, why, with acceptance criteria; sized 1 to 13 at sprint planning by the team, not by me. 44 stories; AI drafts the first cut from the feature; the team rewrites.
Sized twice.
The sum after breakdown sits beside the estimate before it. E2 was 55, then 40, then 50: the gap is kept, not hidden, and it calibrates the next estimate.
- Three epics, estimated before the breakdown.
- Thirteen features, summed: the epics were 26, 40 and 34.
- Two increments; two features parked unbroken.
- 44 stories, sized by the team at sprint planning.
- The gap between the sizes is kept and calibrates the next.
The signed scope broke into three epics, thirteen features and 44 stories. Each level was sized in Fibonacci before the breakdown and summed after it: the second epic was 55, then 40, then 50, and the gap stayed on the sheet. Two program increments of five sprints took the work; the two parked features were never broken down.
Counting back from Tuesday 23 June put the first interview on Monday 17 November, thirty-one weeks earlier.
- Dial-up 6
L, week of 22 June.
Launch on Tuesday 23 June: every clinic live, the launch note sent to stakeholders, the customers' message queued for after the review. Depends on stage 3's gates.
L−2, week of 8 June.
Dial-up stage 3 on Tuesday 9 June: one region, about a quarter of clinics. Depends on stage 2's gates and the quiet-hours fix being live.
L−4, week of 25 May.
Dial-up stage 2 on Tuesday 26 May: three clinics. Depends on stage 1's gates after two weeks.
L−6, week of 11 May.
Dial-up stage 1 on Tuesday 12 May: one clinic, the first live offer, every acceptance called the next day. Depends on the readiness review's go.
- Build 16
L−7, week of 4 May.
Launch readiness review on Thursday 7 May: go or no-go; the three SOPs signed on the Monday; training complete on the Wednesday; rollback rehearsed once. Depends on the SOP drafts and sprint 8.
L−9, week of 20 April.
Training sessions start; SOP drafts reviewed by the front-desk leads. Depends on the MVP demo's sign-off.
L−12, week of 30 March.
PI 2 planning on Tuesday 31 March: objectives, dependencies, risks owned; 19 points of slack stated. Depends on the PI 1 system demo.
L−13, week of 23 March.
The MVP demo to clinic leads and the sponsor on Thursday 26 March, with the sign-off to dial up given in the room. Depends on sprints 1 to 4.
L−21, week of 26 January.
Sprint 1 starts. Depends on PI planning.
L−22, week of 19 January.
PI planning on Tuesday 20 and Wednesday 21 January: objectives drafted, dependencies drawn, risks resolved, owned, accepted or mitigated in the room. Depends on the signed scope.
- Decide and plan 5
L−23, week of 12 January.
Scope signed on Friday 16 January: the PR/FAQ, the requirements, the success metrics, the non-goals; three sign-offs by name. Depends on the roadmap review.
L−24, week of 5 January.
Roadmap and prioritization review: the score sheet, the bands, the milestones. Depends on the problem statement.
- Discover 4
L−27, week of 15 December.
Problem statement signed on Friday 19 December: the number, the owner, the cost. Depends on the interviews and the baseline.
L−31, week of 17 November.
Interviews start; the baseline extract requested from the scheduling system. Depends on the discovery brief.
- Start from the launch date: Tuesday 23 June.
- Six weeks of dial-up, each step two weeks apart.
- Sixteen weeks of build, the MVP demo in week ten.
- Five weeks to decide and plan, four to discover.
- So the first interview was on 17 November.
The launch date came first, 23 June, and everything was counted back from it: six weeks of dial-up, sixteen of build with the MVP demo in week ten, five to decide and plan, four to discover. That put the first interview on 17 November. A workback plan is how a date becomes a sequence of things that must be true.
The method at this stage
- Break the signed scope into epics, features and stories; size each level in Fibonacci before the breakdown and sum it after; keep both numbers.
- Split any feature over 13 before planning.
- Plan in program increments of five sprints with every function in the room; draw dependencies and own every risk there.
- Reserve 20% of each sprint for tech debt and let engineering spend it.
- Fix the launch date, then count back: dial-up, readiness, build, decisions, discovery; each milestone names what must be true and what it depends on.
- Three design sessions with engineering before the increment starts.
- Write the pre-mortem and the risk register at PI planning.
Stage 5 of 8 · Build
Eight build sprints; the burn-down shows the five points added in sprint 2 and the MVP demo held on the day the plan set.
| sprint | remaining | added |
|---|---|---|
| S0 | 83 | |
| S1 | 59 | |
| S2 | 37 | +5 |
| S3 | 12 | |
| S4 | 0 |
Sprint 1.
24 points done of 83; velocity 26 with 20% spent on tech debt as planned. Demo on Friday 6 February to the front-desk leads: the cancellation event and the eligibility rules.
Sprint 2.
27 done; 5 added: the consent wording was reworked after the compliance review of the first text. Scope moves to 88 and the line shows it; the plan is not redrawn to hide it.
Sprint 3.
25 done; the text offer with one-tap accept demoed end to end on sample patients; the usability test with six patients changed the wording of the expiry line.
Sprint 4.
12 done and the rest of the sprint on hardening for the demo; PI 1 closed at 0 remaining. The system demo and the MVP demo on Thursday 26 March.
The scope step.
Scope lines move up, never quietly. Every addition is a two-way-door decision in the sprint note with its points; the chart is the record.
The MVP demo.
Working software on real-looking data to clinic leads and the sponsor; the sign-off to dial up is given in the room or the gap is written down. Held on the date the workback set in January.
| sprint | remaining | added |
|---|---|---|
| S6 | 22 | |
| S7 | 15 | +6 |
| S8 | 0 |
- 83 points planned for four sprints.
- Sprint one: 24 done.
- Sprint two: 27 done, 5 added and shown.
- Sprints three and four: 25 and 12, to zero.
- The MVP demo on Thursday 26 March, as planned.
- PI 2: 43 points, 6 added before the dial-up.
Two-week sprints, a demo after each, and a burn-down that shows added scope: when the compliance review reworked the consent wording in sprint 2, five points went onto the scope line. The MVP demo was held on 26 March, the date the workback set in January, and the sign-off to dial up was given in the room.
Engineers were in the room from the first interview; stakeholders signed three times and got a note every week.
- Customers
- Engineers and technology partners
- Stakeholders: clinic managers, front-desk leads, compliance, the sponsor
- 1 Discover
Interviews, 24.
Twelve patients, eight front-desk staff, four clinic managers, one at a time, thirty minutes each.
In 8 interviews.
Two engineers sat in on eight interviews and read every verbatim; they wrote the first draft of the data questions.
Half-day offsite, 12 people.
Clinic managers, front-desk leads, compliance and the sponsor: the problem, the baseline and a brainstorm of options; nothing decided that day.
- 2 Decide
Press release read by 5 patients.
The PR/FAQ's press release read aloud to five patients; two sentences rewritten because they did not recognise the problem in them.
Data model, AI test.
The engineering lead decided the data model, a one-way door, and sat the AI test's gates 1 to 4 with the product owner.
2 requirement reviews; sign-off 1.
Two reviews of the PR/FAQ and the requirements, then the first sign-off: scope, success metrics and non-goals, by name.
- 3 Prioritize
Effort estimates.
Fibonacci estimates per feature from the team, and the tech-debt ratings, which only the engineering lead gives.
Sign-off 2: the roadmap.
The score sheet and the bands reviewed; the roadmap and milestones signed, including decision 7.
- 4 Plan
3 design sessions, PI planning.
Three design sessions (events, eligibility, messaging) before PI planning; dependencies drawn and owned in the room.
PI planning (stakeholders).
Clinic leads and compliance attend the second day to own risks and confirm the dial-up order.
- 5 Build
Usability test, 6 patients.
The text flow tested on six patients in sprint 3 on sample data; the expiry line reworded.
Demos; 20% tech-debt budget.
A demo every sprint; 20% of each sprint spent on tech debt by the engineering lead's choice, reported at the demo.
Sprint demos; weekly note.
Every demo open to them; a weekly one-page note: the number, what changed, what is asked of them.
- 6 Launch
Stage-1 patients called.
Every patient who accepted an offer in the first week was called the next day: eleven calls, two findings.
Rollback switch; on-call.
The rollback switch built and rehearsed; an engineer on call for each dial-up step with the front desk's number.
Sign-off 3: go or no-go; SOPs; training.
The readiness review's go; the three SOPs signed by the front-desk leads; training delivered before each clinic's step.
- 7 Measure
Reasons captured.
Every expired or declined offer records a reason, themed monthly; it feeds the second loop of the flywheel.
Instrumentation owned.
Every offer, acceptance, expiry and reason is an event the team owns; the metrics tree is instrumented before stage 1.
Monthly business review.
The metrics tree, the gate numbers and the defects, once a month, with the decision log.
- 8 Learn
Told what changed.
The 24 people interviewed hear the result in one message after the post-launch review.
Post-launch review (engineers).
Engineers present the defect metrics and the tech-debt ledger; two items go to the roadmap.
Post-launch review (stakeholders).
At six weeks: what the metrics say, what was wrong, what goes back to the roadmap; the paper list retired.
- Discover: customers, engineers and stakeholders, from day one.
- Decide: two reviews and the first sign-off.
- Prioritize: the team sizes, the stakeholders sign.
- Plan: design sessions, then everyone at PI planning.
- Build: a demo every sprint, a note every week.
- Launch: the third sign-off, SOPs, training, on-call.
- Measure: owned instrumentation, a monthly review.
- Learn: one review, two findings, the customers told.
Two engineers sat in on eight interviews and wrote the first data questions; the engineering lead decided the data model and sat the AI test; the team sized every feature and spent a fifth of every sprint on debt. Stakeholders had a half-day offsite, two requirement reviews, three sign-offs, a demo every sprint and a one-page note every week.
The method at this stage
- Two-week sprints; no story enters without acceptance criteria.
- A demo at the end of every sprint, open to every stakeholder.
- The burn-down shows scope added as a step up; every addition is a recorded two-way-door decision.
- 20% of each sprint is the tech-debt budget, spent by the engineering lead and reported at the demo.
- A one-page weekly note to stakeholders: the number, what changed, what is asked.
- Engineers in discovery, design sessions before the increment, and on call for the dial-up.
- Customers in the build: a usability test on the flow before the MVP demo.
- The MVP demo is working software for those who sign the dial-up.
Stage 6 of 8 · Launch
One clinic, then three, then a region, then all: each step opened on its numbers, with the paper list one switch away.
Readiness review.
Go or no-go on the gate numbers from the MVP's test runs, the SOPs signed, training complete, the rollback rehearsed, on-call named. Thursday 7 May; the go was given with one condition: the quiet-hours test added.
SOPs.
Three standard operating procedures for the front desk: an offer was sent and the patient calls; a patient reports a wrong offer; switch to the paper list. One page each, signed by the front-desk leads.
Training.
Two 45-minute sessions per clinic, a one-page quick card, a ten-question check; every front-desk member trained before their clinic's step.
Gates.
Four numbers, the same at every step: wrong-offer rate under 0.5%, double bookings zero, messages inside quiet hours or to an opted-out patient zero, support contacts within the budget. A miss stops the ladder.
Stage 1.
One clinic, Tuesday 12 May: the first live offer. Every acceptance called the next day. Two weeks on the numbers before stage 2.
Stage 2.
Three clinics, 26 May. One message went out inside quiet hours: a time-zone defect. Counted as a gate failure, fixed in sprint 9, re-tested, then stage 3.
- 1 breach · fixed in sprint 9
Stage 3.
One region, about a quarter of clinics, 9 June. Support contacts stayed within budget; the override was used four times, each one reviewed.
All.
Every clinic, Tuesday 23 June, with the launch note to stakeholders and the customers' message queued for after the review.
Rollback.
The paper list stays, and the switch back to it is rehearsed before stage 1. A two-way door is only real if someone has walked back through it.
- SOPs, training and the readiness review come first.
- The paper list stays one rehearsed switch away.
- One clinic; two weeks on the gate numbers.
- Three clinics; one breach, fixed before the next step.
- A region, then all, on Tuesday 23 June.
Nothing went live before three SOPs were signed, every front desk trained and the switch back to the paper list rehearsed. One clinic on 12 May, three on 26 May, a region on 9 June, all on 23 June, each opened on four gate numbers. Stage 2 breached one, a message inside quiet hours, fixed and re-tested before stage 3.
The method at this stage
- Write the SOPs with the people who will run them and have them sign.
- Train every operator before their site's step; a quick card and a check, not a deck.
- One readiness review with the gate numbers, the SOPs, the training and the rehearsed rollback; a written go or no-go.
- Dial up in steps two weeks apart: one site, a few, a region, all; the same gates at every step; a miss stops the ladder.
- Call the first customers the next day.
- Keep the old way one switch away until the post-launch review retires it.
Stage 7 of 8 · Measure
Six weeks after launch, 63% of late-cancelled slots were filled, against 22% by phone before, and the defect lines held after one fix.
- Outcome
Days to appointment.
Median days from a request to the next available slot; the outcome the whole tree serves. Baseline 19, target 16, 16 at six weeks.
Outputs.
What the customer and the clinic feel; reviewed monthly; the target set against the baseline. Open slot: slots filled, front-desk minutes.
Late-cancelled slots filled.
The share of slots cancelled late or missed that were filled from the waitlist. Baseline 22% by phone, target 60%, 63% at six weeks.
Front-desk minutes a day.
Minutes a day spent on the waitlist per clinic, measured by the front-desk leads over a week. Baseline 40, target 10, 9 at six weeks.
Inputs.
What the team can move this week; reviewed weekly; each one named with its target in discovery. Open slot: offer speed, coverage, acceptance.
Offer within 2 minutes.
The share of late cancellations with an offer sent within two minutes; the batch job it replaced ran nightly. Target 95%, 97% at six weeks.
Coverage.
Slot types with three or more eligible patients waiting; the input the team pushes. No baseline, because it did not exist; target 80%; 82% at six weeks. Under three, offers expire.
Acceptance.
The share of offers accepted before they expire. Target 35%, 38% at six weeks; it rises with coverage and falls when the expiry is too short.
Defects.
What must stay at zero or under a line whatever the outputs do; reviewed daily during the dial-up, weekly after; one breach is a gate failure.
Wrong-offer.
Offers to a patient the rules should have excluded, found by the audit log and the front desk. Line: under 0.5%; 0.3% at six weeks.
Double bookings.
A slot offered to two patients or booked twice. Line: zero; zero at six weeks, held by the lock on the slot while an offer is open.
Quiet-hours breaches.
Messages sent inside quiet hours or to an opted-out patient. Line: zero; one at stage 2 from a time-zone defect, then zero.
Delivery failures.
Texts that did not reach the phone. Line: under 2%; 1.1% at six weeks; each failure falls back to a call by the front desk.
- Three inputs the team can move each week.
- Two outputs the customer feels: fill and minutes.
- The outcome: three days sooner, from 19 to 16.
- Four defect lines that held, one after a fix.
The tree was written in discovery and instrumented before the first clinic went live, so launch day had numbers. At six weeks, 63% of late-cancelled slots were filled, against 60% targeted and 22% before; the median wait fell from 19 to 16 days; forty front-desk minutes became nine; and four defect lines held, one after a fix.
Filled slots shorten waits, shorter waits grow the waitlist, and coverage is the one input that turns the wheel.
Slots filled.
A late-cancelled slot taken from the waitlist. Each one is a patient seen sooner and a slot that earns; this is where the wheel's energy comes from.
Shorter waits.
The median wait falls as slots fill; the clinic notices within a month and the front desk starts telling patients about the list.
More patients join the waitlist.
The text invites them, the front desk recommends it, and self-enrolment makes it one form; the list grows as the wait shrinks.
Coverage rises.
Slot types with three or more eligible patients waiting; the input the team pushes. When it holds, an offer is accepted within one round; below it, offers expire. Weekly question: which slot types are under three, and why.
Offers accepted in one round.
With coverage, the first or second offer is taken, the slot is filled before the appointment, and the wheel turns again.
The weekly note.
Coverage leads it every week, with the slot types under three named and the reason for each.
Expired or declined offers record a reason.
Every expired or declined offer records a reason: too soon, wrong site, wrong time, already seen. The reasons are the second loop's input.
Reasons themed monthly.
The reasons are themed once a month the way the interviews were, by count and by severity.
Fixes go upstream.
A theme becomes a roadmap item: a reminder the day before, a shorter expiry at peak hours, a site preference on the waitlist form.
Fewer late cancellations, fewer expired offers.
The upstream fixes reduce what the first loop has to repair; both loops turn faster.
A flywheel is set up in three acts: name the one input you will push, instrument the loop before launch, and put the number in the weekly note so the loop is watched, not assumed. For Open slot the input was coverage: slot types with three or more eligible patients waiting. A second loop turned every expired offer's reason into a fix upstream.
The reply classifier missed its 95% target in week one; the error analysis fixed the data and a guardrail, not the model.
Offline evaluation set.
300 de-identified replies, labelled by two people, disagreements settled by a third; the model never trains on it. Target 95% intent accuracy; 94% before launch, 96% after the fix.
Online metrics.
A weekly sample re-labelled for accuracy; override rate, target under 10%: 12% in week one, 7% at six weeks; cost per thousand replies; p95 latency 1.4 s against 2; a safety check for echoed health detail: zero incidents.
Goal met? Each metric against its target, weekly, in the note.
One miss sends the loop round; the fallback keeps serving while it does.
- yes
Hold and keep watching.
When the goals hold, the loop still runs weekly on the sample; a drop sends it round again.
- no
Error analysis.
Every miss in the week grouped by kind. Week one: 70% were "yes, but" replies carrying a time, which the model read as a plain yes.
The fix.
Cheapest first: data (40 "yes, but" examples added), prompt (the instruction names the case), model (larger or newer), guardrail (any reply containing a time goes to a person). Week one took data and a guardrail; the model and the prompt were untouched.
Re-evaluate offline.
The offline set again before anything ships: 94% to 96%, and the "yes, but" group from 61% to 93%.
Ship through a two-way door.
Behind a flag per clinic, one clinic first; if the online number drops, the flag goes back and the fallback carries every reply. Rollback is a switch, rehearsed.
- Measured offline first, then online every week.
- Week one: 91% online, against 95.
- Seventy percent of misses were "yes, but".
- Data and a guardrail; the model untouched.
- Offline again: 94 to 96.
- Shipped behind a flag, one clinic first.
The reply classifier was measured on 300 labelled replies before launch, then online weekly: accuracy, override rate, cost, latency, safety. Week one came in at 91% against 95. Seventy percent of misses were "yes, but" replies carrying a time; forty examples and a guardrail fixed it, offline accuracy rose from 94 to 96, and it shipped again behind a flag.
The method at this stage
- Write the metrics tree in discovery: one outcome, the outputs that move it, the inputs the team can move this week, the defect lines that must hold.
- Instrument every metric before the first customer.
- Name the one input you push and put it in the weekly note.
- Review inputs weekly, outputs monthly, defects daily during a dial-up.
- For AI: an offline set the model never trains on; online accuracy, override rate, cost, latency and safety every week.
- A miss goes round the loop: error analysis, the cheapest fix, re-evaluation, a two-way-door ship with rollback.
- Theme the reasons behind every failure monthly and send the fixes upstream.
Stage 8 of 8 · Learn
The post-launch review sent two findings back to the roadmap and retired the paper list, and the people interviewed heard what changed.
On 6 August the review read the tree, the gates and the defect log and sent two findings to the roadmap: a shorter offer expiry at peak hours, and cancellation reasons captured when a patient cancels. The paper list was retired on 10 August, and the 24 people interviewed got one message saying what their words had become.
| What | When | Who | What it decides |
|---|---|---|---|
| Sprint demo | every two weeks | the team and every stakeholder | what ships next sprint |
| Weekly note | every week | stakeholders | the number, what changed, what is asked |
| Monthly business review | monthly | clinic managers, compliance, the sponsor | the tree, the gates, the defects, the decision log |
| Quarterly roadmap review | quarterly | the same, with engineering | re-scoring P2 and parked items |
| Post-launch review | six weeks after launch | everyone above and the customers' voice | findings back to the roadmap; deprecation |
| Back to the customer | after the post-launch review | the people interviewed | what their words became |
The method at this stage
- Hold the post-launch review at six weeks with the tree, the gates, the defect log and the decision log on one page.
- Every finding becomes a roadmap item or a written reason why not.
- Retire the old way on a date.
- Tell the people who were interviewed what their words became.
- Keep the cadence: demo, weekly note, monthly review, quarterly roadmap review.
- Start the next discovery with the reasons the last loop captured.
Real work, on the case pages
The Open slot example is invented. These three are not; each is a case page with the site's own figures, from three business lines and three kinds of work.
The toolkit, by stage
- interviews one at a time
- voice of the customer, captured and tagged
- the baseline
- the problem statement
- working backwards: the press release and FAQ
- one-way and two-way doors
- the decision record
- the AI gates
- the prioritization score
- P0 to nice-to-have
- roadmap and milestones
- epics, features, user stories
- Fibonacci sizing
- PI planning
- the workback plan
- pre-mortem and risk register
- two-week sprints
- the burn-down
- the tech-debt budget
- sprint demos and the MVP demo
- design sessions
- requirement reviews and sign-offs
- the readiness review
- dial-up
- SOPs and training
- rollback
- input, output and defect metrics
- the flywheel
- offline and online AI evaluation
- the error-analysis loop
- the post-launch review
- deprecation