Trial Physics — live demo · Sixty prototypes, two engines

Live demo

Two onboarding flows. Same conversion rate, different physics.

An 1,800-trial book with real start dates, so 520 of these trials are still running right now. Every hazard, curve, and p-value below is computed in your browser from the per-trial records, with in-flight trials censored at their current day rather than counted as failures.

Conversion hazard by day of trialDecisive day: 13 and 3of those still undecided at the start of the day, this many converted during it

Control peaks on

Day 13

8.7% hazard, 86 conversions

Guided peaks on

Day 3

10.5% hazard, 31 conversions

Converted by day 4

9.4%

control, share of its conversions

Converted by day 4

81.3%

guided, share of its conversions

Control is not converting anyone through the product. It is converting them with the expiry email on day 13, which is a deadline making the decision. Guided setup resolves on day 3, which is the product making it. Those two flows want opposite onboarding sequences and the end-of-trial percentage says they are the same.

Comparing variants mid-flight

The completed-trials test cannot answer. The log-rank test can.

Guided setup shipped eighteen days ago to half the traffic. Only 100 of its 360 trials have finished, which is all the completed-trials-only calculation is allowed to look at. The survival estimate uses all 360.

Two ways to compare the same two variantsboth computed here, live

Cannot concludeCompleted trials only

p = 0.103

Control 28.3% on 1180 finished trials

Guided 36.0% on 100 finished trials

Difference 7.7 pts, z = -1.63, and 520 trials discarded

DifferentLog-rank on every trial

p = 2.2e-11

Control onboarding: 351 conversions against 394 expected, 0.89×

Guided setup: 91 conversions against 48 expected, 1.88×

χ² = 44.8, in-flight trials censored at their current day

Cumulative conversion, Kaplan-Meiershaded bands are 95% Greenwood intervals

Converted by day 3

1.5% / 17.5%

control / guided

Converted by day 7

4.3% / 27.4%

control / guided

Converted by day 14

28.2% / 32.0%

control / guided

Trials still running

520

used as evidence, not dropped

By day 7 the guided flow has converted 27.4% of its trials and control has converted 4.3%. By day 14 the two intervals overlap, so this is not a claim that guided converts more. It is a claim that guided decides seven days earlier, which is a different and more useful fact.

Three numbers, one question

Your dashboard is reading 24.6%. It is 30.1%.

520 of 1800 trials have not finished. Counting them in the denominator without their future conversions understates the rate. Removing them throws away the users the dashboard exists to report on. Censoring keeps them at their current day, which is the only option that uses what they have actually told you.

Variant	Trials	In flight	Live rate	Completed only	Kaplan-Meier
Control onboarding	1440	260 (18.1%)	24.4%	28.3% on 1180	28.2%
Guided setup	360	260 (72.2%)	25.3%	36.0% on 100	32.0%
Everything	1800	520 (28.9%)	24.6%	28.9% on 1280	30.1%

Live rate understates by

5.5 pts

across the whole book

Data completed-only discards

28.9%

520 trials, and 72.2% of the new variant

Data the survival estimate uses

100%

every trial, censored at its current day

The gap between the live rate and the real one scales with how fast you are growing, because growth is what puts trials in flight. So the number is least trustworthy in exactly the week you are watching it hardest, and it recovers on its own the moment signups flatten, which reads as a conversion problem that fixed itself.