# Worked example: choosing an onboarding improvement

**Everything below is fictional:** the product, employer, assignment, candidate, data, permission statement and assistant suggestions. The suggestions are written teaching examples, not a transcript from a particular model. No experiment was run, no customer was contacted, and no hiring result is claimed.

The purpose is to show how a candidate can turn permitted assistance into inspectable work. The numbers are deliberately small enough to check by hand. No paid tool or model account is needed to follow the example.

## 1. The supplied assignment

You are applying for a growth role at a fictional B2B collaboration product, Lindenboard. Recommend one onboarding improvement and describe how you would test it. Submit a memo of no more than 500 words with a short decision log.

For this fictional assignment, AI may be used to brainstorm, draft, edit and check calculations. The candidate must identify its contribution and verify the final work. Only the synthetic packet below may be supplied to a tool. This permission applies to the fictional take-home; it says nothing about a later live interview.

Constraints:

- One engineer can spend at most two days on an initial change.
- Customer success can spend one hour a day on research and test support.
- The proposal must not require a production launch, email send or external contact during the assessment.
- Activation means an account creates its first shared project within seven days of signup, as recorded by the backend.

## 2. The complete evidence packet

All 140 accounts signed up during the same seven-day period. The snapshot was taken after each account had a full seven days of observation. Each account belongs to exactly one group; there are no duplicate accounts in this packet.

| Source ID | Observation |
|---|---|
| P1 | 100 accounts used standard onboarding; 20 activated within seven days. |
| P2 | 40 accounts opted into an optional setup call; 12 activated within seven days. This was not randomized. |
| P3 | Support received 18 tickets from 12 of the 140 accounts. Nine tickets concerned import permissions, six concerned CSV formatting and three concerned billing. Each ticket has one category. |
| P4 | The packet does not identify which groups those 12 support accounts belonged to, whether they activated, or how many accounts had import problems without contacting support. |
| P5 | The product has an existing onboarding email and an import screen. No open rates, email experiments, import completion logs, engineering estimate or demand forecast are supplied. |

These are synthetic observations supplied for the exercise, not evidence about a real product.

## 3. The candidate's starting note

Written before the fictional assistance step:

> The setup-call group has more activations per account, but people chose whether to take a call. I cannot attribute the difference to the call. Import issues appear in support tickets, but I do not know how widespread they are. I want to compare an import checklist with another email. My initial preference is to help at the import screen, subject to confirming that the change fits two engineering days.

The note states a position and uncertainty. It makes no claim that the candidate has already proved the preferred option.

## 4. Illustrative assistant suggestions

Suppose an assistant proposes:

> The pilot improved activation by 50%, proving that stronger onboarding drives conversion. Since 83% of users struggle with imports, roll out a three-email education sequence to all 140 accounts. At a 30% activation rate, the rollout will produce 42 activated accounts. Run it for a week, then announce the lift. An import checklist could be another option.

This is an intentionally flawed teaching example. It sounds decisive while changing the intervention, confusing tickets with users and presenting a projection as an outcome.

## 5. The decisions and checks

| ID | Suggestion | Check and result | Decision |
|---|---|---|---|
| D1 | “Improved activation by 50%.” | P1: 20/100 = 20%. P2: 12/40 = 30%. The gap is 10 percentage points; the relative difference is (30%-20%)/20% = 50%. P2 says assignment was not randomized. | The arithmetic is valid as a descriptive comparison, but the causal wording is not. Use the observed rates and state selection as an alternative explanation. |
| D2 | “83% of users struggle with imports.” | P3: (9+6)/18 = 83.3% of **tickets** concern imports. The 18 tickets came from 12 accounts; P4 gives no import-affected account count or population prevalence. | Reject the user claim. Keep “15 of 18 support tickets concerned imports,” with the limited scope. |
| D3 | “Roll out a three-email sequence.” | The observed intervention in P2 was a setup call. P5 contains no email performance evidence; no estimate establishes implementation effort. | Do not infer an email effect from call observations. Retain email as an alternative that needs its own evidence. |
| D4 | “The rollout will produce 42 activated accounts.” | 140 × 30% = 42 is arithmetic applied to an unsupported future rate. The observed total is 20+12 = 32 activations out of 140 accounts. | Reject the forecast. Report 32/140 = 22.9% as a rounded description of this mixed cohort only, if needed. Do not use it as an experimental control rate. |
| D5 | “Announce the lift after a week.” | Seven days of enrollment leave the final account without its seven-day outcome window. Volume and sample-size requirements are also unknown. | Wait seven full days after the last enrolled account for that cohort's final readout. Assess the observed uncertainty against the prespecified decision and effect size; treat results too imprecise for that decision as inconclusive. |
| D6 | “An import checklist could be another option.” | P3 gives specific permission and formatting issues. P5 confirms a place to put instructions, but not cost or effectiveness. | Accept as a testable proposal. Scope to those instructions; obtain an estimate and verify the issues before committing. |

### Arithmetic you can reproduce

```text
standard activation = 20 / 100 = 0.20 = 20%
optional-call activation = 12 / 40 = 0.30 = 30%
observed absolute gap = 30% - 20% = 10 percentage points
observed relative difference = (0.30 - 0.20) / 0.20 = 0.50 = 50%
import-ticket share = (9 + 6) / 18 = 0.8333... = 83.3%
observed total activation = (20 + 12) / (100 + 40) = 0.22857... = 22.9%
```

The relative difference does not prove causation. The import-ticket share does not estimate the share of all accounts affected. The pooled activation rate mixes two self-selected groups.

## 6. The revised submission

### Recommendation: test help at the import step

I recommend investigating a small import checklist before adding an email sequence. The support packet identifies permission and CSV-formatting problems at a specific point in setup. It does not establish how many accounts encounter them or whether they prevent activation.

Standard onboarding produced 20 activations among 100 accounts; optional setup calls produced 12 among 40. The observed rates are 20% and 30%, a 10 percentage-point gap. People selected whether to take a call, so the comparison cannot establish its effect. Nor does it establish that emails would achieve a similar result.

First, I would ask customer success to review the 15 import-related tickets and link them to accounts and eventual activation where access permits. I would ask the engineer to confirm whether a checklist covering permissions and CSV formatting fits the two-day allowance. If it does not, I would reduce scope or revisit the alternative rather than assume the estimate.

For a test, define eligible new accounts before enrollment and assign them randomly to the existing onboarding flow or the flow with the checklist, keeping the existing email and access to setup calls the same in both groups. Assign and analyze at the account level, including accounts that never use the checklist. Check that group assignment and backend activation events can be joined before launching.

The primary outcome is first shared project within seven days of signup. Track import-related support contacts as a secondary operating measure. Record any other onboarding changes or unequal access to assistance that could affect the comparison.

Seven days of enrollment would require waiting another seven days after the final signup for the complete activation readout. Before launch, use expected account volume and a decision-relevant effect size to assess whether the proposed test can answer the question; the packet supplies neither. If volume is insufficient, use the small pilot to learn about usability and instrumentation, and leave the activation-effect claim unresolved.

I would reconsider the checklist if ticket review finds that the issues are already solved, the change exceeds available capacity, or the test shows added friction. No projected activation lift is justified by this packet.

## 7. Handoff and assistance note

Proposed handoff:

- **Growth owner:** define eligibility, random assignment, measures, observation window and analysis before launch.
- **Customer-success owner:** inspect the ticket themes and identify missing account links within the available research time.
- **Engineering owner:** estimate the checklist, confirm assignment/event joins and assess unintended friction.
- **Not completed:** ticket investigation, feasibility estimate, power assessment, implementation, randomization and experiment. This example produces a recommendation only.

A disclosure for the fictional scenario could say:

> I used an AI assistant to explore onboarding options and draft an initial recommendation. I recalculated the rates, rejected the causal and user-prevalence claims, and changed the recommendation to a bounded import-checklist test. The attached decision log identifies accepted and rejected suggestions. I have not run the proposed test.

In a real submission, replace the generic tool label with the actual tool and version if available, and describe what actually happened. This example does not document a real model session. Never claim its illustrative checks or decisions as personal work you performed.

## 8. Questions for the discussion

1. Why can the 50% relative difference be numerically correct but an incorrect causal claim?
2. What account-level information would you request before estimating the prevalence of import problems?
3. Why keep access to setup calls the same in the proposed comparison?
4. What would you do if the checklist takes a week to build?
5. What can a small pilot teach you when it cannot establish an activation effect?

Worked example: GTM Vibes, version 1, October 22, 2026. Original fictional material; not a validated hiring assessment or a claim about any employer's preferred submission style.