Churn and expansion.
Beyond a score.

A customer success manager receives a list of accounts with risk scores. They already suspected most of them. For the ones they did not, the score does not say what changed, why it matters, or what to do — so the list is read once and then ignored, and six months later someone concludes that churn prediction does not work here.

What makes the difference is not accuracy but actionability. The output has to be a specific observation with a specific suggested action: this account's weekly active users dropped forty per cent after their admin left in March, three integrations have gone stale, and the renewal is in ninety days. That is something a person can act on this afternoon.

Assembling that narrative from telemetry, support history, billing and CRM notes is exactly where a language model contributes, sitting on top of a classical model that does the actual prediction. The division of labour is the same one that recurs across this site: statistics for the number, language for the explanation.

Churn model evaluation is unusually easy to get wrong, and two mistakes account for most of it. The first is leakage: a feature that only exists because the account was already churning — a cancellation-related support ticket, a billing status change — makes the model look excellent in backtest and useless in production. Every feature needs checking against a strict as-of date.

The second is judging the model on prediction accuracy rather than on retained revenue. The model exists to enable an intervention, so the question is whether accounts flagged and worked retain better than comparable accounts that were not. That needs a holdout — some flagged accounts deliberately not worked — which is uncomfortable to propose and is the only way to know whether any of this is working.