SaaS

Churn and expansion.
Beyond a score.

Churn models are among the most commonly built and least commonly used pieces of machine learning in SaaS. The model is rarely the problem. The problem is that a ranked list of at-risk accounts arrives with no instruction attached.

WHY THESE PROJECTS STALL

A customer success manager receives a list of accounts with risk scores. They already suspected most of them. For the ones they did not, the score does not say what changed, why it matters, or what to do — so the list is read once and then ignored, and six months later someone concludes that churn prediction does not work here.

What makes the difference is not accuracy but actionability. The output has to be a specific observation with a specific suggested action: this account's weekly active users dropped forty per cent after their admin left in March, three integrations have gone stale, and the renewal is in ninety days. That is something a person can act on this afternoon.

Assembling that narrative from telemetry, support history, billing and CRM notes is exactly where a language model contributes, sitting on top of a classical model that does the actual prediction. The division of labour is the same one that recurs across this site: statistics for the number, language for the explanation.

THE SIGNALS THAT ACTUALLY PREDICT

Depth, not logins

Whether the features that create the value the customer bought are being used, by how many people, and whether that is growing. Login counts are a weak proxy that survives in dashboards mainly because they are easy to compute.

Champion departure

One of the strongest predictors in most B2B products, and one that is usually visible in your own data — the admin account that stopped being used, the email that started bouncing. Detectable weeks before it becomes a renewal problem.

Support sentiment trajectory

Not ticket volume, which is ambiguous, but the direction of tone across a relationship. Language models are genuinely good at this and it is hard to compute any other way.

Expansion signals, symmetrically

The same telemetry that predicts churn predicts growth: a team hitting a plan limit, a new department appearing, usage patterns matching accounts that later upgraded. The expansion side is usually the larger commercial opportunity and is built second, if at all.

MEASURING IT HONESTLY

Churn model evaluation is unusually easy to get wrong, and two mistakes account for most of it. The first is leakage: a feature that only exists because the account was already churning — a cancellation-related support ticket, a billing status change — makes the model look excellent in backtest and useless in production. Every feature needs checking against a strict as-of date.

The second is judging the model on prediction accuracy rather than on retained revenue. The model exists to enable an intervention, so the question is whether accounts flagged and worked retain better than comparable accounts that were not. That needs a holdout — some flagged accounts deliberately not worked — which is uncomfortable to propose and is the only way to know whether any of this is working.

FROM SCORE TO ACTION

ACCOUNT SIGNAL PATH

Product telemetry

depth of use, not logins
Risk model

classical, leakage-checked
Assemble narrative

what changed, and when
Specific action

in the CRM, not a dashboard
CSM acts

or records why not
Holdout

some flagged accounts unworked

Solid: the signal path Dashed: the holdout that proves the intervention works

The narrative is what makes the score actionable. The holdout is what makes the whole thing measurable.

QUESTIONS

QUESTIONS — 3

Enough churn events to learn from, which for a low-churn enterprise product can mean years of history and a small number of positives. Where that is the case we would say so and suggest starting with rule-based signals plus narrative assembly, which delivers most of the practical value without pretending to a model the data cannot support.

Only if it arrives where they already work and tells them something they did not know, with a specific action. We build it into the CRM rather than as a dashboard, because a separate dashboard is a thing people visit twice.

It can draft it from the specific observation, which saves time and improves relevance. It should not send it — a poorly judged automated message to an at-risk account can be the thing that finishes the relationship.

Keep reading

30-minute technical call · no deck