How to measure predictive targeting

Start with a fixed baseline

Measure predictive targeting at three levels: the ranking, the probabilities, and the effect of the action. These answer different questions. Good ranking does not prove that a campaign caused a sale.

Before testing, fix the eligible audience, the definition of success, the observation window, the selection capacity, and the baseline. Record exclusions and allow enough time for outcomes to mature.

Worked example: ranking lift

All numbers below are hypothetical teaching examples, not site results or industry benchmarks.

Suppose an untouched evaluation set contains 1,000 eligible accounts. Exactly 100 purchase within 30 days. The overall purchase rate is 10%. A model's top 100 accounts contain 30 of those purchasers.

The model concentrates purchasers in the top group. It does not establish that contacting them generated those purchases. Compare with your existing selection method at the same capacity as well.

Precision measures the share of selected cases that are positive; recall measures the share of all positives captured. Google Machine Learning Crash Course: Precision and recall ↗.

Check whether percentages mean what they say

If accounts scored near 0.70 purchase only 40% of the time, the displayed probabilities are misleading even if the ranking is useful. Group comparable predictions and compare their mean score with the observed rate; include sample counts so small groups do not look more certain than they are.

Evaluate and calibrate with records separate from model fitting. A reliability diagram helps reveal systematic overconfidence or underconfidence. scikit-learn: Probability calibration ↗.

Worked example: incremental outcomes

Now consider a separate randomized test of the action. Assign 500 eligible accounts to receive the intervention and 500 to a control group. Suppose the intervention group has 60 purchases and the control has 50, with the same 30-day window.

These are point estimates. They do not establish statistical significance. Account for uncertainty, assignment quality, contamination between groups, and the planned stopping rule before declaring success. Distinguish a two-percentage-point improvement from a 20% relative improvement.

Translate outcomes into contribution

For the hypothetical experiment, suppose each incremental purchase contributes $80 after variable fulfillment costs, and the intervention costs $500. The estimated net incremental contribution is (10 × $80) − $500 = $300, before any additional platform or implementation costs.

Use contribution rather than total revenue when evaluating whether the action pays for itself. Include discounts, extra service costs, and an appropriate observation horizon. A short experiment may miss later refunds or retention effects.

A practical reporting checklist

• Scope: audience, exclusions, dates, and outcome window.
• Ranking: selected volume, precision, recall, and comparison with the baseline.
• Probability quality: observed rates by score band and sample counts.
• Experiment: group sizes, outcomes, uncertainty, and deviations from the plan.
• Economics: incremental contribution and all relevant costs.
• Operations: missing inputs, stale scores, and behavior changes.

Preserve the evaluation definition across reporting periods. If it changes, label the change so readers do not interpret a new measurement method as a real improvement.

Sources & further reading

• Google Machine Learning Crash Course: Precision and recall ↗
• scikit-learn: Probability calibration ↗

Continue reading