Inspect binary probability error with a small example

A worked example using a hypothetical scenario to show how the reasoning applies.

The scenario

The teaching tool accepts probability and outcome pairs and computes their mean squared probability error, commonly called the binary Brier score. Lower is better for the same task and sample. A few entered rows are a demonstration, not enough evidence to declare a useful racing model.

What to take from it

Do not compare scores computed over different populations without explaining the difference.

What to check next

• Freeze the test period before changing the model.
• Store probabilities with their publication timestamp and outcome definition.
• Report the baseline and the full evaluation population.

Keep the conclusion specific

Separate the observation, the interpretation and the unresolved question. A worksheet is most useful when its conclusion stays within the evidence entered into it.

No model is trained or advertised as profitable, and no live predictive service is supplied.