RESEARCH DOSSIER
Proper Scoring Rules for Insurance Prediction
Abstract. Insurance prediction requires a distinction between forecasting an expected claim cost, forecasting a quantile, assigning the probability of an insured event, and specifying the entire distribution of count, severity, or aggregate loss. A scoring rule provides an operational criterion for comparing such forecasts against subsequent observations. Its mathematical propriety determines whether an informed forecaster minimizes expected penalty by reporting the actual predictive probability law; strict propriety requires uniqueness within the evaluated class. This monograph develops the foundations of proper scores, analyzes logarithmic and quadratic scores for events and count distributions, derives the continuous ranked probability score and its threshold representation, and traces their consequences for zero-inflated insurance losses, compound frequency–severity models, temporal validation, censoring, exposure, portfolio dependence, and tail-risk assessment. Explicit algebraic examples establish the divergence between proper and outcome-weighted objectives, the difference between marginal and joint evaluation, and the role of observational identifiability. The study relates predictive evaluation to pricing, reserving, risk capital, and model governance while keeping statistical forecast quality distinct from economic and regulatory decision criteria. Its original contribution consists of a source-grounded comparative synthesis, transparent derivations, falsifiable counterexamples, and an insurance-specific assessment protocol. No new theory, proprietary model, empirical portfolio result, or claim of exclusive scientific novelty is proposed. The central finding is that truthful distributional assessment depends simultaneously on the score, the forecast target, the observation mechanism, and the validation population; validity of a score in isolation does not establish validity of the operational experiment in which it is used.
Keywords: Proper scoring rules; strictly proper scores; probabilistic prediction; insurance frequency; claim severity; aggregate loss; logarithmic score; Brier score; ranked probability score; continuous ranked probability score; calibration; elicitation; forecast evaluation; compound Poisson; Tweedie; zero inflation; censoring; exposure; tail risk; predictive validation; actuarial modeling; forecast comparison.
Introduction
An insurance forecast is a statement about a specified uncertain outcome at a specified point in time. The outcome can be whether a covered event occurs within a year, how many claims one policy generates, the monetary size of a claim conditional on occurrence, the total cost for a portfolio, the probability of exceeding an attachment point, or the discounted value of a long-settlement liability. Each object is associated with an information set available when the forecast was issued and an observation process that eventually provides evidence about it. Scientific comparison of insurers' prediction methods begins by aligning these elements.
For a policy with exposure covariates X, let Y denote a future loss random variable and let P(·|X) be the predictive probability law delivered before observing Y. A model that issues only E[Y|X] is a mean predictor. Another that issues the conditional probability P(Y>u|X) answers a threshold question. Neither is, by that fact alone, a full distributional predictor. Confusing these targets makes statements about “best predictive accuracy” ambiguous: the best method under squared error of the mean need not be the best method under a distribution-sensitive proper score, and a threshold score can ignore errors elsewhere in the law.
The formal modern treatment in Gneiting and Raftery (2007) relates probability forecasts to proper rewards, convex geometry, categorical and continuous scoring rules, and statistical estimation. Earlier, Brier's probability verification study established a quadratic method for checking categorical forecasts. The extension of forecast elicitation to continuous distributions by Matheson and Winkler (1976) forms another foundational contribution. These results give actuarial prediction a defensible statistical language, but they do not by themselves determine which insurance quantity an institution must predict.
The evaluative perspective is sequential. In Dawid's prequential approach, a forecast is judged by what it said before the outcome, using information then available, rather than retrospectively fitted explanatory power. An insurer can estimate a claim model with extensive historical data and nevertheless produce a poor forecast if its validation design leaks knowledge of subsequent inflation, settlement, catastrophe events, or underwriting decisions into what is presented as a past prediction. A proper score cannot repair this leakage; it assumes that the comparison itself is defined correctly.
The study treats propriety as a mathematically testable property of a mapping from distributions and outcomes to real or extended-real penalties. It then examines the practical conditions needed to use the mapping for insurance research. Its method consists of formal derivations under explicit integrability and identifiability conditions, comparisons among verified peer-reviewed results, worked numerical counterexamples, and a reproducibility specification. The insurance-specific constructions in this monograph are analytical illustrations, not results estimated from confidential claims or a newly collected international dataset.
Scientific scope includes the evaluation of distributions for policy-year counts, individual claim amounts, annual aggregate paid or incurred amounts, survival or lapse-related outcomes when censored, and joint multi-line losses. Relevant decisions in underwriting, pricing, reserving, and solvency are discussed as applications. Calibration diagnostics, benchmark model selection, and actuarial risk-measure estimation each warrant independent investigation; they appear here only to establish the scientific meaning and limitations of a scoring-rule result.
Scoring Rule Foundations
The forecast, outcome, and observation law
Let (Ω, A) be a measurable outcome space and let P be a class of probability laws. A loss-oriented scoring rule is a mapping s(P,y) that evaluates an issued forecast P when outcome y occurs. Write L(P,Q)=E_{Y∼Q}[s(P,Y)] whenever the expectation exists. The convention adopted throughout is that smaller scores are better. Under the reward convention used in part of the foundational literature, every inequality reverses and one may put s=-S. Stating the orientation prevents a common software-independent reporting error: a model can appear to improve or deteriorate merely because competing studies use opposite signs.
A rule is proper on P when L(Q,Q)≤L(P,Q) for every pair P,Q in the admitted class. It is strictly proper when equality for a fixed Q implies P=Q as probability laws, subject to the equivalence actually identifiable from the observations. The term “strict” refers to the minimizer of expected score, not to an empirical guarantee that the true model wins on every realized test sample. Even a correctly specified model can obtain a worse finite-sample average than a competitor through random variation.
There is a precise information condition behind that statement. The event being scored must be sampled from the same conditional law whose prediction is under evaluation, and the forecast must have been fixed using the admissible pre-outcome information. In insurance the scoring unit might be a policy-year, vehicle-year, claim event, accounting year, or portfolio catastrophe aggregate. Those units have different selection and dependence structures. A metric that is proper for claims sampled conditional on a claim having occurred evaluates the conditional severity law, not the unconditional annual cost of a policy.
A fundamental expected-score identity
For binary event Y∈{0,1}, suppose its true probability is q and a forecast reports p. The binary quadratic loss is s(p,Y)=(p-Y)^2. Direct expansion gives
The last term is nonnegative and vanishes exactly when p=q. Thus the loss is strictly proper for probabilities on [0,1]. The irreducible term q(1-q) depends on intrinsic uncertainty. If q=0.1, an accurate probability of 0.1 has expected quadratic loss 0.09; a forecast of 0.25 has expected loss 0.1125. The increase of 0.0225 is an exact expected-score penalty for misspecification, not a realized effect estimated from a portfolio.
This simple decomposition explains why ranking models only by the frequency of correctly predicted event labels discards information. For rare insurance claims, a classifier that predicts “no claim” for all policies can have high event-label accuracy and yet fail to communicate the heterogeneous occurrence probabilities necessary for pricing and risk aggregation. A proper probability score evaluates the probability report itself.
The observed score and its expectation must also be separated. If Y=1, the issued probabilities 0.1 and 0.25 obtain observed quadratic losses 0.81 and 0.5625: the less accurate probability relative to the generating law happens to do better on this event. Propriety is an expectation criterion across correctly specified repeated forecasts, not retrospective knowledge of an unobservable true probability for one policy.
Transformations, comparability, and scaling
If s is proper, then a s(P,y)+b(y), with positive constant a and integrable observation-only function b, remains proper. The observation-only addition changes all forecast scores by the same amount for a given verification outcome. Multiplication by a negative constant reverses preference and changes the penalty convention. Multiplication by an outcome-dependent weight w(y), however, can shift the expected optimum; it is generally not a permissible transformation of this kind.
The domain and dimension of the outcome also matter. A logarithmic score for a density has a dependence on the measurement unit because densities transform under a change of scale; for the same realized observation and common transformation, relevant forecast comparisons may nevertheless be invariant up to an observation-only Jacobian term. CRPS has the units of the outcome. Comparing CRPS values for losses quoted in dollars and in thousands of dollars without conversion is meaningless. Likewise, comparing unweighted annual totals for insurers of vastly different size answers a different question from comparing scores per policy-year.
A forecast should be specified as a distribution together with its observation horizon, monetary basis, claim-development convention, currency, inflation treatment, deductible and limit assumptions, and admissible conditioning information. This is a necessary scientific object description. A scoring rule does not choose any of these elements for an investigator.
Convexity and the divergence perspective
Proper scoring rules can be characterized through convex functions under regularity conditions, as developed in the general scoring-rule characterization. Define the regret D_s(Q,P)=L(P,Q)-L(Q,Q). For a proper rule, D_s≥0; for a strictly proper rule, D_s vanishes only when P=Q in the relevant forecast class. For logarithmic loss, the regret is Kullback–Leibler divergence under the usual absolute-continuity conditions. For the binary quadratic rule it is (p-q)^2.
As summarized in Dawid and Musio's survey (2014), the divergence interpretation is more informative than merely announcing that a score is “proper.” It shows which forecast differences the evaluation can distinguish. A score for the mean only is indifferent to distributions sharing the same mean; a score for one exceedance probability is indifferent to changes that preserve that exceedance probability. Strict propriety must always be stated relative to the chosen forecast object.
In empirical work, estimating average regret is difficult because the oracle expected score L(Q,Q) is unknown. Comparing mean observed scores for methods A and B instead estimates a difference in predictive risk. The comparison can be statistically meaningful without identifying the true conditional law, provided the test sample and its dependence structure support the estimator. A model's positive skill against a baseline therefore establishes relative performance for the chosen score and target, not complete distributional truth.
Repeated observations and predictive dependence
For verified cases i=1,…,n, an empirical mean score is
If cases are conditionally independent or satisfy an appropriate weak-dependence law of large numbers, and if their score differences have suitable finite moments, the average can estimate the corresponding target-population expected loss. These conditions require attention in insurance: policies share drivers, regions share storms, claim payments share inflation and legal development, and successive portfolio-year forecasts share historical estimation samples.
Where the research target is an average score per policy, each policy should generally receive a declared weight consistent with that target. Where the target is the total cost distribution of a portfolio, the score should be applied to the aggregate loss law itself. Averaging policy-level scores cannot reconstruct dependence-sensitive quality of the aggregate distribution. This distinction underlies the later treatment of multivariate and aggregate losses.
Propriety and Elicitation
Probability distributions and numerical functionals
Forecast evaluation has at least two levels. A distributional rule asks for a probability law P. A consistent scoring function for a statistical functional T asks for a number t=T(P), such as a conditional mean, median, quantile, or expectile. A function l(t,y) is strictly consistent for T when its expected loss under Q is uniquely minimized at t=T(Q), allowing for the usual qualifications when a quantile is not unique. The full distinction is explained in Gneiting's analysis of point forecasts.
For the mean μ=E[Y], squared-error loss l(t,y)=(t-y)^2 is consistent when the second moment exists. For an α-quantile, the pinball loss l_α(t,y)=(1{y<t}-α)(t-y), equivalently (α-1{y<t})(y-t), is minimized at an α-quantile. The median is the special case α=1/2 and is linked to absolute-error loss. These objectives select different point summaries of a skewed claim-severity distribution and may produce different optimal numerical forecasts from the same correctly specified P.
An insurance report can be correct about expected cost and incorrect about tail probability. For example, distributions placing all mass at 100 and distributions placing half the mass at 0 and half at 200 both have mean 100; they differ radically in exceedance probability at 150. Squared error applied to a mean forecast cannot distinguish these distributions when both reports are 100. The scoring protocol must select the target level before interpreting the resulting ranking.
Quantile and interval forecasting
Suppose a regulator, reinsurer, or risk committee requests a high conditional loss quantile. A fixed high-level pinball score properly evaluates the reported quantile rather than a complete distribution. However, a single quantile cannot characterize the frequency or severity of losses beyond it. A forecast of a 99th percentile and a forecast of the probability that losses exceed a contract limit answer related but distinct questions.
For a central (1-α) prediction interval [l,u], an established interval score takes the form
Under regularity and uniqueness conditions its expected value is minimized by choosing the corresponding lower and upper predictive quantiles, as described in the interval forecast results. Its first term rewards narrower intervals, while the two further terms penalize misses outside the interval. One realized interval score does not measure nominal coverage: it combines interval width and the magnitude of violations.
Actuarial models often report both loss expectations and risk-capital quantiles. A demand for several functionals should lead to an evaluation corresponding to those specific functionals, or to a proper full-distribution score from which they can be recovered. Mixing incompatible target functions in an undeclared composite index can obscure what is actually being optimized.
Expected shortfall and joint elicitation
Risk management often uses expected shortfall (ES), a tail-average measure, alongside value at risk (VaR). Their evaluation requires care: the inability to elicit ES alone in a broad class by a strictly consistent one-dimensional score does not preclude jointly consistent scoring of VaR and ES. The result in Fissler and Ziegel (2016) establishes important higher-order elicitation conditions and shows the joint relevance of these two functionals. The 2021 correction note identifies errors in two specialized results and additional imprecisions; the present monograph relies only on the established joint-elicitability conclusion and retains the correction in the scientific record.
In insurance, a proper scoring rule for the entire aggregate-loss law can be used to evaluate distribution forecasts from which VaR and ES are computed, provided the moments and tail quantities exist. If the insurer submits only a pair (VaR, ES), the joint-functional literature offers an appropriate evaluation language. It is a category error to assess a point report as though it were an arbitrary probability density.
Honest revelation and decision incentives
Propriety is sometimes described as an incentive for honest probability reporting. That interpretation presumes the forecaster's payoff or selection procedure is aligned with the specified score and the forecaster has genuine subjective or objective uncertainty represented by Q. If a commercial team is rewarded instead for low measured capital, claims-denial efficiency, or a targeted premium, the institutional incentive may differ from the scientific scoring incentive. The mathematics ensures a particular expected-score optimum; governance determines whether that objective is actually used.
Different proper scores weight aspects of the distribution differently while maintaining propriety. A score is therefore a legitimate scientific evaluation criterion only within its declared task. No theorem implies that selecting the forecast that wins on one score automatically maximizes policyholder welfare, minimizes insolvency probability, or satisfies a jurisdiction's regulatory capital requirements.
Conditioning and forecast populations
Suppose a prediction is a function P_X of rating variables X. If s is strictly proper for unconditional probability laws, then under suitable integrability its conditional expected score E[s(P_X,Y)|X] is uniquely minimized by the true conditional law of Y given X, almost surely. Taking expectations over X preserves the optimum. In practice, selection on policy acquisition, underwriting acceptance, reporting, and ultimate claim development can alter the observed population. The actual condition under which the score is proper must be matched to the population whose forecasting task is claimed.
This matters when a model is evaluated only on policies that were accepted by an earlier underwriting system. The exercise can rank forecasts for those accepted policies without establishing accuracy for applicants who were rejected. Similarly, severity forecasts scored only on paid claims describe a different conditional law from all incurred losses before coverage and dispute resolution. No change of scoring rule alone identifies outcomes that the process never records.
Logarithmic and Brier Scores
Binary Brier score and logarithmic loss
For an event with forecast probability p and observed value y∈{0,1}, the binary Brier penalty is s_B(p,y)=(p-y)^2. For 0<p<1, the logarithmic penalty is
Both are strictly proper. The expected logarithmic penalty is minimized at p=q, with regret
The logs are interpreted using conventional limits at probabilities 0 and 1. A forecast assigning zero probability to an outcome that occurs incurs infinite logarithmic loss, while the Brier penalty is bounded between 0 and 1 for the binary convention used here. That distinction governs both tail sensitivity and numerical robustness. The logarithmic score detects unsupported categorical outcomes sharply; the Brier score limits the contribution of a single event to a finite bound.
In a portfolio with rare liability events, a hard zero probability can arise from smoothing choices, insufficient historical data, or exclusion of event types from a model. Replacing exact zeros with arbitrary floors changes the submitted predictive distribution; the floor must be fixed as a prospective modeling rule rather than introduced after observing an unexpected claim. A log-score comparison must report how probability support is handled and whether the model is being evaluated under the same outcome definition across methods.
Categorical outcomes and normalization conventions
For K mutually exclusive outcome classes with forecast probabilities p_k summing to one and realized class j, the multicategory quadratic score is
The original categorical Brier framework and later common binary conventions use differing normalizations. For K=2 with probabilities (p,1-p), the multicategory formula equals 2(p-y)^2 if y indicates the positive class. Researchers comparing published figures must establish whether a factor of two, or an average over classes, has been applied; multiplying by a positive constant preserves propriety but alters reported score magnitude.
Multicategory insurance applications include defined claim-cause categories, mutually exclusive bands of claim amount, claim closure status at a fixed horizon, or severity strata. If categories overlap, they cannot be scored as one mutually exclusive categorical variable without redefining the outcome. In particular, a policy may generate multiple claims in different categories, so the appropriate object may be a joint count vector rather than a single nominal class.
Log score for discrete counts
For a predictive count law with probability mass p_k=P(N=k), the logarithmic score is -log p_n for observed n. It compares a Poisson distribution, negative binomial distribution, zero-inflated model, or a nonparametric discrete law on the same observation space, provided predictive probability mass is calculated consistently. A negative-binomial forecast can assign greater mass to large counts than a Poisson forecast with identical mean; the log score may discriminate between them even when the compared models have nearly identical mean squared prediction errors.
Count-distribution evaluation has a developed literature. Czado, Gneiting, and Held (2009) discuss ordered count outcomes, predictive deviance, calibration checks, and proper scores. Their empirical case studies concern patent counts and cancer counts rather than insurance claims. Their mathematical and diagnostic tools transfer to insurance count forecasting after adjusting the outcome, exposure, and conditional population; they do not provide empirical evidence that one insurance algorithm is superior.
For a Poisson forecast with mean λ and observed count n, -log p_n=λ-n log λ+log(n!), with the usual convention at λ=0. The factorial term is constant across forecasts for the same observation, so a Poisson-family comparison can omit it without changing within-case rankings. If predictive distributions belong to different families, one must use a common score definition for all competitors. Using model-specific deviance constants destroys valid comparability.
Dawid–Sebastiani and moment-based alternatives
A score based only on forecast mean μ and variance σ²>0 is the Dawid–Sebastiani expression
For a law Q with mean μ_Q and variance σ_Q², the expected value depends on P only through its mean and variance. It is minimized by matching those two moments, but different distributions with identical first two moments tie. Thus it is proper for the moment functional where the score is well defined and cannot be strictly proper for arbitrary full distributions. It is useful to screen location and dispersion without treating a moment match as verified tail calibration.
The proper scoringRules implementation study documents practical computation for several predictive families and score types. A reproducible actuarial evaluation must record parameter conventions, truncation tolerances, simulation counts, and software versions, because numerical approximation can create apparent differences unrelated to the underlying forecast.
A counterexample to outcome-weighted scores
Consider the quadratic event score multiplied by a realized-outcome weight. Let w(1)=9 and w(0)=1. Under a true event probability q=0.1, the expected weighted penalty becomes
Differentiating yields the minimizer p*=0.9/(0.9+0.9)=0.5. The actual probability 0.1 is no longer optimal. This counterexample is exact and demonstrates how an apparently reasonable scheme giving severe claims nine times as much influence rewards distorted occurrence probabilities. The same difficulty can arise when individual loss magnitudes are used as post-outcome multipliers of a probability score.
This finding does not imply that large insurance losses are unimportant. It implies that their importance should be represented by choosing the correct forecast target, a proper score with suitable built-in emphasis, or a transparent decision-theoretic loss for the economic choice. Outcome multiplication of a proper score changes the elicited forecast unless a separate proof preserves propriety.
Continuous Ranked Probability Score
Definition, interpretation, and integrability
For a real-valued forecast CDF F with suitable first-moment integrability, the continuous ranked probability score is
Its historical development is traced through Matheson and Winkler (1976) and the general treatment by Gneiting and Raftery (2007). CRPS is a proper distributional penalty, and strictly proper on an appropriate class of distributions with finite first moments. It has the units of the underlying loss variable, permitting direct interpretation of scale but requiring careful harmonization of exposure and currency.
The integral decomposes the forecast into a continuum of threshold-event probabilities, each evaluated with the binary quadratic penalty. For each fixed z, the forecast states F(z)=P(Y≤z), the probability of not exceeding that threshold. Integration across z aggregates those proper threshold evaluations. This representation gives a direct interpretation for insurance: the score checks the entire projected CDF rather than only its mean or one rare-event probability.
An equivalent identity, for independent X,X' drawn from F under finite first-moment assumptions, is
The first term rewards proximity between simulated losses and realized loss; the second term compensates for the spread of the predictive distribution. Omitting the second term converts the objective into a different and generally improper assessment of the full predictive law. For a degenerate forecast concentrated at a single a, CRPS reduces to |a-y|, showing why absolute-error point prediction appears as a limiting special case.
Why CRPS is proper
For a true CDF G, take the expectation of the integrand under Y∼G. Since the indicator 1{Y≤z} is Bernoulli with mean G(z),
Integrating the nonnegative difference gives
When the integral is finite, the difference vanishes exactly when F and G agree almost everywhere as CDFs; their monotonicity and right-continuity then determine the same distribution. This derivation establishes strict propriety on the specified class and also shows its sensitivity: CRPS penalizes the squared area between predicted and true CDFs.
The identity demonstrates an important difference from log score. Logarithmic loss inspects predictive mass or density at the realized outcome; CRPS accumulates errors in threshold probabilities over the outcome axis. Large-severity regions matter over their numerical width, so changing monetary scale changes the score proportionally. It is scientifically legitimate to choose a score with an explicit tail emphasis, but the weighting must occur in a construction that preserves propriety.
Discrete ranked probability score
For nonnegative integer counts N, a ranked probability score can be defined as
This is the discrete analogue of the CRPS threshold representation. The infinite series must converge for the forecast class under study; computational practice may use an adequately controlled truncation or analytic expressions. The score attends to ordered count categories and their cumulative probabilities rather than treating an observation of three claims as an unrelated nominal label.
If all possible claim counts are bounded by a known maximum, the sum terminates and strict propriety follows from the propriety of all included threshold scores. For an unbounded count distribution, transparent tail truncation tolerances matter: naively truncating at a small cutoff can remove differences in the high-count tail and change the effective evaluation problem.
Tail-weighted CRPS and the difference between weights
A nonnegative deterministic threshold weighting w(z), fixed before observing the test outcomes, yields
By the same calculation, expected regret is ∫w(z)(F(z)-G(z))²dz≥0. The rule therefore remains proper when integrability holds. Strict propriety for the full distribution can fail if w vanishes on a region that permits distinct CDFs to differ without changing the score. This distinction is essential when the evaluation emphasizes insurer ruin thresholds or reinsurance attachment points.
Gneiting and Ranjan (2011) analyze threshold- and quantile-weighted rules for distributional forecasts. The underlying principle gives an actuarial interpretation: emphasizing a stated range of claim amounts is legitimate when the weighting construction is itself proper. Multiplying an already calculated score by a realized claim amount, as in the earlier counterexample, is a different mathematical operation with a different optimum.
For a tail threshold u, selecting w(z)=1{z≥u} focuses the score on upper-threshold forecasts. Such a score does not identify how a forecast distributes probability below u, aside from restrictions implicit in being a valid CDF. An insurer wishing to evaluate general severity forecasts and extreme-loss forecasts can report both ordinary CRPS and a predeclared upper-threshold weighted CRPS, interpreting each against its own scope.
Zero mass and continuous positive severity
Insured loss per exposure frequently has an atom at zero and a continuous distribution over positive amounts. Let π=P(Y=0) and let H be the conditional positive-loss CDF. For y>0, the full unconditional CDF is F(y)=π+(1-π)H(y), with F(y)=0 for y<0. The CRPS applies directly to this mixed law as a CDF-based score. It captures both the frequency of zero outcomes and the positive-loss severity distribution, provided the monetary loss definition and horizon are consistent.
If an investigator evaluates H only on Y>0, the score is proper for positive severities conditional on a positive insured payment. It does not test whether the issued π is accurate. Conversely, a Brier score for 1{Y>0} evaluates π but cannot test the magnitude of positive claims. A coherent total-cost evaluation needs a distribution for the full Y, while component diagnostics may assess each mechanism separately.
Empirical and Monte Carlo computation
For a predictive sample X_1,…,X_m from F, one estimator of CRPS is
This is the CRPS of the empirical distribution placing mass 1/m at each simulated draw. It is not automatically an unbiased estimator of the population CRPS for finite m, because diagonal pairs contribute zero. An unbiased U-statistic version replaces the second average by the distinct-pair denominator 2m(m-1) when m>1, under independent sampling. An evaluation protocol should state which estimator is used, whether the simulations are conditionally independent, and whether common random numbers are used across models.
Numerical implementation studies such as Jordan, Krüger, and Lerch (2019) supply practical formulae and algorithms, while the present derivation explains why a numerical approximation must be checked against simple analytic cases. Two indispensable software tests are the deterministic forecast CRPS(F,y)=|a-y| and the nonnegative regret identity for a two-point distribution. Integration limits, monetary units, and Monte Carlo approximation error belong in a reproducible calculation record.
Insurance Count and Severity Forecast Scores
The frequency–severity system
A collective insurance loss model often writes aggregate loss as
where N is claim frequency during a defined exposure period and Z_j is the monetary severity of claim j. Under conditional independence and suitable moment assumptions, E[S]=E[N]E[Z] for identically distributed severities, but the entire law of S depends on both component distributions and their dependence structure. A scoring method applied only to the expected value E[S] cannot certify predictive tail behavior.
The compound Poisson–gamma family is one actuarially relevant construction. Jørgensen and Paes de Souza (1994) study its fitting to insurance data and its relation to a Tweedie exponential-dispersion representation. The model allows exactly zero total loss when no claim occurs and a continuous positive amount otherwise. It provides an instructive setting for comparing distributional log scores and CRPS to mean-only prediction error, but the cited study does not imply universal appropriateness of Tweedie laws across insurance lines.
An actuarial forecast can be decomposed into frequency P(N|X), positive severity P(Z|N>0,X), and, where needed, dependency between count and severities. Evaluating each factor separately answers a diagnostic question. Evaluating the predictive aggregate law of S answers the ultimate total-cost question. The two evaluations should be reconciled: good component scores do not imply a correct aggregate distribution if joint dependence or aggregation rules are misspecified.
Exposure as an offset and as a scoring weight
A policy observed for three months and one observed for a full year do not have the same exposure to an annual event rate. Under a homogeneous Poisson intensity model with rate λ per exposure-unit and observed exposure e, N|e has mean eλ. Its probability law, not λ alone, is the object scored against the observed count N. An incorrect comparison between count N and unit-rate forecast λ confounds model error with exposure duration.
Insurance portfolios may also differ in insured sums, geography, business mix, deductibles, and reporting maturity. Such characteristics should enter the predictive conditioning variables or the definition of the target outcome. Their mere inclusion as outcome-dependent evaluation weights after observing a loss can undermine propriety. Exposure weights fixed at forecast time can be used to define a target portfolio average of proper conditional scores, with their exact interpretation stated in the protocol.
For weights a_i≥0 fixed independently of verification outcomes and forecasts, the weighted sum ∑a_i s(P_i,y_i) retains conditional propriety when each forecast is evaluated under its appropriate conditional law. This is a result about the chosen population mixture; it does not license selecting weights after seeing which insurer or model performed best. Reporting both equal-policy and exposure-weighted evaluations can reveal material dependence of a ranking on the assessment population.
Alternative count laws and excess zeros
Policy-year claim counts can exhibit overdispersion, extra zeros, and cluster dependence. A Poisson model, a negative binomial model, a hurdle model, and a zero-inflated mixture may produce similar fitted means but different tail probabilities and different atom sizes at zero. Strictly proper count-distribution scores can discriminate these laws when the data genuinely reflect the claimed count population and sufficient independent information accumulates.
A proper score does not decide why a large fraction of records has zero claims. Some zeros represent the absence of covered loss; others can arise from late reporting, incomplete claims triangles, missing records, deductibles, or underwriting attrition. If the meaning of zero changes across datasets, score differences mix statistical quality with data-definition change. The valid methodological response is to reconstruct the observation and coverage definitions rather than assign a new score to unexamined data.
A more subtle issue concerns rare events. In a finite test set with very few large claims, a model that underpredicts extreme counts may achieve a better average CRPS than one that better represents the extreme tail but is less accurate around typical outcomes. This does not contradict propriety: CRPS is proper in expectation for the entire law, while different proper scores and small samples rank finite forecasts differently. The analyst should prospectively select the weight function or functional target corresponding to the scientific question.
Claim severity, deductibles, and limits
Let gross severity be G≥0, deductible d and policy limit m>d. A stylized insurer-paid amount is Y=min(max(G-d,0),m-d). Forecasts for G and for Y are not interchangeable. The payment transformation creates an atom at zero for claims below the deductible and an atom at the limit for claims exceeding the coverage ceiling. Scoring an unconstrained gross-severity density against bounded paid amounts creates a misspecified sample space.
A proper scoring rule for Y must account for the mixed discrete–continuous payment law. If the prediction of G is available, transform it prospectively to a forecast of Y and apply a score defined on the resulting law. A log score built solely from a continuous density cannot assign the correct probability to the zero and cap atoms; one needs appropriate mass-versus-density treatment relative to a declared dominating measure. CRPS can accommodate such mixtures through the CDF.
The same logic applies to inflation adjustments, claims made versus claims occurring coverage, reinstatements, and reinsurance recoveries. Gross, ceded, and net loss are distinct random variables. A good score on gross claims does not prove accuracy of net retained loss after a nonlinear treaty. Scoring rules follow the random quantity chosen by contract and research design.
Development and right censoring
Insurance observations may be incomplete: a policyholder survives beyond an evaluation date, a disability claim remains open, a liability claim has reported but not settled, or ultimate loss will only emerge after years of development. Suppose T is a time to event and C is a right-censoring time; observed data are U=min(T,C) and Δ=1{T≤C}. Substituting U for T in a full-data score treats censored event times as observed failures and generally biases evaluation.
A classical foundation for evaluating time-dependent event prediction under censoring is Gerds and Schumacher (2006), who analyze inverse-probability weighting and conditions for consistent estimation of expected Brier scores. The full-data expected-score target is identifiable under stated independent or conditionally independent censoring and positivity assumptions; violating these assumptions can invalidate score estimation. These conditions are substantive, not technical details.
The 2026 preprint Jonkers and colleagues on censored scoring develops an observed-data-law perspective that maps a forecast through a censoring mechanism and analyzes propriety and the identifiable region. As a recent preprint, its claims are identified here as research in progress rather than an established peer-reviewed insurance standard. The underlying conceptual distinction is clear independently: proper assessment of an observed-data law need not uniquely identify every aspect of an unobserved event-time distribution.
For delayed insurance claims, observational incompleteness can be more complex than right censoring: claim counts are subject to reporting lag, severities can be revised, reopened claims affect ultimate totals, and inflation alters future payment amounts. Any evaluation of “ultimate loss” from immature triangles must state its development model and observation window. A score on an estimated ultimate outcome incorporates uncertainty and potential model dependence in the verification target; describing it as an observed ground truth would overstate the evidence.
Multivariate and aggregate insurance loss
Consider lines A and B with marginal loss distributions F_A and F_B. A model may forecast both marginals correctly while misrepresenting their copula or common-catastrophe dependence. Separate univariate CRPS values cannot detect the difference between joint laws with identical marginals. Yet their sum A+B, and therefore insurer solvency and reinsurance attachment probability, can vary substantially with dependence.
For a joint forecast P of d-dimensional loss vector X and realized vector y, an energy score can be written
with a suitable metric and finite moment conditions. It generalizes univariate CRPS. Strictness and sensitivity depend on the metric and distribution class. Scheuerer and Hamill (2015) investigate variogram-based proper scores and explain limits in detecting misspecified dependence with commonly used multivariate scores. Their empirical work concerns multivariate environmental forecasts; the mathematical concern is directly relevant to multi-line insurance losses.
An insurer can use proper scores for individual marginals, the joint vector, and the aggregate sum, with each score separately identified. It would be unsound to claim that a good joint score establishes accurate catastrophe accumulation in every extreme region: different scores have different sensitivity, and finite data can provide very little information about extreme dependence. A multivariate score improves conceptual alignment, while rare-event uncertainty and model misspecification remain research questions.
A counterexample involving joint structure
Let two claim indicators have marginal probability 1/2 each. Under model A they are perfectly dependent, with (Y_1,Y_2)=(0,0) or (1,1), each with probability 1/2. Under model B they are independent, assigning probability 1/4 to each of the four binary pairs. All marginal Bernoulli forecasts and their expected Brier scores coincide under A and B. Yet aggregate count N=Y_1+Y_2 has support {0,2} under A and {0,1,2} under B. In particular, P(N=2) is 1/2 under A and 1/4 under B.
A portfolio with two correlated catastrophe-triggered claims can therefore have identical marginal forecast verification and different aggregate attachment risk. Scoring the law of N using a proper count score distinguishes the two laws in expectation. This is an algebraic illustration; it is not evidence about the magnitude of dependence in any real insurance book.
Score Decomposition and Limits
Calibration, reliability, and resolution
A proper score is an integrated measure of forecast performance, but a single mean value cannot reveal every source of error. The influential calibration and sharpness analysis distinguishes statistical consistency between forecasts and observations from the concentration of predictive laws. In insurance, calibrated predictions may be uninformatively similar across different risk levels, while a sharply concentrated forecast can be badly calibrated. The scoring rule rewards an appropriate combination in expectation.
For a binary event, the Brier score admits a classical decomposition into reliability, resolution, and uncertainty, under a grouping scheme and corresponding interpretation. Murphy (1973) developed a vector partition of the probability score, while Bröcker (2009) treats general proper-score decompositions. For population versions the schematic relationship is score = reliability − resolution + uncertainty, with definitions depending on the forecast grouping and scoring convention. In finite samples, binning, shrinkage, sampling variation, and the distinction between observed and expected terms alter estimated components.
Reliability asks whether event frequencies align with issued probabilities; resolution asks how informative the forecasts are relative to a climatological or portfolio-wide base rate; uncertainty reflects the unconditional event variability of the target population. A low overall Brier score may arise from the predominance of non-events and cannot alone establish useful discrimination among risk categories. A responsible dossier reports the mean proper score together with relevant diagnostic decompositions rather than treating one scalar as a complete model audit.
Calibration can be checked at event thresholds, through cumulative residuals, and through probability integral transform diagnostics for appropriately continuous outcomes. For ordered count data, Czado and coauthors (2009) describe nonrandomized alternatives adapted to discreteness. Calibration diagnostics complement proper scores; they should not be retroactively tuned on the same final test set used to claim an independent comparison.
Portfolio segmentation and aggregation
Insurance prediction quality is often heterogeneous across geography, hazard, age, duration, deductible, coverage layer, and distribution channel. Average score differences can conceal underprediction for a thin but financially significant segment. A scientifically interpretable analysis reports declared segment definitions, sample sizes, event counts, outcome maturity, score distributions, and uncertainty intervals where estimable. Post-hoc selection of segments with extreme gains should be labeled exploratory.
Consider a score difference d_i=s(P_{Ai},y_i)-s(P_{Bi},y_i). The average \bar d measures the observed relative score, with negative values favoring method A under the penalty convention. Independence of d_i is often untenable: one hurricane generates correlated observations for many property contracts, a legal shock affects several liability claims, and one inflation period affects reserving portfolios across years. Standard errors based on treating every policy as independent can substantially understate uncertainty.
Comparison should use a sampling unit and variance method consistent with the data-generating hierarchy, such as cluster resampling by event, geography or underwriting period, block resampling of time sequences, or a justified sandwich estimator. The forecast-comparison literature in Diebold and Mariano (1995) is relevant to loss-difference testing under dependence; a test's asymptotic conditions must still be checked in the insurer's setting. With very few catastrophes, neither an asymptotic p-value nor a narrow confidence interval should be accepted without sensitivity analysis.
Time-based model validation
A reproducible scoring experiment defines a forecast origin t, training information H_t, prediction horizon h, target Y_{t+h}, and realized verification information. The method must be fixed before Y_{t+h} enters H_t. Historical rolling-origin evaluation reproduces this sequence for multiple origins. Randomly splitting individual rows can be inappropriate when the same claim, policy, catastrophe, or economic period appears in both training and evaluation sets.
Prequential evaluation allows model refits as new information arrives, but every refit must be documented at its historical forecast origin. Hyperparameter selection on the entire held-out period changes the meaning of the resulting score. An evaluation may include separate development and final test windows or a nested time-based design; the important condition is separation of model selection from independent assessment.
Reserve prediction presents a specific problem: settlement and ultimate loss outcomes arrive years after the forecast date. The score at one-year emergence horizon evaluates a different target from the score on ultimate liabilities. Reporting which triangle valuation date was known, which accident or underwriting years were forecast, and whether subsequent development was incorporated only into verification is essential for credible comparisons.
Benchmark performance and skill
A score difference has meaning relative to a baseline chosen for the same forecast target, horizon, and test population. A baseline might be a frequency-severity GLM, a historical cohort law, or a previously operational predictive system. A score skill index of the form 1-\bar s_A/\bar s_B can be used when the denominator is nonzero and both scores are comparable, but its interpretation depends on the selected loss and whether zero is a meaningful bound. Under scores whose numerical values may be negative, or after observation-only transformations, ratios and percentage improvements can change while model rankings do not.
For this reason, raw mean score differences accompanied by uncertainty assessments are usually easier to audit than an unqualified percentage “accuracy improvement.” When positive scaling or unit changes occur, record the factor and its effect. Claims of superiority should specify the tested models, fixed training period, observational target, and designated criterion. A single favorable in-sample deviance value is neither a prospective proper-score comparison nor evidence of predictive advantage.
Distribution shift and changing insurance conditions
Insurance forecasts operate in changing environments. Exposure composition, climate hazards, healthcare practice, construction costs, claims inflation, benefit terms, legal interpretations, and reporting incentives can alter the conditional law of losses. A scoring rule remains mathematically proper for the distribution evaluated at a future forecast origin, but a historically fitted estimator can fail because its assumed law is stale.
A forecast evaluated under one country's motor insurance laws cannot automatically be ranked as predictive evidence for a different regulatory and claim-settlement system. Nor can a score computed for gross claims be carried across an accounting change that redefines incurred loss. International comparison requires harmonized target definitions and an explicit account of residual institutional heterogeneity.
The way to address shift is empirical and design-based: preserve model versions, evaluate successive out-of-time cohorts, test transportability across declared segments, and examine whether score differences persist under reasonable shift scenarios. Proper scoring remains the evaluation language; it does not solve the econometric identification of a future population from a historically selected dataset.
Extreme losses and rare-event evidence
Tail risks have asymmetric financial consequences. Underwriting catastrophe exposure, reinsurance purchasing and solvency capital may depend predominantly on low-frequency, high-severity outcomes. An ordinary mean proper score can be statistically efficient for broad distributional evaluation while being weakly informative about an extreme quantile when data contain almost no observations beyond that threshold.
Threshold-weighted proper scores and high-quantile consistent scores permit ex ante emphasis on such decisions. Their use should be accompanied by the chosen threshold, business rationale, distribution support and effective number of relevant events. A tail-only score may be only proper rather than strictly proper for the full law, so it should not be described as validating all aspects of the distribution.
The mathematical theory of risk-measure elicitability and joint VaR–ES scoring helps formalize the distinction between evaluating complete loss laws and evaluating capital-relevant summaries. The study's interpretation is to report both a general proper distribution score and a separately specified capital-relevant evaluation when the scientific objective genuinely requires both. This is an analytical recommendation for transparent research design, not a new formal scoring theory.
Identification under incomplete observations
If only the indicator Z=1{Y>u} is observed, a binary proper score can identify the exceedance probability P(Y>u) but cannot identify the severity distribution above or below u. Even infinite data on Z cannot distinguish two distributions with the same exceedance probability. If Y itself is censored at a coverage cap, a proper rule for the capped outcome is strictly proper for the distribution of the capped payment, not necessarily for the gross loss law that generated it.
This observation yields a general methodological test: identify the mapping from the scientific target to the observable verification record, then ask whether distinct candidate target laws can induce the same record. When they can, no observed-data score can strictly distinguish those target laws without additional assumptions or data. The same issue arises when an insurer trains only on settled claims and the unsettled tail is missing nonrandomly.
The censored-outcome literature, including Gerds and Schumacher (2006), supplies weighting methods under specific assumptions. It does not license universal inference from incomplete records. A scientific report must state the censoring mechanism, positivity condition, administrative observation window, and sensitivity to departures from the assumed mechanism.
Research synthesis: what a valid insurance score comparison establishes
The formal and applied findings can be integrated into a sequential interpretation. First, a precisely defined prediction object must be chosen. Second, a score must be proper or consistent for that object on the relevant forecast class. Third, the actual reported forecast must be computed using information available at its forecast origin. Fourth, the observation and sampling design must support inference for the population and horizon claimed. Fifth, uncertainty in the observed score difference must be reported at the correct dependence level. Sixth, the model ranking must be stated for the declared score and target, with additional decision implications separately argued.
These conditions explain apparently conflicting evaluation outcomes. A model may improve logarithmic score yet lose on CRPS because the two scores attach different finite-sample sensitivities; a joint insurer catastrophe forecast may fail despite excellent separate line-level results; a conditional severity forecast may be accurate yet give poor annual aggregate cost predictions because occurrence probability is wrong; a capital quantile forecast may be meaningful but remain silent on general distributional calibration.
The proper conclusion of a comparison is correspondingly precise: for this stated target, horizon, population, score and verification mechanism, the tested method has this estimated predictive loss relative to these alternatives, with these sampling and modeling limitations. Claims beyond that sentence require additional evidence. This precision is particularly important in insurance because mathematical forecast accuracy, contractual claims-paying performance, financial resilience, and equitable market outcomes are connected but not identical quantities.
Reproducibility specification and internal verification
A complete computational replication package should disclose a data dictionary for outcome and exposure, permitted covariates, forecast origin and horizon, policy and claim identifiers after privacy-preserving transformation, split assignment, model fitting and version, prediction-support conventions, score formula and orientation, weighting rules, numerical precision and truncation, random seeds where simulation is involved, and the exact clustering or resampling method for uncertainty. Confidential insurer data can be replaced by synthetic demonstrations without claiming an empirical replication of proprietary outcomes.
For mathematical verification, the following exact checks are sufficient to reveal several common implementation errors. The binary Brier regret must equal (p-q)^2. The outcome-weighted counterexample with q=0.1 and weights (1,9) must attain its minimum at p=0.5 rather than p=0.1. For any deterministic forecast a, CRPS must equal |a-y|. For a finite-support CDF, numerical integration of CRPS should agree with the energy identity. Independent and comonotonic two-indicator constructions must have identical marginals but different aggregate distributions. A censoring routine must not treat a censored U as an observed event time without a specifically justified target transformation.
The worked demonstrations in this monograph are exact symbolic or explicitly numerical arguments. No insurer-specific predictive accuracy percentages, stress-test results, regulatory compliance certifications, or cross-country model-performance rankings are inferred. The evidence ledger distinguishes publisher full-text or source-level review from abstracts, institutional records, and the identified 2026 preprint. This is necessary for traceable scientific editorial decisions.
Conclusion
Proper scoring rules provide a rigorous basis for evaluating probabilistic insurance predictions because they align the expected assessment criterion with the probability statement being requested. Strict propriety, on a defined forecast class and observable target, makes the truthful conditional distribution the unique expected-score optimum. The logarithmic and Brier scores operationalize that property for categorical outcomes; the ranked probability score and CRPS extend it to ordered counts and monetary distributions; threshold-weighted constructions can retain propriety while focusing on specified tail events; and appropriate multivariate scores address aspects of portfolio dependence hidden from marginal evaluation.
The scientific conditions of their use are as important as the score formula. Claim frequency requires an exposure-consistent count law. Severity evaluation must respect conditional selection, deductibles, limits, atoms at zero or caps, and monetary definition. Compound loss distributions require aggregation of frequency, severity, and their dependence. Incomplete development and censored observations require a valid observational model. Temporal splitting, cluster dependence, and forecast-time information govern whether empirical score differences support a genuine out-of-sample ranking.
The analytical counterexamples establish three durable results. Outcome-dependent multiplication of an otherwise proper binary score can reward a systematically false probability. Correct marginal forecasts can coexist with a materially wrong aggregate-loss distribution. A proper score on a censored or transformed observation identifies only those aspects of the target law preserved by the observable mechanism. These are applications and derivations of established statistical principles to insurance, not declarations of a new theoretical system.
The appropriate scholarly standard is therefore an exact claim: a proper-score comparison evaluates the particular predictive object, on a specified population, with a specified verification design. It supports actuarial model criticism and forecasting research while leaving pricing decisions, capital policy, insurance regulation, and distributional justice to their additional economic and institutional criteria. A complete insurance forecasting study should preserve both the mathematical incentive property and the empirical chain that makes its observed use scientifically meaningful.
References
Brier, G. W. (1950). Verification of forecasts expressed in terms of probability. Monthly Weather Review, 78(1), 1–3. Publisher article. DOI: 10.1175/1520-0493(1950)078<0001:VOFEIT>2.0.CO;2.
Bröcker, J. (2009). Reliability, sufficiency, and the decomposition of proper scores. Quarterly Journal of the Royal Meteorological Society, 135(643), 1512–1519. https://doi.org/10.1002/qj.456.
Czado, C., Gneiting, T., & Held, L. (2009). Predictive model assessment for count data. Biometrics, 65(4), 1254–1261. https://doi.org/10.1111/j.1541-0420.2009.01191.x.
Dawid, A. P. (1984). Present position and potential developments: Some personal views—Statistical theory: The prequential approach. Journal of the Royal Statistical Society, Series A, 147(2), 278–290. https://doi.org/10.2307/2981683.
Dawid, A. P., & Musio, M. (2014). Theory and applications of proper scoring rules. Metron, 72, 169–183. Author preprint and bibliographic record.
Diebold, F. X., & Mariano, R. S. (1995). Comparing predictive accuracy. Journal of Business & Economic Statistics, 13(3), 253–263. https://doi.org/10.1080/07350015.1995.10524599.
Fissler, T., & Ziegel, J. F. (2016). Higher order elicitability and Osband's principle. The Annals of Statistics, 44(4), 1680–1707. https://doi.org/10.1214/16-AOS1439. See journal correction published in 2021 for specialized characterization results.
Fissler, T., & Ziegel, J. F. (2021). Correction note: Higher order elicitability and Osband's principle. The Annals of Statistics, 49(1), 614. https://doi.org/10.1214/20-AOS2014.
Gerds, T. A., & Schumacher, M. (2006). Consistent estimation of the expected Brier score in general survival models with right-censored event times. Biometrical Journal, 48(6), 1029–1040. https://doi.org/10.1002/bimj.200610301.
Gneiting, T. (2011). Making and evaluating point forecasts. Journal of the American Statistical Association, 106(494), 746–762. https://doi.org/10.1198/jasa.2011.r10138.
Gneiting, T., Balabdaoui, F., & Raftery, A. E. (2007). Probabilistic forecasts, calibration and sharpness. Journal of the Royal Statistical Society, Series B, 69(2), 243–268. https://doi.org/10.1111/j.1467-9868.2007.00587.x.
Gneiting, T., & Raftery, A. E. (2007). Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association, 102(477), 359–378. https://doi.org/10.1198/016214506000001437. Author-hosted full text.
Gneiting, T., & Ranjan, R. (2011). Comparing density forecasts using threshold- and quantile-weighted scoring rules. Journal of Business & Economic Statistics, 29(3), 411–422. https://doi.org/10.1198/jbes.2010.08110.
Jonkers, J., Van Wallendael, G., Duchateau, L., & Van Hoecke, S. (2026). Proper scoring rules for right-censored survival data. arXiv preprint arXiv:2606.06393. Peer-review status not independently established. https://arxiv.org/abs/2606.06393.
Jordan, A., Krüger, F., & Lerch, S. (2019). Evaluating probabilistic forecasts with scoringRules. Journal of Statistical Software, 90(12), 1–37. https://doi.org/10.18637/jss.v090.i12.
Jørgensen, B., & Paes de Souza, M. C. (1994). Fitting Tweedie's compound Poisson model to insurance claims data. Scandinavian Actuarial Journal, 1994(1), 69–93. https://doi.org/10.1080/03461238.1994.10413930.
Matheson, J. E., & Winkler, R. L. (1976). Scoring rules for continuous probability distributions. Management Science, 22(10), 1087–1096. https://doi.org/10.1287/mnsc.22.10.1087.
Murphy, A. H. (1973). A new vector partition of the probability score. Journal of Applied Meteorology, 12(4), 595–600. Publisher article. DOI: 10.1175/1520-0450(1973)012<0595:ANVPOT>2.0.CO;2.
Scheuerer, M., & Hamill, T. M. (2015). Variogram-based proper scoring rules for probabilistic forecasts of multivariate quantities. Monthly Weather Review, 143(4), 1321–1334. https://doi.org/10.1175/MWR-D-14-00269.1.
Ziegel, J. F. (2016). Coherence and elicitability. Mathematical Finance, 26(4), 901–918. https://doi.org/10.1111/mafi.12080.