Insurance by BogdanovaThe Encyclopedia of Insurance and Risk Science

RESEARCH DOSSIER

Model Selection and Predictive Benchmarking in Insurance

Abstract. Predictive modeling in insurance is a problem of specifying the quantity to be forecast, the population to which a forecast applies, the information available at the decision date, the loss function that measures success, and the conditions under which rival procedures are evaluated. This monograph examines model selection and predictive benchmarking for insurance claim frequency, severity, pure premium, claim occurrence, and probabilistic loss forecasting. It synthesizes actuarial methods and statistical learning, then addresses the distinctive consequences of contractual exposure, unequal observation periods, repeated policies, evolving portfolios, delayed outcomes, rare losses, insurance pricing constraints, and institutional governance. A model comparison is scientifically interpretable when the candidates answer the same predictive question, use comparable eligible information, and face an evaluation design that represents their intended deployment. The research distinguishes empirical generalization risk from in-sample fit, model selection from final performance assessment, probabilistic accuracy from economic consequences, and cross-dataset robustness from unqualified transferability. It derives the role of exposure offsets in count benchmarks, explains when the Poisson deviance, log score, Brier score, quantile loss, and distributional diagnostics answer different questions, and presents two fully specified constructed numerical examples illustrating the reversal of apparent model superiority under portfolio change. The examples are mathematical demonstrations, not estimates derived from an external insurance dataset. Evidence includes foundational statistical studies, actuarial monographs, peer-reviewed insurance comparisons, primary dataset metadata, and geographically bounded supervisory documents. The principal conclusion is that benchmark validity is established by congruence between target, observation process, validation population, information set, loss function, and decision use. A larger algorithm has no independent claim to superiority. The most defensible model selection is a documented, reproducible comparison supported by separate calibration, stability, subgroup, tail, operational, and governance evidence.

Keywords: Insurance predictive modeling; model selection; predictive benchmarking; actuarial learning; generalized linear models; machine learning; claims frequency; claims severity; pure premium; exposure; proper scoring rules; out-of-sample evaluation; temporal validation; dataset shift; model uncertainty; insurance pricing; reproducibility; governance.

Introduction

Insurance predictive benchmarking belongs to the intersection of insurance statistics, actuarial science, statistical learning, and the design of contractual decisions. The objective is to determine which procedure produces the most useful predictions for a specified insurance population and a specified future period. The unit of evaluation may be an insured risk, policy, exposure period, claim, coverage, claimant, or accident event. Each unit creates a different sampling and information problem. An experiment comparing two models must therefore begin with its scientific object.

The historical development of actuarial modeling supplies two complementary traditions. The first writes a probabilistic relationship between claims and rating factors, often through generalized linear models. The second uses flexible learning algorithms to discover nonlinear prediction functions. Breiman's statistical modeling analysis (2001) articulated the broader statistical distinction. Insurance practice needs both traditions, because parameter meaning, predictive performance, contractual explanation, and operational controls answer related questions. The open-access actuarial learning foundations of Wüthrich and Merz (2023) provide an integrated treatment of the classical and modern approaches.

The empirical insurance literature establishes that algorithm families can differ materially on particular portfolios. Noll, Salzmann, and Wüthrich's French motor comparison (2020) describes a generalized linear model, regression trees, boosting, and neural networks for claim frequency. The authors report that unengineered interactions limited their simple GLM relative to flexible alternatives on their example. Henckaerts and colleagues' tariff-plan study (2021) develops tree-based pricing work while incorporating exposure, zero-heavy counts, scarcity of positive severities, and model interpretability. Those studies establish concrete model-comparison questions; they do not confer a permanent ordering on all insurance portfolios.

Methodologically, the monograph employs conceptual specification, formal derivation, comparison of published results, targeted examination of source records, and explicitly constructed numerical illustrations. It does not present a newly trained insurance model, claim to reproduce the empirical studies, or infer global ranking from a collection of incompatible datasets. Its original contribution is the integration of these established results into a rigorous evaluation analysis, together with arithmetic demonstrations of why an apparently superior model can lose after the evaluation population changes. The analytic distinctions stated below are scientific interpretations and recommendations, not a newly named theory or doctrine.

Research questions and scope

The investigation asks what makes insurance model comparisons scientifically fair; which criteria support a defensible selection across model classes; how error, calibration, complexity, reproducibility, and operational usefulness interact; and how apparent gains vary across portfolios, periods, and institutions. The principal applications are property-casualty claim frequency and severity, with extension to claim occurrence and distributional forecasts. Life, health, catastrophe, and reserving applications require their own target-specific evidence, since their observation and truncation processes differ.

The subject boundary is model selection and comparative benchmarking. Detailed split-algorithm engineering, scoring-rule axiomatization, model interpretability techniques, and monitoring infrastructure are related research subjects with their own depth. This dossier states the necessary conditions for a comparison and derives their consequences for selecting among models.

Comparable Insurance Prediction Targets

The predictive object and information date

Let a decision be made at time tt using information Xi(t)X_i(t) available for insured unit ii. Let Yi(t,h)Y_i(t,h) denote its future target over horizon hh, and let Pt,hP_{t,h} denote the target population. A point-prediction procedure mm trained on eligible historical data DtD_t returns μ^m,i(t,h)\hat\mu_{m,i}(t,h); a distributional procedure returns F^m,i(⋅∣Xi(t))\hat F_{m,i}(\cdot\mid X_i(t)). A benchmark compares conditional forecasts for the same YY, horizon, population, eligibility rules, and information date. It also fixes whether the target is an observed paid amount, ultimate incurred amount, indemnity within policy limits, reported loss, claim count, or another outcome.

The predictive estimand is the statistical quantity that the procedure attempts to estimate. For squared-error loss it is the conditional mean under the evaluation population. For absolute-error loss it is a conditional median; for pinball loss at level τ\tau it is a conditional τ\tau-quantile. A frequency model can predict E[Ni∣Xi,ei]E[N_i\mid X_i,e_i] as a count or a rate normalized by exposure. These representations require explicit conversion before scores can be compared. A conditional claim probability is a different estimand from the expected claim count if repeated events are possible.

Suppose a policy is active for exposure eie_i years during the target period. For an annualized constant-intensity Poisson formulation, Ni∣Xi,ei∼Poisson(eiλi)N_i\mid X_i,e_i\sim\mathrm{Poisson}(e_i\lambda_i) and log⁡E[Ni∣Xi,ei]=log⁡ei+f(Xi)\log E[N_i\mid X_i,e_i]=\log e_i+f(X_i). The logarithm of exposure is an offset, not a freely fitted predictor. In the simplest stable setting, a benchmark λ^=∑iNi/∑iei\hat\lambda=\sum_i N_i/\sum_i e_i is the exposure-weighted portfolio claim rate. This derivation presumes valid positive exposure, matching count windows and comparable definitions of a claim. Its details follow the statistical modeling treatment in Wüthrich and Merz (2023).

The approximation is less straightforward where annual exposure does not represent a constant at-risk process. Short-period policies may carry seasonal effects, cancellations may be informative, multiple vehicles can belong to one policyholder, and an event can create correlated claims across covered units. The benchmarking target must reflect the event-generating and reporting process rather than assume the offset alone has removed all temporal structure.

Frequency, severity, and pure premium

Frequency, severity, and pure premium invite separate comparisons. Let NiN_i be the claim count and ZijZ_{ij} the amount associated with claim jj for policy ii. The total cost is Si=∑j=1NiZijS_i=\sum_{j=1}^{N_i} Z_{ij}. Conditional expected annual cost per unit of exposure is E[Si∣Xi,ei]/eiE[S_i\mid X_i,e_i]/e_i. Under additional assumptions governing conditional claim amounts and frequency, a two-part approach may express that quantity as annual frequency times conditional mean severity. The equality requires attention to dependence, conditioning variables, and the definition of the severity population.

Selecting a frequency model using Poisson deviance and selecting a severity model using a positive-claim loss does not by itself prove that their product is the best pure-premium predictor. Model errors can interact, and claim-free policies disappear from conditional severity samples. A joint total-cost benchmark and a component-specific diagnostic both have legitimate uses. The primary comparison should follow the application: claim-event forecasting, claim handling, product pricing, or loss-cost budgeting.

The contractual definition of SiS_i also matters. Deductibles, limits, attachment points, coverage changes, claim settlement periods, recoveries, and inflation adjustment alter the numerical response. Two insurers may both report “motor severity” while modeling different covered benefits. Pooling their loss labels or treating the results as directly transferable may change the target under study.

Identical information sets and leakage

A fair comparison gives candidate models access to the same eligible historical data, subject to a documented exception when one model requires a permitted input unavailable to another. All preprocessing that learns from data—including category-frequency encoding, imputation, normalization, feature selection, target encoding, and balancing—must be fitted within the relevant training partition. Information derived from later settlements, post-inception renewals, fraud investigation outcomes, or globally computed portfolio statistics can leak future knowledge into an earlier underwriting decision. That leakage can increase holdout performance while destroying deployability.

The time at which information becomes available should be recorded for every feature. A final reported claim amount cannot be used as a predictor for a price fixed before that claim occurred. A geographical score constructed from historical claims can be admissible if computed from information available at the price date and its construction remains inside the training history. The distinction is temporal and procedural, not a label attached to the variable name.

Sampling units and transportability

Policies, people, vehicles, claims, and insured events form overlapping statistical units. A random row split may place the same household, fleet, location or catastrophe into both training and testing. A credible benchmark defines whether the deployment question concerns familiar units in a later time period, new units in the same population, a different distribution of policies, or a new country. This definition determines the valid evaluation split and the meaning of its result.

Roberts and colleagues' structured cross-validation analysis (2017) shows why structured temporal, spatial, and hierarchical dependence can make randomly separated records deceptively similar. The insurance application is a direct methodological implication: repeated insured entities and clustered events must be grouped or evaluated in a forward horizon where the operating question requires such independence. A forward-time test and a new-entity test answer different questions and should be reported separately when both are important.

Baseline Models and Benchmarks

A baseline is a scientific comparator

A valid benchmark includes at least one transparent rule that could reasonably have been deployed without the candidate complexity. For a frequency application, this may be an exposure-weighted aggregate rate; for a binary claim outcome, a historical base probability; for monetary losses, an exposure-adjusted mean or an existing actuarial rating formula. A baseline should be trained on the same eligible history and recalibrated according to the same timetable as its competitors. A poorly fitted or artificially frozen baseline creates an easy contest rather than a meaningful test.

A basic benchmark ladder can be operationalized as aggregate mean, major-risk-cell rate, classical GLM with defensible features and exposure treatment, regularized GLM or additive model, and a flexible learner such as boosted trees. The sequence separates the value of additional predictors, nonlinear response, interactions, and algorithmic flexibility. It can also reveal whether an ensemble improves a commercially established tariff by a practically meaningful amount. The CAS pricing monograph by Chalk and colleagues (2026) identifies the real actuarial complications of high-cardinality predictors, nonlinearity, correlated factors and model blending.

The ladder is comparative rather than prescriptive. A simple GLM with well-designed interactions may equal a much larger learner; a boosted tree may fit an exposure-sensitive nonlinear pattern more accurately; a hierarchical or credibility-based estimator may stabilize rare categories. Selection is an empirical decision conditional on the target and dataset, with structural and operational limits also assessed.

Likelihood benchmarks and count dispersion

For Poisson counts with mean μi>0\mu_i>0, the negative log likelihood omitting constants is ℓi(μi)=μi−yilog⁡μi\ell_i(\mu_i)=\mu_i-y_i\log\mu_i. An overdispersed count distribution may instead support a negative-binomial or other specification. A Poisson mean model can still yield useful point predictions under certain misspecifications; a misspecified predictive distribution will generally fail a full-distribution log-score assessment. The difference between correct conditional mean and correct conditional count distribution must remain explicit.

A frequency model cannot be declared superior solely because its in-sample residual deviance is lower. An unrestricted or flexible model can absorb random variation, particularly in sparsely populated risk categories. Information criteria, shrinkage, cross-validation, and untouched future-period evaluation address different manifestations of complexity. Arlot and Celisse's survey of model selection (2010) distinguishes properties of cross-validation procedures instead of treating all resampling designs as interchangeable.

AIC is 2k−2log⁡L^2k-2\log\hat L for the conventional likelihood setting, with kk the number of estimated parameters; BIC is klog⁡n−2log⁡L^k\log n-2\log\hat L under its usual assumptions. These criteria relate to model fit and penalties within a specified likelihood family. They are not universal cross-family measures when training samples, likelihood constants, observation units or target distributions are inconsistent. They also do not directly answer the operational question of predictive performance after a portfolio transition.

Candidate specification and fair optimization

Every candidate is a complete training procedure, not merely an algorithm label. The procedure includes data selection, feature creation, smoothing or regularization, tuning region, early-stopping rules, missing-value treatment, random seed policy and recalibration method. In a fair contest, candidate hyperparameters are tuned under precommitted, computationally comparable budgets. A sophisticated learner receiving extensive optimization cannot be fairly compared with a deliberately untuned GLM whose relevant interactions were excluded.

Noll, Salzmann, and Wüthrich (2020) provide a useful frequency modeling case study. The documented improvement of flexible methods in their setting partly reflects their ability to handle interactions that a simple comparator omitted. The legitimate conclusion concerns that empirical experiment. It invites further benchmarks against feature-engineered and regularized parametric alternatives rather than a categorical verdict that GLMs cannot capture interactions.

Henckaerts and colleagues' tree-based tariff evaluation (2021) similarly emphasizes that the insurance loss structure and fitting objective should be built into the learning comparison. Random forests, gradient boosting, and smooth regression are not interchangeable if the criterion being optimized differs or if frequency exposure is handled inconsistently.

Two separate comparison stages

Model selection can use repeated training-and-validation cycles, nested tuning, or other resampling designs appropriate to the data. Final evaluation must then occur on observations that did not influence the model family, feature set, hyperparameters, benchmark selection, or stopping decision. If investigators repeatedly revise the model after examining the designated test data, that set has become part of development. A new genuine test period or independent population is then required for an unbiased final performance statement.

A stable benchmark therefore records a selection phase and an assessment phase. The point is especially important in institutional research where many teams experiment against a common “public test” and gradually learn its peculiarities. Reproducibility requires recording which performance comparisons influenced model changes, rather than treating a held-out filename as proof of independence.

Selection Criteria

Predictive risk and objective alignment

Let L(y,f^)L(y,\hat f) denote a declared loss for an outcome and model forecast. The deployment risk of model mm is RP(m)=EP[L(Y,f^m(X))]R_{P}(m)=E_P[L(Y,\hat f_m(X))], where PP is the intended future joint distribution and f^m\hat f_m is produced by the complete training protocol. The sample test estimate is R^(m)=∑iwiL(yi,f^m,i)/∑iwi\hat R(m)=\sum_i w_iL(y_i,\hat f_{m,i})/\sum_i w_i for predeclared weights wiw_i. Weighting defines which observations and exposures the evaluation represents. Changing the weights changes the question and can change the winner.

If policy rows differ substantially in exposure, a per-policy average and a per-exposure average can rank candidates differently. An insurer must decide whether it is evaluating the average contract, the average insured-year, the average claim or the portfolio financial total. The answer belongs in the benchmark protocol before the candidate outputs are inspected. Comparability demands identical weights across models within a given target.

A point forecast and a complete predictive distribution can each be valid outputs, but their metrics differ. A model producing only an expected count is insufficient for a strictly proper assessment of the entire count distribution unless a distributional form is supplied. Conversely, a well-calibrated full distribution need not dominate a dedicated conditional mean estimator under every specialized operational loss.

Poisson deviance for claims

For nonnegative observed counts yiy_i and positive predicted counts μi\mu_i, Poisson deviance is

DP=2∑i{yilog⁡(yi/μi)−(yi−μi)},D_P=2\sum_i \{y_i\log(y_i/\mu_i)-(y_i-\mu_i)\},

with 0log⁡(0/μi)=00\log(0/\mu_i)=0. It is proportional to a log-likelihood contrast against the saturated Poisson count model, with equal observed counts and observation units. Smaller values indicate closer Poisson mean predictions on the particular sample. Constant terms and normalization should be handled identically in every comparison.

A common reported reduction is 1−Dcandidate/Dbaseline1-D_{candidate}/D_{baseline} where the baseline deviance is positive and computed on the same observations. This relative statistic can be useful within a dataset, but its denominator varies with claim variability and sample composition. It should not be interpreted as a universal percentage of causal risk removed, nor should it be directly compared across datasets with different definitions and aggregation. When the baseline deviance is zero or near zero, the ratio becomes undefined or unstable.

Poisson deviance handles the observed count outcome, not the commercial cost of a pricing decision. A frequency forecast can be excellent yet still yield inferior total-loss or profitability decisions if severities, policy constraints, market behavior and cost assumptions differ.

Probabilistic scores, calibration, and discrimination

Gneiting and Raftery's proper scoring-rule analysis (2007) establishes why the forecast scoring criterion should reward accurate probability statements. Under a loss convention, a proper score has minimal expected value when the predictive distribution matches the true distribution; strict propriety makes the optimum unique. The Brier score for a binary outcome is (p−y)2(p-y)^2, where pp is the forecast probability and y∈{0,1}y\in\{0,1\}. Brier's original probability-verification work (1950) supplies its historical origin. For a predictive distribution, negative log score evaluates −log⁡f(y)-\log f(y) or discrete probability −log⁡P(Y=y)-\log P(Y=y), and other distributional scores assess different aspects of prediction.

Insurance counts often have excess zeros and right-skewed positive values; forecast checking benefits from count-specific calibration tools. Czado, Gneiting, and Held's assessment of count forecasts (2009) develops proper scores and nonrandomized probability integral transform diagnostics for discrete outcomes. Their worked datasets are not insurance claims; the scoring theory is nevertheless applicable to correctly defined insurance count variables.

Calibration and discrimination answer distinct questions. Calibration evaluates agreement between assigned forecasts and realized event frequencies or loss levels. Discrimination evaluates relative ordering between riskier and less risky observations. A ranking model can separate high and low claim risk while overpredicting every group. A calibrated constant baseline may produce adequate portfolio totals while offering little differentiation. A robust benchmark displays both, tied to the decision target and appropriate credible groupings.

Probability calibration should be examined against exposure, time, key material segments and tail regions, rather than relying only on one global calibration curve. Sparse bins can generate visual artifacts; credible intervals or bootstrap uncertainty may be appropriate. Strong subgroup claims require enough actual exposure and a defensible multiple-comparison approach.

The uncertainty of comparison

Let di=LA,i−LB,id_i=L_{A,i}-L_{B,i} be the paired loss difference on the same test units. Under simple independent, identically distributed conditions, the sample mean estimates the expected paired difference and its uncertainty depends on the variance of did_i. Correlation among claims, repeated insured entities, weather events and time can invalidate an independence-based standard error. Cluster-based bootstrap, event-based resampling or appropriate time dependence modeling must match the evaluation structure.

Diebold and Mariano's predictive accuracy comparison (1995) explicitly treats tests built from loss differentials and permits serial correlation under relevant assumptions. Directly applying a time-series test to arbitrary insurance claim rows is a methodological category error. The dependence and data-generating structure must justify the inferential procedure.

Bates, Hastie, and Tibshirani's cross-validation analysis (2024) further distinguishes the risk of a specific final fit from risk averaged over possible training sets; it warns against naively narrow cross-validation intervals. Varoquaux's small-sample uncertainty analysis (2018) likewise demonstrates the danger of equating low fold variability with low uncertainty. This matters when a model wins a leaderboard by a tiny number of deviance points on a limited number of effectively independent events.

A selection report should therefore state sample size, exposure size, number of independent events or clusters, loss difference, sensitivity to splitting or time windows, uncertainty range where estimable, and number of models tried. If the observed gain is statistically unstable or operationally negligible, the correct finding may be that candidate superiority has not been established.

Practical utility and multi-criterion reporting

Predictive accuracy is one dimension of model fitness for purpose. Regulatory explainability, permissible rating factors, calibration stability, external fairness, operational latency, maintenance cost, and technical resilience can determine whether a predictive gain can responsibly support a specific decision. These are separate measured or assessed dimensions, rather than penalties to be collapsed into a mysterious unverified aggregate score.

An evaluation report should distinguish the primary statistical loss from a small set of secondary diagnostics: calibration, ranking, extremes, subgroup error, computational burden, reproducibility, and business impact under explicit assumptions. A candidate should not quietly change the primary metric after weak performance is observed. Where legal or contractual constraints disqualify an implementation, report that constraint transparently rather than claiming another model has greater pure predictive accuracy.

Complexity versus Accuracy

Flexibility, bias, variance, and effective information

Model complexity changes the tradeoff between approximation error and estimation variation. Simple models can miss interactions or nonlinearities; flexible models can learn patterns too closely tied to their development data. In insurance, the relevant information is not merely the number of policy rows. The amount and independence of claim experience, especially for rare events and large losses, determine whether complex patterns can be estimated reliably. Ten million mostly claim-free rows may provide limited direct evidence about the extreme tail of a catastrophe severity distribution.

This interpretation is compatible with the original algorithmic-modeling perspective and with actuarial attention to exposure and credible experience. Neither viewpoint makes algorithm complexity a scalar measure of scientific quality. A sophisticated model may provide genuine predictive benefit where there are many informative examples and meaningful interactions, yet be fragile across underwriting policy, economic regime, claims handling system or catastrophe exposure.

Constructed comparison I: a stable segmentation advantage

Consider a transparent hypothetical two-segment motor portfolio observed for one year. Segment A supplies 2,000 exposure-years and 100 claims; segment B supplies 1,000 exposure-years and 120 claims. The pooled annualized rate is (100+120)/(2000+1000)=0.073333(100+120)/(2000+1000)=0.073333 claims per exposure-year. The segment-specific training rates are 0.050.05 and 0.120.12. A constant-rate baseline forecasts 146.667 and 73.333 claims for the two exposure groups in an equal-duration future portfolio, while a two-cell model forecasts 100 and 120.

Assume a constructed future validation period has 90 claims in A and 100 claims in B, with the same exposures. Using the displayed Poisson deviance on the two aggregate counts, the constant-rate model has DP≈34.13D_P\approx34.13, and the segment-specific model has DP≈4.57D_P\approx4.57. The segment-based forecast is plainly superior under this chosen validation event, within the metric and grouping used. These figures follow directly from 2{ylog⁡(y/μ)−(y−μ)}2\{y\log(y/\mu)-(y-\mu)\}; no dataset was downloaded, no model was trained, and no statistical significance level is claimed.

This example clarifies what a good benchmark reveals: the single portfolio mean has discarded a real constructed distinction between groups. The calculation also shows why a global error statistic should be decomposed by segment. A serious empirical study would require uncertainty assessment over an independent sequence of such cohorts, not just the one illustrative realized vector.

Constructed comparison II: ranking reversal under shift

Keep the training history and both models unchanged, but suppose that future claim counts are instead 130 in A and 80 in B, again under 2,000 and 1,000 exposure-years. The pooled model retains forecasts 146.667 and 73.333; the segmented model retains forecasts 100 and 120. Their Poisson deviances become approximately 2.562.56 and 23.3423.34, respectively. The winner reverses because the constructed future event aligns more closely with the pooled forecasts.

The examples do not prove that pooling generally outperforms segmentation under shift. They demonstrate, by direct counterexample, that selecting a model from a single fixed evaluation regime cannot logically establish the same ranking for every future portfolio. Distribution shift can alter expected performance, and realized samples can reverse rankings even when the generating process is unchanged. These mechanisms must be distinguished empirically.

A more realistic protocol repeats training-origin and evaluation dates, logs premium and coverage changes, applies stable predeclared transformations, and measures whether the candidate remains well calibrated. The benchmark then reports both average performance and the distribution of performance across realistic regimes. Selection can favor the model whose improvement is stable and material, even if another wins narrowly on one data slice.

Tail risk, rare events, and financial outcomes

Rare, high-severity events generate another form of apparent accuracy. A model trained to minimize average squared error or deviance may perform well across ordinary claims but be insufficient for estimating extreme aggregate losses. Tail metrics, event scenarios and appropriate conditional distributional forecasts require separate evidence. A ranking based on frequency deviance must not silently stand for reserve adequacy, capital adequacy, policyholder solvency or profitability after reinsurance.

Similarly, the most accurate forecast of expected losses need not yield the highest realized revenue or lowest adverse selection in a live market. Premium decisions affect demand, retention and portfolio composition, making historical prediction and a counterfactual pricing intervention different objects. An insurer wishing to compare economic decisions needs a transparent model of rates, take-up, acquisition, expenses and behavioral response, validated or stress-tested for the relevant market. A purely retrospective loss-cost benchmark establishes retrospective predictive discrimination and calibration; it does not by itself estimate the causal value of replacing an entire tariff.

The legitimate place of interpretability

Interpretability supports model diagnosis, communication and professional judgment. A GLM coefficient with a well-specified link has a different explanatory structure from partial dependence or feature attribution derived from a flexible learner. Correlated variables and selection effects can distort causal interpretations for either family. Insurance rates may require additional documentary explanation and legal assessment independent of an algorithm's average predictive error.

The 2026 CAS insurance pricing treatment emphasizes the practical work of combining predictors and applying actuarial judgment. A defensible review records how the final candidate satisfies its purpose, whether proxies and restricted attributes have been examined under applicable law, and whether model sensitivity to inputs is acceptable for the product.

External Dataset Comparisons

What a public insurance benchmark can establish

Open benchmark datasets permit independent method development, diagnostic replication, and comparison under explicit common tasks. They cannot independently establish that a model will perform in every insurer's production portfolio. Dataset records may omit the business process that created the observations, information availability, coverage wording, market selection, or temporal evolution needed to reproduce deployment conditions. A public experiment should therefore be reported as an experiment on a named versioned dataset and precisely defined outcome.

The public freMTPL2freq record (OpenML 2018, dataset ID 41214) contains motor third-party liability risk characteristics, claim count and exposure fields. At the time of this investigation, its structured metadata states 678,013 instances, while its prose description states 677,991 policies. The two counts are recorded as a dataset-provenance discrepancy; the underlying data were not downloaded here to resolve whether cleaning, policy definitions or versioning explain the difference. An empirical paper must state which actual rows it loaded and why, rather than quietly choose one published number.

An experiment with this dataset must document the unit of observation, any duplicate or repeated identifiers, censoring of exposure, positive-exposure restrictions, transformations, losses, train/test construction, and version retrieval. A motor-frequency result on these policies tests an algorithm for that particular historical observed data distribution. Its external-validity claim is narrower than “boosting is better for insurance.”

Comparison with published motor studies

The Noll, Salzmann, and Wüthrich French motor study and the Henckaerts et al. tariff study provide evidence of methodological feasibility and potential gains under their chosen insurance targets and data. Their models, preprocessing, datasets, candidate families and evaluation protocols are not guaranteed to be identical. It would be scientifically invalid to create a single combined winner table by comparing published numeric scores from dissimilar denominators or populations.

A meaningful comparative literature synthesis first groups studies by target: claim count, positive severity, pure premium, claim occurrence, and full distribution. Within a target, it documents the effect of offset handling, risk factors, folds, observation period, calibration and candidate tuning. Where common raw data and code are available, a standardized rerun can yield a genuine shared benchmark. Until then, a narrative comparison can identify the conditions of each study and the robustness questions it leaves open.

The reported improvement of boosted trees in Henckaerts et al. (2021) is an empirical finding about their tariff-design study, whose analysis includes economic implications. It is not a numerical prediction for a new jurisdiction or product. Similarly, the favorable interaction behavior in Noll et al. (2020) invites benchmarks with explicitly engineered GLM interactions rather than treating a comparison against a basic GLM as the final frontier.

Benchmark families across exposures and sectors

The CAS Monograph No. 16 describes illustrative case studies involving aviation incidents, road accidents and simulated automobile insurance. The three settings clarify why data access and target comparability must be considered before ranking algorithms. Incident frequency, road-accident outcomes, and insurance claim costs reflect different event processes. Even when a common tree or regression family is used, the target variable and exposure construction change the scientific question.

An international benchmark should treat country, contract, underwriting environment, risk taxonomy and data collection regime as provenance variables. A common numerical score has meaning only where the outcome unit and prediction horizon can be aligned. For unrelated outcomes, multi-dataset evidence is more defensibly expressed as a set of within-dataset relative improvements, calibration patterns and stability findings, with a carefully disclosed aggregation rule if aggregation is attempted.

Time dependence and external validity

Random train/test partitions are often acceptable for a clearly formulated independent-record interpolation task, but annual policy pricing and long-term claims management often require forward deployment tests. Roberts et al. (2017) provide the broad structured-data validation framework. Bergmeir, Hyndman, and Koo's autoregressive cross-validation study (2018) also demonstrates that blanket rules against every K-fold design for time series are too broad: specific autoregressive conditions can justify it. The implication is to make validation conditional on forecast and dependence structure, not to use a single slogan in place of design.

For underwriting, a useful external check may be to train on prior underwriting years and evaluate policies bound in a subsequent year, with all variables fixed at quote time. For claims settlement, use information available at the reservation date and outcomes measured at a consistent development horizon. For catastrophe, leave-event-out or region-and-event stress tests can reveal sensitivity to dependence, although these may also be severely sample-limited. Each design is a different claim to generalization.

Regulatory and professional context

The U.S. ASOP No. 56 on Modeling (Actuarial Standards Board 2019; effective 2020) addresses model design, choice, fit for purpose, validation, controls and communication. This is a U.S. professional actuarial standard with its own applicability conditions, not an international legal requirement. It reinforces why a comparison must record the intended application and limitations of its output.

EIOPA's 2025 AI governance opinion is directed to national supervisors and interprets established European insurance-sector principles for AI in its designated scope. It discusses proportionate attention to data governance, documentation, fairness and oversight. EIOPA expressly distinguishes the scope of that opinion from AI systems governed as prohibited or high-risk under the EU AI Act; the opinion does not create a new universal model-accuracy threshold.

The NAIC artificial intelligence overview describes the United States' December 2023 model bulletin and subsequent state-level and regulatory work. A model bulletin is a reference instrument; state adoption and application require jurisdiction-specific examination. The relevant scientific conclusion across these documents is that statistical evidence supporting a candidate's predictive performance should be intelligible, traceable and capable of scrutiny. Institutional requirements differ by jurisdiction and cannot be consolidated into a single invented global standard.

Evidence Limits and Reproducibility

What was established and what remains conditional

The reviewed sources support the methodological importance of comparable estimands, exposure-aware prediction, baseline selection, genuine out-of-sample assessment, appropriate scoring rules, structured resampling and professional model controls. They establish that multiple classes of insurance learners can perform differently under stated conditions. They do not furnish a globally representative international experiment ranking every algorithm on every insurer, product and loss process. This monograph therefore makes no unsupported universal numerical performance claim.

The OpenML metadata discrepancy is a concrete example of why research reproducibility includes the provenance of the data record itself. A test that silently drops 22 observations relative to one dataset description can be defensible if the exclusion is explained, but its reported row count must be tied to the actual retrieved version. The discrepancy of 678,013 versus 677,991 is 22 records arithmetically; it is not by itself evidence of what caused the discrepancy.

Theoretical scoring statements can be verified independently of any insurance portfolio. The Poisson deviance formulas above follow from the Poisson log likelihood, while the two worked comparisons are reproducible by substitution of the displayed counts and forecasts. These constructed examples establish logically possible rankings and reversals. They are neither stochastic simulations with reported confidence intervals nor empirical tests of published insurance algorithms.

A reproducible comparison record

A complete comparison record identifies the task, policy and exposure population, observation period, risk factors, information cutoff, target, censoring and claim-development rules, sample unit, response treatment, and permitted feature set. It then identifies raw data versions, immutable preprocessing code and parameters, model families, tuning budgets, seeds, missing-data operations, training and validation windows, test protection, scoring functions, weights and final calibration procedure. Final outputs include the vector of predictions or a controlled reproducible substitute, paired score differences, data-set checksums, compute environment, and material exclusions.

For count predictions, the research record needs to distinguish the annualized rate from expected counts for the supplied exposure. For severity it must specify positive-only sampling, the amount definition and the effect of limits and recoveries. For pure premium it must document whether a two-part construction, a compound distribution or direct total-cost model was used. For any selected model, the training protocol is as important as the fitted coefficients or serialized object.

One useful reproducibility structure is an immutable sequence of: data receipt and checksum; data dictionary and provenance; task definition; baseline specification; candidate configuration; predetermined split and tuning log; final independent scores; uncertainty and sensitivity analyses; and review conclusion. The protocol is a reporting implementation of established scientific methods, not a newly proposed theoretical system.

Error analysis and the meaning of negative findings

Model failures should be reported with the same precision as gains. A candidate may overpredict low-exposure policies, underestimate a high-severity tail, depend on an unstable category encoder, fail during portfolio migration or achieve excellent aggregate scores while doing poorly for a legally or commercially relevant subgroup. Such errors are scientific information about scope and model design. Reporting only the mean benchmark metric can obscure them.

A scientifically meaningful negative result may be that the data do not distinguish two candidate procedures at the available precision, that a complex model is superior only on the training-like population, or that external calibration requires re-estimation. These conclusions support better model choice and more informative future research. Where a credible tail or subgroup check cannot be conducted because data are too sparse, the correct decision is to describe that limitation and its consequences for the intended use.

Locality of supervisory obligations and editorial responsibility

Professional and supervisory documentation should be interpreted within its exact type and jurisdiction. ASOP No. 56 is a U.S. actuarial professional standard; EIOPA's AI document is an opinion addressed to European supervisors; the NAIC model bulletin requires state-level status verification. Their combined relevance lies in governance themes, not a claim that all have identical legal force or impose a particular statistical algorithm. This distinction is necessary for an international encyclopedia that describes the field rather than supplies individualized insurance or actuarial advice.

The manuscript's bibliographic verification was performed by inspecting identified publisher, academic, professional-body and dataset records. For some sources, the verified object was an official abstract, contents or metadata rather than full text; those limitations are declared in the accompanying evidence ledger. There has been no independent computational rerun of the French motor studies, no new market dataset fitted, and no external peer-review report. Final approval of the manuscript package is a separate editorial and source-intake record, not a claim of production publication.

Research agenda arising from the synthesis

A valuable next empirical investigation would reconstruct multiple claim-frequency tasks from versioned open data, implement a fixed transparent baseline ladder and a common split plan, then report paired Poisson deviance and calibration under random, grouped, and forward-time tests. A related investigation would compare direct aggregate-cost and frequency-by-severity models with the same target and identical loss-weighting conventions. The protocols should include rare-claim uncertainty, group stability and the costs of monitoring and retraining.

Such investigations could test, rather than assume, whether any candidate superiority persists when exposures, periods and portfolio mixes change. The present study supplies the definitional and analytical conditions under which the resulting evidence would be interpretable. The ability to produce a ranked leaderboard is secondary to knowing precisely what has been ranked.

Conclusion

Model selection in insurance is a claim about predictive performance for a defined insured population, event process, observation period, information set, scoring rule and operational purpose. An insurance benchmark becomes scientifically comparable when all candidates share those conditions, meaningful baselines are included, tuning is separated from final assessment, and the evaluation observations represent the future question. Exposure-normalized rates, claim counts, positive severities, pure premiums and complete loss distributions require their own carefully specified assessment.

The literature supports sophisticated flexibility where nonlinearities and interactions are materially predictive, and it supports classical methods where transparency, stability, sample information or actuarial structure favor them. Published French motor and pricing studies demonstrate real applications of those choices; their reported successes remain attached to the datasets and protocols in which they were established. Statistical scoring theory supplies coherent losses, cross-validation research identifies the scope and uncertainty of resampled estimates, and professional standards connect model evidence to documented governance.

The constructed examples demonstrate that lower error on one future validation vector does not guarantee lower error on another: a segment-based model wins a stable illustrative portfolio while a pooled model wins the specified changed outcome. This is an exact arithmetic consequence, not an asserted new principle or an empirical forecast. The general consequence is that one leaderboard cannot substitute for tests of changing populations, clustered events, calibration, uncertainty and intended use.

The resulting scientific determination is operationally clear. An insurer or researcher should select a predictive model only after defining the insured quantity and evaluation population, protecting an independent assessment, reporting the primary loss and its uncertainty, examining the consequences of portfolio shift, and documenting controls and limitations applicable to the institution. The selected model is the best supported procedure for the stated task and evidence, and every wider performance claim requires its own corroboration.

References

  • Wüthrich, Mario V., and Michael Merz. 2023. Statistical Foundations of Actuarial Learning and Its Applications. Springer Actuarial. Cham: Springer. Open-access book, 605 pp. DOI: 10.1007/978-3-031-12409-9. Verified source.

  • Noll, Alexander, Robert Salzmann, and Mario V. Wüthrich. 2020. Case Study: French Motor Third-Party Liability Claims. Research paper, 41 pp.; initially posted April 18, 2018; revised March 5, 2020. SSRN abstract 3164764. Verified source.

  • Henckaerts, Roel, Marie-Pier Côté, Katrien Antonio, and Roel Verbelen. 2021. “Boosting Insights in Insurance Tariff Plans with Tree-Based Machine Learning Methods.” North American Actuarial Journal 25 (2): 255–285. DOI: 10.1080/10920277.2020.1745656. Published online July 13, 2020. Verified source.

  • Chalk, Alan, David Deacon, Montserrat Guillen, and Max Martinelli. 2026. From GLMs to Comprehensive Insurance Pricing: Techniques and Challenges. CAS Monograph No. 16. Casualty Actuarial Society. ISBN 978-1-7370028-9-5. Verified source.

  • Gneiting, Tilmann, and Adrian E. Raftery. 2007. “Strictly Proper Scoring Rules, Prediction, and Estimation.” Journal of the American Statistical Association 102 (477): 359–378. DOI: 10.1198/016214506000001437. Verified source.

  • Czado, Claudia, Tilmann Gneiting, and Leonhard Held. 2009. “Predictive Model Assessment for Count Data.” Biometrics 65 (4): 1254–1261. DOI: 10.1111/j.1541-0420.2009.01191.x. Verified source.

  • Arlot, Sylvain, and Alain Celisse. 2010. “A Survey of Cross-Validation Procedures for Model Selection.” Statistics Surveys 4: 40–79. DOI: 10.1214/09-SS054. Verified source.

  • Bates, Stephen, Trevor Hastie, and Robert Tibshirani. 2024. “Cross-Validation: What Does It Estimate and How Well Does It Do It?” Journal of the American Statistical Association 119 (546): 1434–1445. DOI: 10.1080/01621459.2023.2197686. Online 2023; journal issue 2024. Verified source.

  • Roberts, David R., Volker Bahn, Simone Ciuti, Mark S. Boyce, Jane Elith, Gurutzeta Guillera-Arroita, Severin Hauenstein, José J. Lahoz-Monfort, Boris Schröder, Wilfried Thuiller, David I. Warton, Brendan A. Wintle, Florian Hartig, and Carsten F. Dormann. 2017. “Cross-Validation Strategies for Data with Temporal, Spatial, Hierarchical, or Phylogenetic Structure.” Ecography 40 (8): 913–929. DOI: 10.1111/ecog.02881. Verified source.

  • Bergmeir, Christoph, Rob J. Hyndman, and Bonsoo Koo. 2018. “A Note on the Validity of Cross-Validation for Evaluating Autoregressive Time Series Prediction.” Computational Statistics & Data Analysis 120: 70–83. DOI: 10.1016/j.csda.2017.11.003. Verified source.

  • Diebold, Francis X., and Roberto S. Mariano. 1995. “Comparing Predictive Accuracy.” Journal of Business & Economic Statistics 13 (3): 253–263. DOI: 10.1080/07350015.1995.10524599. Verified source.

  • Breiman, Leo. 2001. “Statistical Modeling: The Two Cultures.” Statistical Science 16 (3): 199–231. DOI: 10.1214/ss/1009213726. Accessible copy from BioLab. Verified source.

  • OpenML. 2018. “freMTPL2freq,” dataset ID 41214, version 1. Uploaded by Christophe Dutang, November 5, 2018. OpenML data record; CC0 license. Verified source.

  • Actuarial Standards Board. 2019. Actuarial Standard of Practice No. 56: Modeling. Document No. 195. Revised December 2019; effective for work commenced on or after October 1, 2020. Actuarial Standards Board, United States. Verified source.

  • European Insurance and Occupational Pensions Authority (EIOPA). 2025. Opinion on Artificial Intelligence Governance and Risk Management. EIOPA-BoS-25-360, August 6, 2025. European Union supervisory opinion. Verified source.

  • National Association of Insurance Commissioners (NAIC). 2023–2026. “Artificial Intelligence.” Insurance topic and regulatory overview; notes December 2023 adoption of the Model Bulletin on the Use of Artificial Intelligence by Insurance Companies, subsequent state-level work, and 2026 supervisory evaluation work. United States. Verified source.

  • Varoquaux, Gaël. 2018. “Cross-Validation Failure: Small Sample Sizes Lead to Large Error Bars.” NeuroImage 180 (Part A): 68–77. DOI: 10.1016/j.neuroimage.2017.06.061. Verified source.

  • Brier, Glenn W. 1950. “Verification of Forecasts Expressed in Terms of Probability.” Monthly Weather Review 78 (1): 1–3. DOI: 10.1175/1520-0493(1950)078<0001:VOFEIT>2.0.CO;2. Verified source.