Insurance by BogdanovaThe Encyclopedia of Insurance and Risk Science

RESEARCH DOSSIER

Insurance Predictive Training Design and Validation Splits

Abstract. Training, tuning, and test separation establishes the evidentiary meaning of predictive performance in insurance. Insurance records are generated by contracts, people, insured objects, transactions, reporting systems, and events observed across overlapping time intervals. These structures make an apparently independent row an unreliable unit of evaluation. This monograph develops an integrated methodological analysis of validation design for claim occurrence, frequency, severity, pure premium, claim development, lapse, and other insurance outcomes. It defines the operational forecast and its information set, distinguishes conditional interpolation from genuine future deployment, derives the implications of entity dependence and time-varying claim maturity, examines rolling-origin and nested evaluation, and evaluates exposure imbalance, tail instability, and cross-portfolio transfer. A formal split-admissibility argument shows why chronological ordering alone cannot establish independence when labels and shared events mature after the forecast origin. The work provides explicit design equations, auditable split specifications, worked hypothetical examples, failure diagnostics, an implementation-neutral algorithm, and a reporting protocol. Its central conclusion is that the validation design identifies the population and decision process to which a performance estimate applies. The evidence supports disciplined scenario-specific evaluation, rather than a universally optimal split percentage or algorithm. All numerical illustrations are mathematical demonstrations, not insurer-data findings. This is a methodological research synthesis; no new named theory or empirically established superiority claim is advanced.

Keywords: Insurance prediction; cross-validation; temporal validation; data leakage; policyholder clustering; claim development; event grouping; rolling forecast origin; nested validation; risk estimation; exposure weighting; out-of-time testing; generalization; actuarial modeling; model governance; reproducibility.

Introduction

A predictive study claims that a rule learned from observed contracts will produce usable information for a different operational decision. An insurance model trained on rows of a warehouse does not automatically inherit the meaning of that operational decision. Insurance observations share household members, vehicles, policies, buildings, brokers, geographic exposures, catastrophe events, administrative revisions, and reporting calendars. Claims may be reported well after occurrence and settled years afterward; incurred amounts can change as later information is attached to the same originating claim. A technically legal random split can therefore answer a narrow question about interpolation among already known entities while producing a misleading estimate for a future underwriting or pricing cohort.

The general problem is described in Kaufman et al.'s analysis of target leakage, which identifies the entry of unavailable target-related information into predictors and the analytic process. Roberts et al.'s review of structured cross-validation demonstrates, in another field, how temporal, spatial, and hierarchical dependence can make conventional random cross-validation too optimistic. Their mechanisms are transferable to insurance as design warnings; their empirical domain is ecology and their quantitative findings should not be presented as insurer-specific measurements. The objective here is to establish an insurance-specific interpretation of these principles.

An evaluation has three logically separate commitments. It specifies the target quantity and permissible observation history; it specifies the unit on which independent evidence can be obtained; and it specifies the deployment population and time horizon over which error is to be evaluated. None can be inferred from a generic instruction to divide records 80/20. Cross-validation is a sampling operation on the available data-generating structure. When its partitions change, its scientific estimand changes.

This monograph examines predictive evaluation, rather than choosing the best machine-learning architecture or providing a universal pricing rule. It compares split mechanisms under explicit assumptions and develops reproducibility requirements for research and supervisory communication. The CAS insurance GLM monograph and the CAS comprehensive-pricing monograph supply insurance modeling context, while the statistical literature establishes the inferential basis of performance evaluation.

Training and Testing Roles

The estimand comes before the partition

Let an insurance record be indexed by observation or contract unit i, with features X_i(t), outcome Y_i(t,h), measurable exposure e_i(t,h), contract/entity identifiers G_i, and an observation origin t. The prediction horizon h might be one policy year, 90 days of lapse, 12 months of reported claim payments, or an ultimate-cost target assessed through development. Let F_t denote information legitimately available to the decision maker at time t, including actual arrival and processing delays. A forecast function f_t must be measurable with respect to F_t. A variable recorded later or a retrospectively revised claim amount cannot be used merely because its database timestamp precedes the validation extraction date.

For a prospective target population P* and a declared loss L, predictive risk can be written as

RP∗,t,h(f)=E(X,Y,e)∼Pt,h∗[L{Y,f(X)}∣It],R_{P^*,t,h}(f)=\mathbb E_{(X,Y,e)\sim P^*_{t,h}}[L\{Y,f(X)\}\mid \mathcal I_t],

where I_t denotes the permissible operational information set. This expression makes the evaluation target explicit: different policy populations, forecast origins, loss horizons, and available data histories imply different risks. The conditional notation does not assume that the future distribution equals the historical one. To estimate risk from an evaluation sample, investigators must additionally state how the sample relates to P*, what dependence persists, and which weighting rule is justified.

A claim-frequency model trained for annual renewal pricing may legitimately rely on fully developed claims from prior policy years; an early-warning model intended to operate on day 14 after first notice has a substantially smaller information set. A severity predictor used when a claim is opened must not use its final paid amount as a covariate. Whether a measure is a feature or a label is therefore a function of prediction origin, not just column name.

Training, tuning, audit, and final testing

The training partition estimates model coefficients and all learned preprocessing. The tuning or validation partition chooses hyperparameters, feature transformations, early stopping, or a candidate family. The audit partition, when maintained separately, evaluates design compliance and fairness or stability against predeclared requirements. A final locked test partition estimates predictive loss for the already frozen development process. Independent test use requires that the final test neither influence the candidate search nor become an informal source of model redesign.

A practical research project may use inner cross-validation instead of one static tuning set, outer folds for design-level assessment, and a terminal out-of-time holdout for the most relevant deployment check. These roles can be combined differently when data are scarce, but their information flows must remain explicit. Varma and Simon (2006) demonstrated selection-induced optimism in cross-validation when the same cross-validation estimates are reused for selecting a classifier and reporting its performance. Cawley and Talbot (2010) explain overfitting of selection criteria even when the original estimator seems nearly unbiased. The methodological consequence is to evaluate the entire adaptive selection procedure on evidence unused for that selection.

Three distinct claims of generalization

Within-entity interpolation asks how the model handles another row of a policyholder, claim, or contract that is otherwise observed. New-entity generalization asks how the model works on people, assets, or counterparties never present during training. Future-time generalization asks how the model operates after the training cut-off under changes in underwriting, inflation, claim handling, exposure mix, and event environment. These questions require different test designs. A study should identify whether its primary claim is conditional interpolation, prospective renewal risk, a new-market launch, or a stress scenario.

For example, if a 2022–2025 motor portfolio contains monthly extracts for each automobile, randomly splitting rows may measure recognition of already observed vehicles and their histories. Assigning each vehicle permanently to one side instead tests generalization to unseen vehicles, but mixing origin dates still prevents a direct claim about 2026 deployment. Splitting by decision dates tests future-time performance, but leaving the same vehicle in both partitions can be legitimate if operational deployment predicts renewal of previously insured vehicles from permissible past history. The study must state precisely which combination it wants.

Loss functions and weighting are part of the target

An unweighted record mean Lbar = n^{-1} sum_i L_i answers the question about a randomly selected recorded row. A policyholder mean, an insured-risk mean, an exposure-weighted loss, and a premium-volume-weighted loss answer other questions. With annualized exposure e_i >= 0, a common exposure-weighted diagnostic is sum_i e_i L_i / sum_i e_i; whether this is scientifically suitable depends on how the target and model output are defined. If Y_i represents aggregate claim cost over an observed partial year, comparing raw totals across heterogeneous exposure durations confounds claim risk with observation time. Conversely, scaling rare extreme-loss outcomes by a tiny exposure can magnify variance.

For probabilistic predictions, the proper score must match the target event or distribution. Gneiting and Raftery (2007) provide a theory of proper scoring rules; properness addresses incentives in predictive distribution reporting, while sample independence, score aggregation, and decision relevance remain separate validation choices. A score computed correctly on the wrong partition does not justify prospective generalization.

Data extraction, as-of versions, and eligible population

A fully specified dataset has an observation eligibility rule, a source system, a freeze date, a reconstructible as-of query, and a label maturity definition. Policies cancelled before inception, rewritten midterm, transferred across carriers, or linked to group programs require documented treatment. Claims with multiple payments need coherent claim-level aggregation; duplicate extracts of the same claim must never count as independent labels. Missing claims are not automatically true zero losses: reporting lag, file closure, and residual development must be analyzed separately.

In the United States, Actuarial Standard of Practice No. 23, Data Quality, offers professional guidance concerning selection, review, use, and disclosure of data for actuaries subject to its professional scope. It does not prescribe a universal machine-learning train/test ratio. Its relevance is the need to understand data sufficiency, suitability, limitations, and reliance on suppliers before treating a split as probative.

Policyholder and Claim Group Leakage

Dependence is attached to entities and events

Insurance data are naturally multi-indexed. A policyholder can hold several policies; one vehicle can appear across successive renewals; one building can be insured by several layers; one occurrence can generate multiple coverage claims; one storm can generate thousands of policy-level losses. Claims observations can also contain repeated payments and reserve updates. If rows in different partitions share a latent event or entity characteristic, errors are dependent even when no column directly copies the target.

Let c(i) be the connected component of record i in an entity-event graph whose nodes represent policies, persons, assets, claims, events, transactions, and data source versions. An edge means that the same outcome information, a shared event realization, or an intentionally excluded latent unit links two observations. A strict new-component test requires that no connected component intersects both training and test indices:

{c(i):i∈T}∩{c(i):i∈V}=∅.\{c(i):i\in T\}\cap\{c(i):i\in V\}=\varnothing.

This constraint is stronger than merely grouping by policy number. It can also be unnecessarily strict for tasks involving legitimate known-customer history. The choice of component boundary follows the deployment claim and is recorded before results are inspected.

Repeated monthly extracts of the same accident claim, for example, may show an open-reserve estimate in January, a payment in February, a revised incurred amount in March, and closure in June. A row-level split could place January in training and June in test; the model then recognizes the claim's identity or revised values without having produced an independent claim prediction. The scientifically meaningful unit may be the claim at a fixed development age, with snapshots organized around the permissible information origin.

Claim leakage, label leakage, and processing leakage

Claim leakage occurs when multiple representations of the same underlying claim are treated as independent evidence, or when the same catastrophe event introduces a shared realized loss pattern into both sides of a test intended for new-event generalization. Label leakage occurs when a predictor uses settlement dates, ultimate-paid information, post-decision reserve movements, fraud outcomes, or final claim classifications unavailable at forecast time. Processing leakage occurs when transformations are fitted on the combined dataset: target encoders, feature screening, imputation medians, percentile cutoffs, frequency maps, and learned embeddings can all transmit information from the test period or test population back into training.

The scikit-learn documentation discusses preprocessing data leakage and explains why parameters of data-dependent transformations are fitted within the training portion. In insurance, this rule extends to domain features. A historical territory risk factor estimated from all accident outcomes, including the held-out year, is a learned outcome-dependent statistic even if it appears in a convenient lookup table. Each fold must recompute it using only the corresponding eligible training observations and, for time-respecting designs, only information released by the fold's origin.

A four-way leakage inventory is useful for reviews: (a) direct target proxy, (b) entity overlap, (c) future-information arrival, and (d) test-informed adaptation. This is an analytical checklist of established mechanisms rather than a new named conceptual system. All four can coexist; resolving one does not resolve the others.

The correct unit depends on the decision

For new-customer acquisition, grouping by person or household can be decisive; for property-catastrophe prediction, grouping by physical occurrence or catastrophe identifier can test event generalization; for claim triage at first notice, grouping by claim prevents multiple updates from crossing folds; for renewal pricing, repeated observations of the same insured may be the intended longitudinal use case. Where the commercial aim is to transfer a model to another insurer, the independent unit may be portfolio or source organization, not policyholder.

A concrete evaluation table clarifies this distinction.

Deployment question Primary separation Additional exclusion Performance statement
Renew existing policies next year Decision-time origin Post-origin features and immature labels Future renewals for previously observed customers
Price never-seen risks Person/asset or linked entity Future values and connected household/asset exposure New-entity transfer under the sampled market
Handle next catastrophe season Origin dates plus event IDs Shared-event labels across partitions New-event seasonal generalization
Predict claim ultimate cost at first notice Claim ID and development-age cut Post-notice payments, reserves, closure and outcome labels New-claim early prediction
Transfer to a second insurance portfolio Portfolio/source organization Local post-hoc recoding based on test outcomes Cross-portfolio transport under observed differences

Grouping algorithms are implementation aids. The documented GroupKFold partitioner prevents a group from appearing in both training and test folds, but it does not automatically impose a chronological boundary or an as-of information rule. A proposed design that requires both group exclusivity and temporal ordering must implement both explicitly.

Why grouped random folds can still be misleading

Suppose a claim model is trained on independent claim identifiers, using policy information gathered retrospectively into a 2025 extraction. Even perfectly disjoint claims may have features revised in 2025 after the date at which historical claim predictions would have been made. Group disjointness thus does not restore historical information availability. Likewise, two claims caused by the same flood are separate claims but share the latent event shock; their group assignment should depend on whether the study aims to predict future events or interpolate claims within a known event.

Even where no target or information leakage exists, test clustering affects uncertainty. If m claim outcomes share an event-level random effect and intraclass correlation rho, the heuristic equal-cluster design effect is 1+(m-1)rho; the corresponding effective count is approximately n/[1+(m-1)rho] under equal clusters and a common correlation structure. This familiar cluster-sampling calculation illustrates why ten thousand catastrophe-related rows cannot be treated as ten thousand independent realizations. Unequal events, common geographic exposures, and complex claim-size dependencies require more tailored variance estimation.

Rolling Time-Origin Validation

Two clocks and a prediction horizon

Insurance evaluation must distinguish occurrence time, prediction time, feature availability time, claim report time, and label maturity. In a simple annual model these clocks are often compressed into one policy-year column. That compression fails whenever the target is known later than the exposure or the record is updated between origin and assessment. For observation i, define t_i as the prospective decision origin, a_i as the earliest date its usable outcome label becomes available, and x_ij as a candidate feature with availability date v_ij. A legally reproducible historical forecast requires v_ij <= t_i for every feature used. For a training row to be included in a model fitted at origin o, a_i <= o must also hold, unless an explicit censored-label procedure is used whose observed portion is available by o.

The primary chronology condition for a rolling evaluation is

max⁡i∈T(o)ai≤o,min⁡j∈V(o)tj>o,max⁡k∈features(j)vjk≤tj.\max_{i\in T(o)} a_i \le o,\qquad \min_{j\in V(o)} t_j > o,\qquad \max_{k\in features(j)}v_{jk}\le t_j.

This definition permits a training claim with occurrence time well before o only when its required label was also known by o. It excludes a claim that occurred before o but whose ultimate amount became known afterward. It also requires an agreed time horizon h and a rule for measuring outcomes in the test window. A terminal test cannot be evaluated faithfully until its labels have matured to the declared target or a valid delayed-outcome estimation method is specified.

The widespread rolling-origin approach is described in Hyndman and Athanasopoulos's time-series cross-validation discussion. It uses successive forecast origins, with later observations reserved for evaluation. The method is particularly relevant when the research claim is forward deployment rather than random interpolation. Cerqueira, Torgo, and Mozetič (2020) compare forecast evaluation designs and report that the behavior of blocked and out-of-sample approaches depends on stationarity and empirical conditions; their conclusion is not a theorem that every insurance portfolio must use exactly one split type.

Expanding, sliding, and fixed-origin tests

An expanding-window design fits at origins o_1 < o_2 < ... on all eligible earlier history and evaluates the next predetermined window. It reflects institutions that accumulate experience and refit periodically. A sliding-window design limits training to the latest w years or months, sacrificing some sample size to reduce reliance on obsolete regimes. A fixed-origin terminal test trains once on data available at o and evaluates a later interval without refitting; it resembles a frozen deployment. All three are defensible for different operating procedures.

For a concrete hypothetical annual comparison, consider fully matured labeled years 2018–2024 and target evaluation year 2025 with labels mature by an agreed 2026 review date. An expanding sequence may train on years through 2021 and evaluate 2022, then train through 2022 and evaluate 2023, and so forth. A three-year sliding design may train on 2019–2021 for evaluation in 2022, then on 2020–2022 for evaluation in 2023. A locked holdout design uses all eligible information before 2025 for a one-time evaluation on 2025. In all cases, features, claims labels, and inflation adjustments must be reconstructed as of the respective fit and forecast origins, rather than as known on the final review date.

A deployment model refitted every quarter requires evaluation of the refitting procedure itself: each origin must reproduce the data extraction and preprocessing that would have been available then. A study reporting a single 2025 test result for an annually refitted 2026–2028 deployment strategy has a narrower evidence base than one with multiple forecast origins and several underwriting cycles.

Embargoes, gaps, and claims runoff

A temporal gap or embargo may be needed when exposure intervals overlap or data become available only after a delay. A gap measured as a fixed number of rows is insufficient when policies vary in start dates, duration, and reporting lag. Instead define the exclusion zone relative to dates, label availability, information release and linked outcomes. If exposures starting in November 2024 extend through October 2025, evaluating a 2025 policy year requires careful handling of overlapping policy intervals even if records are sorted by inception month. If a storm struck in December 2024 and claim settlement appears in March 2025, split assignment must prevent that same event's realized loss from crossing a boundary intended to represent an unknown future event.

The software implementation of TimeSeriesSplit provides ordered folds and a configurable gap, and its documentation notes conditions for comparable equally spaced samples. Its sample-index gap is not automatically a claim-development embargo, catastrophe-event exclusion, or event-time cutoff. Insurance-specific business-time semantics must be implemented and tested above the library interface.

An embargo that simply discards the final six months before the test window can reduce direct adjacency but does not solve retrospective label leakage if the training labels include ultimate losses estimated years after the origin. Conversely, discarding a year may be unnecessary for fully mature, independent contracts with fixed features and immediate outcomes. The gap has a causal or information-flow justification when it corresponds to a dependency and availability mechanism. It is selected from the mechanics of the data, rather than from a grid search on the test score.

Reporting and development lags

Let Y_i^ult be eventual ultimate loss for claim i and Y_i(o) the observed amount or best available estimate at evaluation origin o. If the chosen target is ultimate loss, Y_i^ult is not generally available for recent claims. Three options serve different inferential purposes. First, wait until an explicit development maturity threshold is met and report the resulting older-cohort target. Second, evaluate at a fixed development age, in which case the target is deliberately limited-development experience. Third, use a specified reserving or censoring model that estimates ultimate outcomes and propagates its uncertainty, while recognizing that such a test is partly a comparison to constructed labels. Calling a reserve estimate the observed ground truth obscures that distinction.

For claims prediction at first notice, features require their own development-age cut, and comparisons should be made at like observation ages. A more recent accident period with immature paid losses can appear deceptively inexpensive; a model may seem to underpredict mature periods and overpredict new ones because the outcome definition changes across folds. Paid-plus-reserve incurred values are sensitive to claim-handling and case-reserving practices, especially if those practices change between periods. The validation report must record development date, valuation date, coverage basis, currency and inflation treatment, and source of ultimate estimates.

When conventional time-series k-fold may be valid

Temporal splitting is a design recommendation matched to prospective insurance deployment, not a blanket statistical prohibition on other methods. Bergmeir, Hyndman, and Koo (2018) show conditions for autoregressive k-fold validation in particular purely autoregressive settings with uncorrelated errors. This result does not extend automatically to insurance policy rows with delayed labels, recurrent claims, event clusters, renewal selection, or nonstationary risk mix. A researcher invoking it should demonstrate that the model and sampling conditions actually correspond to the theorem's conditions and the claimed operational target.

If a regulator or reinsurer asks whether a model can be applied to new but contemporaneous risks sampled from an exchangeable portfolio, grouped random folds may be informative. For predictions about the next underwriting year, out-of-time evidence is normally the primary validity claim. Independent future periods and independent held-out entities often expose different failure modes and can be reported as complementary tests.

Deliberately hypothetical date example

Suppose the target is 12-month claim frequency for policies effective January 1, 2025. The intended prediction is made December 15, 2024, and allowed features are frozen by that date. A claims data mart extracted September 30, 2026 may include previously unreported 2025 claims, revised fraud flags, and 2025/2026 payment changes. Selecting records by effective date alone still allows an analyst to use those later values in predictors or in tuned feature aggregates. Reconstructing December 15, 2024 feature snapshots, using only labels available for training at that date, and postponing evaluation until an appropriate outcome maturity date eliminates this example's direct information-timing violation. It does not automatically prove group independence, portfolio representativeness, or stability under other economic conditions.

Nested Validation Structure

The adaptive procedure is the model under evaluation

In modern insurance studies, the completed prediction rule is usually the outcome of many choices: candidate covariates, imputation rules, family or loss distribution, feature engineering, regularization, hyperparameter search, calibration, manual actuarial constraints, and business review. Each choice can exploit validation noise when its performance is repeatedly observed. A trial sequence that tests 300 candidate models on the same future holdout is adaptively fitting that holdout even if no gradient passes through its labels.

Formally, let A(D_T; lambda) be a model trained on training data at hyperparameter lambda, and let lambda_hat(D_T,D_V) minimize an inner validation estimate. The operational procedure returned by model development is A(D_T; lambda_hat), not the model at a hypothetical prespecified lambda. A valid outer evaluation fits all development choices using only outer-training observations and evaluates the resulting selected procedure on outer-test observations. The two-stage separation follows the logic of Varma and Simon's model-selection study and Cawley and Talbot's selection-bias analysis.

The dependence between training and selection is permitted within an outer fold; the prohibited information flow is from that fold's test labels into its development process. When outer folds are time-based, all inner folds must use only the earlier outer-training period, and must reproduce feature availability as of their own origins. If a model is compared against a prior incumbent, both must be evaluated under the same origin, target, exposure denominator, and allowable-information rules, even when one has far fewer tunable parameters.

Nested grouped and temporal design

One robust design for future claims uses a terminal locked 2025 cohort and development years up to 2024. Within the development period, several outer forecast origins assess the tuning procedure across business cycles. Inside each outer-training set, inner origins tune hyperparameters and learned preprocessing. Each candidate pipeline is fitted anew within each inner fold; feature ranking and target encoding cannot be computed on all earlier-plus-inner-test data. After an outer evaluation, that outer test fold can enter the eventual development history for a later outer origin if the actual operational refitting policy would then have legitimately observed its mature labels. It cannot enter backward into an earlier fit.

Where new-event generalization is the main target, nested folds additionally exclude event IDs across training and testing. For storm-driven home claims, it may be computationally difficult to satisfy temporal, event, and location constraints simultaneously. The research design should first specify which constraints are mandatory for the claimed deployment situation and then acknowledge small effective sample sizes. A forced stratification that breaks catastrophe-group exclusivity has changed the scientific question and is not a neutral optimization.

The StratifiedGroupKFold interface can approximately preserve class proportions while keeping groups disjoint, but its documentation says balancing can be constrained by group composition. The function does not enforce a forecast origin and is inappropriate as a substitute for chronological eligibility tests.

Where calibration belongs

Probability recalibration, severity scaling, credibility shrinkage, blend weights, and manual rate-level corrections are model-development operations. If calibration learns from outcome labels, it is fitted entirely within the relevant development data, possibly through cross-fitted predictions. The outer test assesses the calibrated predictor. Choosing a calibration method or a probability threshold after inspecting final-test outcomes invalidates the claim of an untouched final test. Estimating business costs or constraints from external fixed sources is compatible with independent evaluation when those sources were available by the decision origin and were fixed before test review.

A calibration curve plotted on an outer test can describe performance without affecting it. If it motivates another training iteration, that outer test should be recorded as development feedback, and the next scientific estimate should use fresh independent evidence. This simple record prevents a circular history in which every model release is 'validated' against the same continuously consulted holdout.

Comparison of candidate algorithms

A paired comparison on the same test policies often has lower variability than comparisons on different random test policies, because common weather, underwriting, and business shocks affect both models. Let d_i=L(Y_i,f_A(X_i))-L(Y_i,f_B(X_i)). A simple mean of d_i estimates average comparative loss for the sampled target units; a credible uncertainty calculation must account for clustering, serial correlation, unequal weighting, and any selection of the candidates using this same test. Diebold and Mariano (1995) developed forecast accuracy comparison methods under conditions allowing serially correlated forecast errors. Translating the method to insurance requires examining its assumptions, horizon overlap, and the unit of the loss differential.

Nadeau and Bengio (2003) show generalization-error inference limitations when variability from changing training sets is ignored. Thus, variance computed from row-wise fold scores, divided as though fold estimates were independent, is rarely an adequate uncertainty statement for overlapping rolling windows. Confidence intervals should specify what random mechanism they cover: sampled policies conditional on the historical years; future independent catastrophe events; or variation arising from retraining across time. One interval cannot silently claim all three.

Design matrix for choosing an evaluation

Scientific target Development-stage strategy Final independent evidence Critical limitation
Contemporaneous new policyholder Nested group-disjoint folds Unused group-disjoint holdout Later temporal drift remains untested
Next policy-year claim frequency Nested rolling origins with as-of labels Locked future cohort One future period may be atypical
New catastrophe event Event-disjoint temporal origins New event season or independent events Few independent extreme events
Transfer to another insurer Portfolio-disjoint development Unseen insurer/market source Commercial and regulatory transport conditions vary
Continuously updated claim triage Claim-disjoint, as-of snapshot replay Later claim-arrival batches Operational feedback and outcome revisions
Frozen one-time credit or lapse model Fixed training and tuning window Untouched later cohort Refit benefits are outside estimated performance

The table describes possible research architectures; it is not a proof that any architecture has a particular risk value. A definitive validation claim requires the actual partition inventory, code, output, limitations and appropriate statistical uncertainty analysis.

Distribution and Exposure Imbalance

The test sample represents a population, not merely a file

Even when a test partition is independent of development, its empirical loss describes the risks that it contains. The distribution of vehicle types, insured values, limits, deductibles, territories, claim reporting practices, and exposure durations may differ sharply from those expected after deployment. Exposure weighting cannot repair every discrepancy. A simple representative risk estimate depends on selection into the observed data and the way the target population is defined. If a renewal cohort excludes customers who were rejected or did not renew, its validation estimate cannot be extrapolated without argument to all applicants.

Let P_S be the sampled test distribution and P* the intended deployment distribution. Under covariate shift, if the conditional outcome relationship P(Y|X) is invariant and P* is absolutely continuous with respect to P_S, the population risk may be expressed as an importance-weighted expectation:

RP∗(f)=EPS[p∗(X)pS(X)L{Y,f(X)}].R_{P^*}(f)=\mathbb E_{P_S}\left[\frac{p^*(X)}{p_S(X)}L\{Y,f(X)\}\right].

This identity depends on the invariance and support assumptions. Neither is guaranteed in insurance: coverage wording changes, claim inflation may alter severities conditional on vehicle or territory characteristics, climate hazards may change event processes, and shifting case management can modify observed loss labels. Weights estimated from a small portfolio may also be highly unstable. Reweighting is a sensitivity analysis, not evidence of universal transportability.

A scientifically useful split report therefore presents at least the distribution of exposure, policy count, policyholder count, claim count, event count, zero-claim proportion, average outcome, incurred and paid basis, and key coverage/territory strata for each partition. It should report whether transformations, source exclusions, policy rewrites, and selection rules changed the composition across time. When a portfolio is very small, confidentiality may require aggregate or suppressed statistics, but the reviewer still needs a description of the material differences.

Rare events and the false comfort of balanced classes

Severe claims and fraudulent claims may be too rare for a nominal 20% test sample to contain enough independent cases for stable estimates. Artificially balancing positive and negative examples in the test set changes event prevalence, which matters for calibrated probability and positive predictive value. An investigator can deliberately oversample rare examples for diagnostic precision, but must label the resulting test design and use appropriate population weighting when evaluating population-level probabilities or costs. If the sampling probability depends on the outcome, the simple unconditional mean of test scores generally estimates the sample design rather than the real portfolio.

Consider 10,000 independent eligible policies with a one-percent annual claim-event rate. A random 20% test contains 2,000 policies and an expected 20 claims. Its event count can vary appreciably, and confidence bounds for many rare-event performance statistics will be wide. If those claims belong to three weather events, the independent-event count is only three for new-event inference. The difference between expected row count and independent-event count matters far more than the nominal 20% allocation.

Group-aware stratification can reduce empty folds but may distort the process if it forces rare catastrophe labels equally across each test year. Genuine season-to-season variation is itself part of the performance question. The scientist should report natural prevalence and separate optional balanced challenge tests. Test sets can intentionally include severe-stress portfolios, but such stress results need a clearly distinct interpretation and cannot replace the representative primary estimate.

Exposure denominators, multiple coverage layers, and duplicate claims

An insured asset may have multiple policies, layers or endorsements covering the same underlying occurrence. Raw policy rows and a unique-asset count represent different units. For a frequency estimand per exposure-year, policy exposure must reflect the denominator aligned with the outcome, including coverage effective dates and cancellation. For a liability excess-layer study, event claims below attachment may be recorded differently from attachment-layer claims, generating truncation. For joint home and motor portfolios, correlated household claims can be meaningful to aggregate capital management while requiring different group units for product-specific pricing.

An evaluation design should document whether repeated coverage layers are considered distinct contract obligations or correlated records of one event, and whether policy-year exposure is earned, written, or observed to date. If raw claim count is predicted from partial exposure, offsets or rates may be needed, but their choice belongs to the target specification. Fold balance assessed by rows alone can mask an extreme imbalance in earned exposure or total insured value. The split manifest should record both unit counts and the economically meaningful denominators.

Drifting risk and mixture decomposition

A reported change in average loss can arise from a change in portfolio mix, a change in conditional risk, a change in reporting and settlement, or a change in inflation and currency. These processes have different effects on a validation estimate. Let P_t(X) denote the portfolio distribution in period t and m_t(X)=E[Y|X,t] the conditional risk. The period mean is integral m_t(x)dP_t(x). A difference between periods can reflect both P_t and m_t. Merely stratifying on static age or region does not establish invariance of m_t, and a fixed model may fail because the relation between geography, climate and loss has changed rather than because the test set was too small.

Time-based validation should separate diagnostics of covariate changes from predictive loss changes. Reporting standardized risks under a reference portfolio can clarify changes in mix, but it requires documented weights and leaves residual uncertainty about unmeasured variables. Source documents from the CAS pricing-validation discussion emphasize sound validation and avoidance of overfitting in actuarial applications. This supports reporting controls and model judgment; it is not a license to claim that any particular geographic or telematics model is valid in a country or portfolio not examined.

Cross-jurisdictional and regulatory interpretation

Validation design also becomes part of model governance when predictions influence pricing, underwriting, claims, fraud investigation, and customer treatment. The National Association of Insurance Commissioners' 2023 AI Systems Model Bulletin describes governance expectations in an adopted model text. Its status as an NAIC model means that binding legal effect and practical expectations depend on action by the relevant U.S. jurisdiction; it must not be described as a single nationwide insurance law. Its attention to life-cycle validation and monitoring explains why split provenance is an audit concern.

In the European supervisory context, EIOPA's 2025 Opinion on AI governance is addressed to national supervisors and interprets existing sectoral principles for the relevant AI applications. EIOPA states that the Opinion is risk-based and proportionate, and its explanatory release clarifies its scope relative to high-risk and prohibited AI systems under the AI Act. It should be cited as a supervisory Opinion in its specific context, not as a directly binding worldwide statistical standard. The shared methodological implication is that insurers need reviewable training data provenance, validation rationale, documented limitations and monitoring without converting regulatory source language into an unqualified mathematical mandate.

Performance Stability and Reporting

Multiple dimensions of stability

A validation report should distinguish: stability across resampling seeds; stability across independent entity or event groups; stability across forecast origins; stability across lines of business and geographic strata; stability across claim maturity definitions; and stability across operational changes such as new forms or revised case handling. Good performance on one chronological test year may coexist with instability under a new loss-development view. The test result is evidence about the declared environment, and its precision must be evaluated relative to the effective units that generate uncertainty.

Let \hat R_k be a score at forecast origin k, calculated using test weights w_ik and loss L_ik:

R^k=∑i∈VkwikLik∑i∈Vkwik.\hat R_k=\frac{\sum_{i\in V_k}w_{ik}L_{ik}}{\sum_{i\in V_k}w_{ik}}.

The pooled score across all test rows and the equally weighted average of origin-specific scores differ when the origins contain unequal exposure or event counts. The first targets a mixture weighted by the observed exposure distribution; the second targets an average over evaluation origins. Both can be defensible, but each needs an explicit estimand. A rolling procedure should publish individual \hat R_k alongside its aggregation, so adverse periods are visible rather than washed out by more numerous ordinary risks.

Resampling and uncertainty

For a group-disjoint test with approximately independent groups, nonparametric resampling of the primary groups—not individual claim payment rows—can provide a distribution of a chosen statistic. If event shocks are common across regions, resampling independent event seasons or event clusters may be more meaningful than resampling policies. In a time series, moving-block methods or models of serial dependence can be considered, but their calibration requires sufficiently many effective time units. Neither a bootstrap nor a sandwich standard error rescues a test whose features contain future information.

The research literature warns against optimistic uncertainty calculations that treat cross-validation folds as independent. Nadeau and Bengio's generalization-error variance analysis addresses the variability of training-set choice. Diebold and Mariano's comparison of predictive accuracy addresses paired loss differentials across forecast sequences. Applied to insurance, the appropriate inference depends on whether the primary stochastic units are policies, policyholders, events, forecast origins or markets. A model tested on two flood years does not gain two thousand independent flood years simply because thousands of affected properties appear in the claim file.

Simple numerical intervals can be useful descriptive summaries when assumptions are stated. For an independent Bernoulli event at rate p, the approximate standard error of a sample rate from n trials is sqrt(p(1-p)/n). At p=0.01 and n=2,000 the nominal value is about 0.00222, or 0.222 percentage points; a 95% normal approximation is about 0.44 percentage points on each side. This approximation ignores clustering, finite-sample discrete-count refinements, model estimation and drift. It is shown only to convey the limited information carried by a modest number of rare events; it is not a confidence interval for predictive accuracy of an actual insurer model.

A worked exposure-weighting identity

Suppose two purely hypothetical test policies have exposures 1.0 and 0.25 year, and unit-level losses under an announced scoring rule of 4 and 12. Their unweighted mean is (4+12)/2 = 8. Their exposure-weighted mean is (1.0 times 4 + 0.25 times 12)/1.25 = 5.6. If the exposure is the target measure, 5.6 estimates an exposure-weighted loss on these two units; if the target is a randomly chosen policy, 8 addresses a different target. The exercise illustrates that a score can be mathematically correct and still answer the wrong population question. The chosen weighting formula must also be compatible with the scale of the response and the loss function; weighting a per-exposure rate and weighting an aggregate count can require different constructions.

For an event-level demonstration, consider three hypothetical catastrophes generating 50, 300, and 800 policy-claim rows. An equal-row average gives the largest event roughly 70% of row weight; an equal-event average gives each event one third. Both computations can be valid for distinct objectives, such as aggregate exposure impact versus future-event transfer. Neither is intrinsically representative of a future catastrophe distribution. The report should state whether prediction applies to a randomly sampled covered risk, a randomly sampled event, or a portfolio aggregate.

Minimum auditable split manifest

A reproducible evaluation can be specified without proprietary data disclosure by publishing a manifest with the following fields and privacy-appropriate identifiers:

Field Required scientific information
Prediction decision Forecast origin, horizon, target, allowable information and intended users
Unit of analysis Policy, policy-year, insured asset, claim, exposure record or event
Population and eligibility Coverage, jurisdiction, business line, inclusion and exclusion dates, canceled and rewritten business
Group constraints Policyholder/asset/claim/event/source identifiers and cross-fold exclusion rules
Data clocks Occurrence, inception, feature-availability, report, valuation, maturity and extract times
Split generator Algorithm, exact cutoffs, purge/embargo logic, random seed where applicable
Dataset lineage Source versions, extraction query or immutable snapshots, transformation chronology
Preprocessing controls Fold-specific fitting and as-of application of target encodings, imputers and learned statistics
Validation roles Inner selection, outer assessment, final locked holdout, audit and stress-test partitions
Scoring and weighting Metric, loss basis, exposure weights, aggregation unit and denominator
Uncertainty Effective sample units, interval or resampling assumptions, origin-specific results
Drift and coverage Covariate, label, exposure, calendar and claim-maturity diagnostics
Governance Model version, reviewer, change log, disallowed test feedback and report freeze

This list is an audit artifact based on established statistical principles and insurance data structure, not a new regulatory standard. For shared research across jurisdictions, it is often more informative than publishing only a model architecture name and a single score. It allows other investigators to decide whether a validation result is transportable to their own target, without demanding access to protected policyholder data.

Implementation-neutral split construction

The following pseudocode is an illustrative protocol. It intentionally separates information eligibility, entity grouping and temporal design rather than assuming one off-the-shelf splitter enforces all of them.

INPUT: immutable as-of data snapshots; target and horizon; entity/event links;
       forecast origins O; maturity rules; development search protocol;
       final-test embargo and freeze; scoring estimand.

1. Define target population, policy/claim observation units, and prediction time.
2. Build feature availability and outcome availability timestamps.
3. Construct connected group identifiers appropriate to the declared transfer target.
4. Reserve the final test cohort by origin time and/or independent group before modeling.
5. For each outer forecast origin o in the development history:
     a. Admit training records only when labels and features were eligible by o.
     b. Construct inner folds within eligible historical records, preserving the
        stated temporal and group constraints at every inner origin.
     c. Fit every learned transformation within the inner training partition.
     d. Choose candidate specification using only inner scores.
     e. Refit chosen pipeline on all eligible outer-development data.
     f. Score untouched outer-test units; record weights, clusters, and dates.
6. Freeze candidate family, training protocol, calibration and reporting code.
7. Fit the final model on training data eligible at the final allowed origin.
8. Evaluate the locked terminal test once on its predeclared target/weights.
9. Produce group/period diagnostics, uncertainty and the immutable split manifest.
10. Archive code, hashes, source versions, decisions, deviations and limitations.

The design must reject a split if a purported test observation has been consulted for feature selection or calibration, if a group overlap violates the stated new-entity target, or if the training data would have been unavailable at the simulated deployment origin. A split should also be flagged when essential target labels lack maturity, when test exposure is too small for the claimed precision, or when the deployment population has material unsupported strata. The final condition is especially important: eliminating leakage does not establish representativeness.

Methodological failure modes and corrective interpretation

A high random-fold score and a much lower rolling-origin score indicate sensitivity to time, information availability, changing populations or some combination; the difference alone does not identify which mechanism caused it. A high grouped-fold score with weak new-market performance indicates a transportability problem or a difference in target populations; it does not establish that the model was intrinsically underfit. When a model's 2024 predictions are evaluated with immature paid claims but its 2022 predictions are evaluated with ultimate incurred estimates, the comparison is confounded by label construction. When a test set is repeatedly viewed during model development, no amount of clever resampling turns it back into an untouched prospective test.

The most credible diagnosis starts with reproducible snapshots and counterfactual replay of the available-information set at each origin. It then audits group intersections, target-dependent preprocessing, claim maturity and weighted portfolio composition. Only after those checks should investigators attribute differences to candidate algorithm performance. This ordering prevents the modeling algorithm from becoming a substitute explanation for errors in the study design.

Limits of the research and scope of the conclusion

This dossier provides a formal methodological analysis, literature synthesis, and transparently hypothetical arithmetic examples. It does not conduct a proprietary insurance training experiment, infer actual insurer lift, compare vendor performance, produce jurisdiction-specific pricing or legal advice, or validate the completeness of a particular insurer's claims database. The original contribution is in the synthesis and insurance-specific articulation of validation constraints, information timing, deployment estimands and audit requirements. It does not claim a first discovery of data leakage, grouped cross-validation, temporal backtesting, or nested model selection.

The available sources establish principles and applications with varied evidentiary reach. Kaufman et al. (2012) address leakage in data mining; Roberts et al. (2017) analyze structured dependence in ecological validation; Varma and Simon (2006), Cawley and Talbot (2010), and Nadeau and Bengio (2003) examine selection and inference in general machine learning; Bergmeir et al. (2018) specify conditions for a particular autoregressive exception; Cerqueira et al. (2020) compare time-series estimation empirically. CAS monographs provide actuarial application context; ASOP No. 23, the NAIC model bulletin and EIOPA's Opinion provide profession- or jurisdiction-specific governance context. Their combined relevance does not transform any single empirical result into a general actuarial law.

Conclusion

The scientific object of insurance predictive validation is an estimate of performance for a specified decision process, population, observation history and forecast horizon. Training and testing roles derive meaning from that object. The independence of a test is consequently a multidimensional property: the test must be protected from model selection, its labels must be excluded from predictors and learned transformations, its observations must obey the chosen entity/event separation, and its simulated time origin must be respected by every data item used in training and scoring.

The principal methodological findings are as follows. First, conventional row-wise random splits can estimate a legitimate within-entity interpolation target but are generally insufficient evidence for new-entity, new-event and forward-time insurance claims. Second, group disjointness and chronological ordering protect different boundaries and frequently have to be combined. Third, the relevant chronological condition is the availability of features and training labels at the forecast origin, not merely the ordering of occurrence or policy inception dates. Fourth, nested development and a locked final test guard against the selection bias introduced by adaptive model building. Fifth, exposure, claim maturity, concentration of severe events, and portfolio selection determine the meaning and uncertainty of performance estimates. Sixth, performance stability requires origin-specific and group-specific diagnostics, appropriate inference units, and clear disclosure of the evaluation population.

The practical consequence is an insurance validation protocol whose split manifest can be reviewed independently of the model software and whose evidence boundaries are disclosed before optimization. Such a protocol produces a narrower and more useful scientific claim than an unqualified assertion of out-of-sample accuracy. It allows pricing, claims and risk researchers to interpret test results according to their actual operational targets, verify the causal and temporal availability of inputs, and recognize where the dataset does not support broader generalization. Further empirical work should compare these designs in sufficiently diverse, reproducible insurance portfolios using retained as-of data, exposure-aware scoring and independent event-level uncertainty assessments.

References