Insurance by BogdanovaThe Encyclopedia of Insurance and Risk Science

RESEARCH DOSSIER

Insurance Predictive Modeling Reproducibility and Open Benchmarks

Abstract. Predictive models increasingly structure insurance pricing, underwriting, claims management, portfolio monitoring, and actuarial research. Their apparent accuracy can be established through public examples while their reliability in a specific insurance setting remains difficult to assess. This monograph examines the scientific conditions under which insurance prediction results can be recomputed, compared, independently tested, and interpreted beyond the original dataset. It integrates reproducibility studies, actuarial statistical foundations, French motor insurance examples, probabilistic scoring theory, data documentation, validation research, and insurance-specific governance evidence. The investigation distinguishes computational reproduction of the same results from independent replication with new observations and external transport to other portfolios. It defines the necessary observational and contractual meaning of insurance targets; assesses open, synthetic, restricted, and confidential data; specifies baselines, temporal and grouped partitions, loss functions, and uncertainty analyses; and establishes a documentary record for code, parameters, dependencies, licensing, and data transformations. Original mathematical illustrations show how exposure normalization, prediction horizons, and evaluation weights alter a comparison even when the algorithms are unchanged. A fully worked hypothetical Poisson example demonstrates reproducible calculation without purporting to measure commercial performance. The principal analytical conclusion is that an insurance benchmark supports a limited, conditional empirical claim: a specified model, trained under a specified information boundary, performed according to a specified metric on a specified population and observation period. Wider applicability requires additional evidence about sampling, reporting, contractual coverage, delayed claims, changes in risk, and institutional setting. The work provides an original critical synthesis and a fully articulated reproducibility specification; it introduces no new named theory, framework, doctrine, or empirically unverified superiority claim.

Keywords: Insurance predictive modeling; reproducibility; replicability; open benchmarks; actuarial data science; frequency; severity; pure premium; exposure; GLM; gradient boosting; Tweedie regression; proper scoring rules; out-of-sample validation; leakage; dataset provenance; computational environments; uncertainty; distribution shift; restricted insurance data; FAIR principles; reproducible research.

Introduction

An insurance prediction benchmark is an empirical comparison between alternative ways of forecasting quantities that matter to insured exposures. The comparison acquires scientific meaning only after the prediction target, historical information boundary, observation mechanism, training protocol, test population, and scoring procedure have been specified. A score without those conditions is insufficient for a scientific claim because it is impossible to establish what was predicted, what the predictor could legitimately have known, and to which population the score applies.

The modern statistical and computational literature offers several complementary foundations. The US National Academies' definitions of computational reproducibility distinguish a result reproduced using the same data and methods from a new study that gathers new data. Wilkinson et al. (2016) articulate the FAIR data stewardship principles, which address discoverability, accessibility, interoperability, and reusability. Those principles support documented research artifacts but do not guarantee that a dataset is representative, lawful to redistribute, or suitable for an insurance question. Kapoor and Narayanan (2023) demonstrate in their cross-disciplinary leakage investigation that methodological defects can remain in apparently elaborate machine-learning evaluations.

Insurance exposes why the distinctions matter. A policy is sold for a defined term and coverage; claims can be reported, settled, reopened, inflated, or capped after that term; multiple policy records can concern the same person, insured asset, household, event, or catastrophic occurrence. One insurer may record a paid amount, another an incurred amount, and a third a present value or a closed-claim severity. The shared phrase "claim prediction" hides several incompatible tasks. Wuthrich and Merz (2023) develop actuarial learning as statistical modeling of insurance outcomes, with explicit attention to the insurance-specific data and modeling problem.

Open examples are essential scientific infrastructure. Noll, Salzmann, and Wuthrich (2020) compare a frequency generalized linear model with regression trees, boosting, and neural networks in their French motor claims case study. The scikit-learn developers provide an executable Tweedie insurance claims demonstration. These materials enable transparent exploration of multiple techniques on a recognizable type of nonlife insurance data. They cannot establish a universal ranking of model families: results depend on the precise transformations, sampling protocol, features, objective, tuning, and dates.

The object of this monograph is methodological reproducibility and open benchmarking in insurance prediction. Its questions are: Which research artifacts permit independent computation? What makes a common evaluation valid for competing algorithms? Which forms of sampling, reporting, and temporal structure threaten comparison? How does uncertainty qualify rankings? And what can a public benchmark legitimately say about a restricted insurer portfolio, another jurisdiction, or a future underwriting period?

The method is a critical documentary synthesis of verifiable academic and official sources, formal derivation from stated statistical assumptions, and an illustrative calculation. No original insurance dataset was retrieved, refitted, or tested for this monograph. Thus the text reports no new measured predictive performance, causal effect, or original experimental replication. Academic studies are attributed by year, and documentary claims are limited to the scope supported by their linked sources. The findings are conditional deductions and research-editorial specifications rather than the announcement of a new conceptual system.

Documented Insurance Tasks and Estimands

Scientific meanings of reproduction, replication, and transport

Computational reproducibility asks whether an independent investigator can obtain consistent numerical output using the original data, methods, code, and computational conditions. Replicability asks whether independently obtained observations support a consistent answer to a comparable scientific question. Generalizability concerns the application of findings to other contexts. The National Academies (2019) make these distinctions explicit in their study of reproducibility and replicability. In applied actuarial machine learning, a fourth practical question must also be separately reported: whether two genuinely independent software implementations encode the same mathematical specification.

The use of "reproduced" and "replicated" is not uniform across communities. The Association for Computing Machinery's artifact review and badging policy distinguishes results obtained by a different team using author artifacts from results obtained independently without them. A reproducibility report should define its intended operation before applying a label. Terminological compliance alone never demonstrates that a run succeeded.

Five evidence levels can be described without turning them into a new theory: a source is accessible; its executable artifacts can be inspected; the original computation can be rerun; an independent environment or implementation reaches results within justified numerical tolerance; and an independent dataset supports a similar substantive conclusion. Each subsequent statement requires evidence of its own. The existence of open-source code supports the first two propositions but does not establish the last three.

Insurance observations, contracts, and information sets

Let i identify a policy-period record, X_i(t) the covariates legitimately observed as of prediction time t, E_i the exposure measured in policy-years over a specified interval, N_i the number of contractually eligible reported or ultimate claims, and S_ij the monetary outcome attached to claim j. Let Y_i denote an aggregate insurance outcome, such as the sum of covered claim costs. A valid task specifies the policy population P, coverage definition C, accident and observation windows T, reporting maturity M, monetary valuation V, and admissible information set F_t. Formally a conditional target can be written as E[Y_i | F_t, P, C, T, M, V]. The notation is an explicit specification of existing probability theory, not a claim to an original estimand class.

A "claim count" can mean claims with accident date in the insured term, claims reported in that term, unique claim events, coverage-level claim records, or claims exceeding a deductible. These need not coincide. Similarly, a policy's "premium" may be the insurer's charged amount, an earned amount, a risk premium, or a projected pure premium. Benchmarks must identify each field semantically and describe the conversion between policy, exposure, claim, and calendar tables.

The as-of boundary is decisive. Suppose a model predicts loss frequency at issuance on January 1. A vehicle characteristic recorded at issuance is an admissible predictor. A renewal decision in December, a final claim settlement amount available two years later, or a claim-handler category produced after the loss is outside that information set. Keeping such variables in the training and test tables can produce a repeatable computation that measures an illegitimate forecasting task. Kapoor and Narayanan's taxonomy of leakage mechanisms supports systematic inspection of this problem.

The data-generating system also selects which risks appear. Underwriting acceptance, cancellation, geographic marketing, exposure duration, claim notification, deductibles, and fraud review affect observed samples. A benchmark derived from accepted policies conditions on the insurer's historic underwriting. It does not directly identify risk among rejected applicants or in the broader population. The conditional nature of the sample belongs in the reported claim.

Frequency, severity, aggregate loss, and duration

For frequency, an elementary specification is N_i | X_i,E_i approximately Poisson(E_i lambda_i), with lambda_i = exp(beta_0 + X_i beta) in a log-link generalized linear model. Exposure belongs in the expectation or as a log offset. For severity, a common model predicts a positive claim cost conditional on a claim and the chosen settlement definition; the severity training population therefore differs from the full policy population. A separate severity model and a direct aggregate-loss model answer related but distinct questions.

If the desired target is the expected aggregate covered loss per exposure unit, the rate can be defined as r_i = E[Y_i | X_i,E_i]/E_i for E_i greater than zero. Under particular frequency-severity assumptions, expected claim cost per exposure can be expressed as expected count per exposure multiplied by expected positive cost per claim. Those assumptions and the conditional populations must be disclosed, particularly when large losses and claim frequency are associated.

Henckaerts et al. (2021) examine insurance tariff modeling with trees, focusing on exposures, excess zeros, and potentially long-tailed claim sizes. Their published case-specific finding of stronger boosted-tree performance is a result within the stated study, not proof of superiority across insurers, all metrics, or all calendar years. The same careful scope applies to the French motor frequency comparison.

Duration tasks include time to first report, closure, recovery, lapse, or insured event. Their risk sets, censoring conventions, entry times, and right-truncation rules differ from count and severity benchmarks. Applying a single generic regression-error measure to these outcomes can obscure what the study estimates. Research protocols should preserve the actual task rather than force all insurance data into a common label column.

Label maturity, selection, and target comparability

An incurred loss at 12 months after accident date may differ from the final ultimate loss. A claim count at 30 days after renewal may miss delayed notifications. Even if two datasets use identical column names, their outcome maturity can differ. Reproduction requires a frozen extraction date and definitions of reopened claims, salvage and subrogation, expense inclusion, inflation adjustment, currency, and exchange-rate conventions.

A useful reconciliation exercise calculates three quantities separately: the label available at a specific historical evaluation date, the later revised label if legally and operationally obtainable, and the difference caused by development. A ranking change between these two views is evidence about target maturity and evaluation stability, not automatically about algorithm quality. The protocol should distinguish late-label performance from forecasts of ultimate claims and should not silently replace one with the other.

Contractual features can also change over time: deductibles, limits, claims-made provisions, exclusions, and claims-handling procedures influence amounts. In a benchmark drawn from multiple years, normalization should be justified on the basis of the estimand. Dividing losses by contemporaneous premium, for example, creates a ratio influenced by insurer pricing decisions; the result is not a neutral measure of physical hazard or insured expected loss.

The fundamental inference is that a reproducible numerical score depends on a reproducible insurance meaning. A table of exposures, labels, and predictions becomes scientific evidence when the contractual and temporal origin of each component is traceable.

Open and Restricted Data

Public actuarial datasets and their limitations

The publicly accessible CASdatasets collection aggregates actuarial datasets used in teaching and research, including examples associated with Computational Actuarial Science with R. The collection illustrates a practical virtue of open data: multiple investigators can test a mathematical claim using accessible observations and can compare a common transformation with a published implementation. It does not imply that every bundled dataset has the same license, sampling frame, variable dictionary, or suitability for every kind of inference.

The scikit-learn motor insurance example uses French motor third-party liability data accessed through OpenML and describes exposure, claim counts, severity, and pure premium. The example contrasts separate frequency and severity models with a Tweedie aggregate model. It is a useful test case precisely because the transformations are visible. Research reuse should nevertheless record the exact OpenML dataset IDs, data version or checksum, scikit-learn release, feature transformations, download date, and applicable redistribution terms. A link to a dynamic dataset endpoint does not fix the bytes indefinitely.

Open insurance datasets present recurring limitations. They often cover one country, one line of business, one insurer or data provider, and a historical period that differs from the intended deployment. They may be de-identified, feature-reduced, sampled, or cleaned before distribution. Some geographic or behavioral predictors are unavailable; the original coverage and claim-administration rules may be incompletely documented. These features constrain inferences about market-wide performance and about the performance of models using information unavailable in the public benchmark.

The Noll, Salzmann, and Wuthrich study is especially informative as a methodological comparison of model classes on a shared task. It should be reproduced with the authors' exact task and data preprocessing before changing train/test splits or tuning rules. An independent extension that changes the validation protocol is valuable but constitutes a new experiment; mixing its scores with the original table without disclosure would defeat comparability.

Dataset documentation, licensing, and provenance

A reusable dataset needs more than a download link. At a minimum its documentation should state provenance, release version, unit of observation, population, time range, source systems, anonymization or transformation steps, variables, missingness, eligibility, currency, claim development, quality controls, access conditions, redistribution permissions, and the responsible curator. Gebru et al. (2021) motivate comprehensive datasheets for datasets, while Wilkinson et al. (2016) articulate findability and reuse principles. Both can inform insurance documentation without being misrepresented as legal authorizations to disclose personal data.

A benchmark release should assign a persistent identifier to a fixed dataset snapshot where permissible and record a byte-level checksum. A checksum verifies content identity for the retrieved artifact, not its scientific validity. If the dataset is updated, the version and changelog should document variable recoding, corrected errors, removed records, and changed labels. The source lineage should identify whether values arise from original policy systems, claims administration, accounting extracts, provider transformations, or researcher-created features.

The right to access a webpage and the right to redistribute its underlying records are different permissions. A data file may be accessible for research under restrictions that prohibit republication, combination with personal identifiers, or commercial use. The research record should capture the actual license or agreement in effect for that version. If permission cannot be established, a study can describe its source and code without redistributing restricted bytes.

An insurance-specific data dictionary should also explain whether exposure is measured as earned, written, or in-force time; whether vehicles or policyholders are deduplicated; how policies with zero earned exposure are treated; how limits and deductibles enter the outcome; and whether historical claim records have been censored. Documentation of these details is often more important for interpreting a purported performance gap than adding another model family.

Restricted reproduction and privacy-compatible audits

Insurer data frequently contain commercially confidential, personal, health, mobility, financial, and claims-related information. Public release may be impermissible or inconsistent with a participant's legitimate expectations. Reproducibility therefore cannot be defined solely as unrestricted downloading. A strong restricted-data study can supply a publicly reviewable specification and code, a schema, data-quality summaries, a controlled-access procedure, permission checks, and a record of independent execution in an authorized environment.

Possible arrangements include a secure research enclave, a trusted independent verifier, a synthetic fixture for executable testing, and disclosure of aggregate outcomes subject to statistical disclosure controls. Their evidentiary reach differs. Re-running a pipeline in an enclave can support computational reproducibility on protected data. Running only the code against synthetic records can demonstrate pipeline functionality and identify some leakage or type errors, but cannot certify the predictive score claimed for confidential real observations.

The EU insurance-specific discussion by EIOPA's consultative expert group addresses data governance, record keeping, robustness, performance, fairness, and monitoring in its 2021 AI governance report. EIOPA expressly describes the guidance as nonbinding and illustrative. It is relevant documentary context for European insurers; it must not be presented as a universal insurance law or a substitute for applicable national requirements.

Documentation and authorization should be disentangled in an audit. The research team can show that a model pipeline is specified reproducibly even when a data controller refuses broad disclosure. Conversely, a dataset may be public yet unsuitable for privacy-sensitive downstream inference or insufficiently documented for scientific comparison. The question for a research reviewer is what claim the available evidence actually permits.

Synthetic data and international data comparability

Synthetic observations can be generated under an assumed distribution, transformed from restricted observations, or simulated from a fitted generative mechanism. These procedures have different privacy and statistical properties. A synthetic dataset reproducing selected marginal distributions need not preserve interactions, extreme losses, group imbalances, rare events, or temporal dependence. A small artificial fixture is excellent for verifying type conversions and unit-test invariants, while a probabilistic simulation is useful for examining behavior under known assumptions.

Synthetic results should therefore be labeled as simulations with documented data-generating settings. They do not establish empirical market accuracy. Privacy risk also demands independent evaluation: a label of "synthetic" alone does not certify that model outputs cannot reveal information about original individuals. A benchmark report should record whether synthetic records were fitted to real data, by whom, and under what access restrictions.

International comparison introduces additional obstacles. Claim amounts may be recorded in nominal local currency, after taxes, at different settlement points, or under different policy limits. Reporting propensity, coverage, court practice, fraud detection, inflation, medical reimbursement, and catastrophe exposure also differ. A shared algorithm applied to two countries is not a controlled comparison unless the target and the evaluation protocol have been made substantively comparable.

The evidence supports a cautious but constructive conclusion: public data are essential for methods research and educational reproducibility, while restricted data are essential for testing the external validity of methods in material insurance settings. A scientific benchmark can make both channels visible without asserting that the same access conditions or empirical conclusions apply to each.

Baselines and Evaluation Protocols

Minimal credible model comparisons

A credible benchmark begins with models whose behavior can be understood. For count frequency, common reference points include an intercept-only exposure-adjusted Poisson model, a main-effects Poisson generalized linear model, and a regularized or more flexible extension. For positive severity, suitable references may include a constant-mean benchmark and a justified Gamma or log-link model. For aggregate loss, a documented frequency-severity product and a Tweedie model offer interpretable alternatives when their assumptions fit the task.

Wuthrich and Merz (2023) discuss these families within actuarial statistical learning. Henckaerts et al. (2021) demonstrate tree methods for tariff plans, including model tuning and the economic context of insurance pricing. Their work shows why the baseline must reflect relevant insurance structure and why a generic mean-squared-error competition may be insufficient.

All candidates should use the same eligible observations, prediction horizon, admissible input availability, target definition, and test assignment. Differences in feature engineering or preprocessing should be declared as part of the competing pipelines. One model cannot fairly benefit from a much larger search budget or post-test tuning while another is judged from its first specification, unless the experiment explicitly studies such unequal development processes.

Baselines also establish diagnostic context. If a complex model improves a primary scoring rule only marginally, exhibits unstable subgroup behavior, or fails when retrained on a subsequent year, that result is substantively different from an unconditional declaration that complexity is beneficial. The benchmark should report practical effect sizes and the uncertainty around them, not only the order of a leaderboard.

Split design, grouping, and chronological prediction

Random assignment of rows is often inappropriate when insurance observations share people, policies, vehicles, households, brokers, claims, events, geographic areas, or calendar periods. Records from the same policyholder on both sides of a split can reveal stable information that would not be available when predicting a genuinely new customer. Records from a catastrophe on both sides can reveal information about the same event. The correct partition follows the intended target of generalization.

Roberts et al. (2017) review cross-validation under structured dependence. Scikit-learn documents nonoverlapping grouped cross-validation and chronological validation procedures. These references support specific mechanisms, but no one split technique guarantees validity for every insurance problem.

Consider three deployment questions. Prediction for existing customers next renewal year suggests a forward-chaining or rolling-origin evaluation that preserves customer identity and history legitimately available at renewal. Prediction for entirely new customers may demand customer-disjoint groups. Predicting claim costs for geographic events in another region can require spatial or event grouping, with dates that prevent observation of the same catastrophe across training and test. The split design expresses which deployment question is being answered.

Preprocessing must respect partitions. Imputation parameters, category encodings, target encoding, scaling, feature selection, outlier thresholds, and calibration transformations should be fitted inside the training part of each fold. Aggregates calculated on the full table may inadvertently incorporate validation outcomes. Leakage is especially acute for target-encoded high-cardinality insurance categories and features computed after claims settlement. Kapoor and Narayanan's study of methodological leakage provides a wider scientific warning.

A transparent protocol records the exact row indices or hashed grouping identifiers for every fold, the time ordering of extraction and scoring, the seed and random-number generator when relevant, and any embargo or gap applied between train and test. It also discloses whether several rows from a multiyear contract enter different folds for a justified temporal task. A single random seed is insufficient documentation when the underlying statistical task is unclear.

Proper scoring, exposure, and decision relevance

A metric measures a chosen feature of predictive behavior. For frequency, one can compare Poisson likelihood or deviance, assuming that model and variance specifications justify the score. For a Bernoulli claim-incidence target, log loss or Brier score evaluate probabilistic forecasts. For full predictive distributions, proper scoring rules reward honest distributional reporting under their specified conditions. Gneiting and Raftery (2007) establish the general theory of proper scoring rules.

For a Poisson observation y_i with predicted mean m_i > 0, its unit deviance is

D_i = 2 { y_i log(y_i/m_i) - (y_i - m_i) },

where the term y_i log(y_i/m_i) is defined as zero when y_i = 0. For exposure-based count prediction m_i = E_i lambda_hat_i, with E_i positive. One can report total deviance sum_i D_i, average deviance per policy row, or deviance per unit of exposure. These aggregations differ when exposure varies. Their denominator, inclusion rules, and units must be declared.

For pure premium, the researcher should say whether the error concerns total claim cost per policy, monetary cost per policy-year, or expected aggregate loss. A model with lower aggregate residual under one weighting scheme can rank differently under a different portfolio mix. In an insurance benchmark, exposing this choice is central scientific information.

Proper scoring does not imply that a metric captures every business or social consequence. Calibration, severity-tail fit, operational constraints, fairness, and premium stability may require additional diagnostics. A ranking by Poisson deviance cannot establish claims settlement fairness, the adequacy of an insurer's capital, or the desirability of a final premium structure. Those would require additional questions and evidence.

Calibration should be evaluated with an explicit conditioning variable or grouping and adequately sized bins or smoothers. A single global ratio between observed and expected claims may be near one even when geographic or demographic subgroups have large countervailing errors. Disaggregated and temporal calibration displays support interpretation but can be unstable when claims are rare. The model reporting recommendations of Mitchell et al. (2019) reinforce the need to document intended use and subgroup performance.

A worked hypothetical benchmark calculation

The following numerical calculation is original, fully specified, and deliberately hypothetical. It is a mathematical check of metric interpretation, not an empirical insurance experiment. Suppose four policy-period observations have earned exposures E = (1, 0.5, 1, 1.5) policy-years and independently recorded claim counts y = (0, 1, 2, 0). Total exposure equals 4 policy-years and total observed count equals 3. The descriptive aggregate frequency is therefore 3/4 = 0.75 claims per policy-year. That descriptive ratio has used all four outcomes and must not be represented as an out-of-sample fitted estimate.

Imagine two forecast vectors defined before these outcomes are revealed. Model A assigns the fixed frequency rate 0.75 to every observation. Model B assigns rates (0.3, 1.2, 1.4, 0.3). Their predicted claim means after multiplying by exposure are m_A = (0.75, 0.375, 0.75, 1.125) and m_B = (0.3, 0.6, 1.4, 0.45). The vectors are posited for illustration, not trained on this sample.

Applying the stated Poisson deviance formula and retaining the zero-count convention gives total deviance D_A approximately 5.8849755181 and D_B approximately 1.9483510233. Dividing each by the total exposure gives 1.4712438795 and 0.4870877558 deviance units per policy-year, respectively. The raw totals remain the appropriate direct comparison when the same test observations and exposures are used. The exposure-normalized values express a second, explicitly declared scale.

These arithmetic results establish only that the specified predictions differ under the selected score on these four invented observations. They provide no sampling uncertainty, no tuning history, no evidence of generalization, and no claim that B will perform better on any real portfolio. Moreover, the numerical coincidence of A's rate with the descriptive sample mean is irrelevant to model assessment because A was stipulated ex ante. If A had actually been estimated from these test outcomes, the same calculation would be contaminated by training/test leakage.

The example demonstrates several basic reproducibility invariants: inputs, units, predicted means, zero-count treatment, score formula, aggregation, and rounding are all visible. A reader can independently recalculate the scores using any programming language or a spreadsheet. No hidden trained model, proprietary data, stochastic routine, or undisclosed transformation is needed.

Model selection and prevention of benchmark overfitting

The full modeling pipeline must be selected without consulting the locked final test outcomes. Hyperparameters can be searched on training partitions or validation data; a final model can be refitted according to an announced rule before one locked test evaluation. If researchers repeatedly inspect the same held-out outcomes while choosing features, architectures, metrics, or thresholds, they adapt the study to the test set even when their code remains deterministic.

Dwork et al. (2015) study the statistical problem of adaptive holdouts. The insight is relevant to public leaderboards: a widely reused test set can become an implicit component of model development. A claimed state-of-the-art improvement then requires scrutiny of the evaluation history, not merely the publication date and point estimate.

A scientifically informative benchmark records candidate models attempted, search space, search algorithm, training budget, tuning folds, stopping criteria, and the final selection. If the public data have become a repeatedly mined benchmark, researchers should use independent temporal or portfolio validation to assess whether advantages persist. A model comparison with strong out-of-sample design can be reported without turning small numerical gaps into universal rankings.

Code and Parameter Provenance

Executable reconstruction and immutable inputs

Computational reproduction requires the exact computational objects, not a general description such as "a gradient-boosting model was trained." The archive should contain a human-readable run guide, source code, immutable data identifiers, dependency versions, a locked computational environment, machine-readable configuration, split specification, model-selection procedure, scoring routine, output expectations, and instructions for checking hashes and tolerances.

A reusable research repository can separate raw data access, data preparation, training, inference, evaluation, and report rendering into independently invoked steps. The important property is determinable lineage: an evaluator should be able to trace a published table cell through a metric script to saved predictions, a fitted model, a training configuration, a split manifest, and a versioned input dataset. The FAIR principles address metadata and reuse, while the ACM artifact-badging guidance identifies accessibility and independent artifact evaluation as distinct dimensions.

Integrity checks should cover the dataset snapshot and all released executable artifacts. File checksums confirm exact bytes. Source-control commit identifiers establish a particular version of code when the repository history is retained. Neither mechanism proves correctness. A code commit without a data version can execute on silently revised data, and a dataset hash without a feature-construction script cannot reconstruct a claim-cost target.

Stable machine-readable outputs should include the names and orders of evaluation rows, predicted values on their natural scale, the observed targets when permitted, and metadata identifying units. This allows an independent evaluator to separate a disagreement about fitted parameters from one about scoring, rounding, missing values, or row alignment.

Preprocessing, tuning, computation, and record keeping

Claims and policy data often require temporal joins and complex transformations: policy transaction reconciliation, earned exposure calculation, claim-event deduplication, censoring, inflation adjustment, categorical coding, and late-claim development. The benchmark archive must identify how each transformation changes the population and the outcome. A transformation that removes zero-loss policies, for example, changes the frequency experiment. A transformation that winsorizes large losses changes a severity or aggregate-loss estimand. The resulting score should be interpreted for that altered target.

Document all random components: sample splitting, stochastic optimization, subsampling, bagging, neural-network initialization, and simulation. Record software releases, dependency locks, operating system and hardware where relevant, numerical precision, threading, and determinism settings. A fixed seed supports repeatability within a known implementation but does not guarantee bit-for-bit identity across hardware, library versions, or parallel execution orders.

The output verification protocol should distinguish exact-byte reconstruction from scientifically acceptable numerical tolerance. Hash equality is appropriate for an immutable input archive. Floating-point metrics or fitted weights may vary within an explicitly justified tolerance. Tolerances should be set by numerical analysis or substantive decision relevance, not chosen after observing a mismatch to declare success.

Code provenance includes licenses and maintenance context. A model may depend on a statistical package whose version changes its default estimator, optimization implementation, or treatment of missing values. A stable benchmark should capture the effective defaults rather than rely on an unqualified phrase such as "default settings." The date when dependencies were resolved and the observed installation procedure belong in the record.

An executable example also needs negative tests: deliberately permute rows before scoring and ensure alignment checks fail; attempt to insert a post-outcome field and ensure feature validation rejects it; corrupt a data hash and ensure execution aborts; change a split manifest and ensure outputs no longer claim the same evaluation identity. Such checks probe the integrity of the procedure and are distinguishable from research experiments claiming measured accuracy.

Reproducibility claims and artifact evaluation

An article can publish code without proving that others can run it. An evaluator should record actual commands invoked, the exit status, environment, checksums, material deviations from the author's procedure, and whether key results were obtained within specified tolerance. The ACM results validation categories provide an institutional reference for distinguishing available artifacts and reproduced or independently replicated results.

The process must preserve failures. If an original dataset is no longer licensed for access, reproducibility is constrained by that permission, even if the code is flawless. If a dependency has become unavailable, an archived environment may restore execution but introduces maintenance and security questions. If model predictions cannot be matched to their test rows, a metric reproduction is not established. A report should identify each problem specifically rather than bury it under a general approval phrase.

NIST's AI Risk Management Framework 1.0 is a voluntary, cross-sector risk-management resource. Its emphasis on trustworthy and managed AI systems can help organizations organize documentation and controls, while its generality and voluntary scope mean it does not impose an insurance benchmark scoring rule. The distinction between research reproducibility and organizational model governance remains important.

The strongest scientific archive treats the published monograph, code, data descriptions, and evidence manifests as related but separate records. The text explains the meaning of results; the code permits computation; the data documentation permits interpretation; and the provenance record fixes identity. Together they enable an independent investigator to identify precisely what was and was not verified.

Statistical Uncertainty of Benchmarks

Sampling variation and paired comparison

A model's score is estimated from finite observations drawn through a particular insurance process. Even when the score calculation is computationally exact, its population interpretation is uncertain. The uncertainty depends on the amount of independent information, the heaviness of the claim-cost tail, exposure weights, dependence between records, the prevalence of claims, and whether the test design matches the target population.

For two models A and B evaluated on the same n eligible policy units, let d_i = L(y_i, f_Ai) - L(y_i, f_Bi) denote the paired difference under an announced loss function. Its sample average estimates a specified mean loss contrast under assumptions about sampling. When observations are independent and similarly distributed with finite variance, an ordinary standard error can be constructed from the sample variance of d_i divided by n. Insurance data often violate the independent-unit condition; a household, catastrophe event, or geographic portfolio may supply many correlated rows.

The uncertainty analysis should therefore correspond to the effective unit of sampling. A cluster bootstrap might resample customers or catastrophe events, while a time-block design might retain dependence across nearby periods. A procedure that resamples individual claims independently after the model was evaluated on clustered policies can substantially understate variation. Varoquaux (2018) explains in a different predictive domain why cross-validation error bars can be much larger than naive fold variation suggests. The analogy concerns the statistical caution, not a direct empirical estimate for insurers.

A credible comparison reports the metric contrast, its units, an uncertainty interval based on a justified dependence structure, the number of independent groups and events, and robustness to reasonable scoring choices. It can also report the frequency with which the ranking reverses across predefined periods or portfolios. An interval spanning zero should be interpreted as inconclusive for the stated contrast, rather than converted into a categorical preference by relying solely on the point estimate.

Dependence, rare losses, and uncertainty under selection

Claims are highly uneven. Numerous policy-periods record no covered claims; a small set may generate severe losses. Average error can be dominated by large events or, under inappropriate scaling, by the large mass of zeros. A benchmark should show the claim-count distribution, positive-loss distribution, concentration of aggregate cost, observation horizon, and sensitivity of the metric to large events. Where a tail cap is applied, the capped and uncapped target interpretations must be kept distinct.

An insurance catastrophe can affect thousands of insured assets simultaneously. Treating those assets as independent training or evaluation units exaggerates the amount of independent evidence about event performance. A benchmark intended for catastrophe portability may need event-disjoint or region-disjoint validation and event-level uncertainty. An evaluation intended for household pricing may need repeated-customer grouping. The structured cross-validation review by Roberts et al. (2017) supplies a general statistical foundation for respecting such dependence.

Validation uncertainty also arises when a dataset is selected through the commercial decisions of an insurer. Policies that are accepted, retained, marketed to, or renewed are influenced by past price and underwriting rules. The observed test distribution is conditional on these practices. Changes in acceptance or renewal can alter feature distributions and outcome frequencies independently of any technical modification to the prediction algorithm.

Claims settlement practices are another source of measurement change. The same physical event can generate different ultimate paid losses due to limits, adjustment methods, litigation, inflation, and recoveries. A reproducibility study should identify whether it evaluates reported claims, incurred liabilities, paid costs, or closed claims, and should record label completeness by event period. Failure to do so can make two studies numerically comparable while their targets are substantively different.

Calibration, subgroup behavior, and research integrity

Calibration asks whether predicted quantities are consistent with observed outcomes at appropriate levels of aggregation and across relevant conditions. For count models, observed-to-expected ratios can be computed for prespecified cohorts; for binary events, reliability plots compare probability predictions with observed event fractions; distributional diagnostics address tail and quantile behavior. The proper scoring analysis of Gneiting and Raftery (2007) provides a reason to consider distributions and elicited quantities instead of relying on a single ad hoc accuracy statistic.

Subgroup evaluation matters when risk, exposure, coverage, and measurement differ across regions or classes. Mitchell et al. (2019) propose structured model reporting, including specified uses and performance in relevant groups. In insurance, such analyses require attention to applicable legal restrictions, group sample sizes, privacy, product differences, and interpretation. A narrow empirical imbalance cannot on its own identify unlawful discrimination, and a favorable overall score cannot establish fairness.

The literature on adaptive data analysis warns against selecting a metric or subgroup after observing where a favored algorithm performs best. The reusable holdout study formalizes an important part of the underlying problem. Accordingly, the primary metric, important subgroups, adjustment for multiplicity where appropriate, and stopping rules should be declared before looking at final test outcomes. Exploratory analyses remain useful when transparently marked as exploratory.

The scientific lesson is that uncertainty does not weaken the value of benchmarks. It identifies the range of conclusions the evidence can support. A ranking accompanied by valid uncertainty and a documented population is more informative than a confident numerical leaderboard whose independence and target definitions cannot be reconstructed.

Reproduction and International Transfer

Reproduction across implementations and organizations

An independent reproduction begins by checking the original statement of the task and the artifact inventory. The evaluator confirms whether access to the exact inputs is lawful and possible, reconstitutes the declared environment, verifies data identity, executes the pipeline, and compares predictions and metrics against published tolerances. The evaluator records every discrepancy without quietly editing the author's source and then claiming exact reproduction.

A second implementation can test the mathematical semantics of a reported algorithm. For example, two implementations of a Poisson frequency GLM must agree on exposure offsets, intercept treatment, categorical reference levels, regularization, numerical tolerance, and test row ordering before parameter or deviance agreement becomes meaningful. For tree ensembles, splitting rules, missing-value handling, categorical processing, and sample weights require comparable specificity. When two implementations implement materially different algorithms under a shared label, a comparison should be treated as a new methodological experiment.

The scikit-learn motor claims example and open actuarial dataset library give researchers concrete starting points for this kind of work. Their existence does not establish that this monograph's author has rerun those experiments. The distinction between verified source description and independently executed research is maintained throughout this study.

Organizational independence can also be meaningful: an insurer's internal audit team might have access to the same protected data through independently controlled permissions, while an external team sees only a public summary. Both can contribute evidence, but they cannot claim the same verification scope. The audit report should name the class of verification performed and identify who controlled the computational environment.

Transfer across insurance markets and periods

Transfer has at least three empirically different targets: later periods in the same portfolio, other portfolios within the same jurisdiction, and different markets or jurisdictions. A model that replicates a public historical score may still deteriorate when driving patterns, repair costs, medical inflation, reporting incentives, regulation, or product design changes. Recalibration can improve a forecast under some changes but cannot be assumed to restore validity under every new exposure or coverage structure.

A sound transfer study identifies the transported prediction target, the admissible new information set, and a common or explicitly reconciled loss definition. It reports differences in exposure units, portfolio composition, contract limits, relevant inflation, claims development, region, data capture, and evaluation period. If the conditions are not comparable, the study should estimate performance separately rather than pool numbers into a single model-family ranking.

The statistical literature on validation under temporal dependence supports deliberate partitioning for these transfer questions. The actuarial literature's nonlife prediction foundations support the inclusion of contractual and distributional details. The relevant conclusion arises from their synthesis: the applicability of an algorithm is conditional on the stability of the insurance task and the observation process, not merely on the similarity of feature labels.

Research summaries should separate descriptive transport results from explanations of why performance changed. A model failing on a later-year portfolio could reflect new hazards, distribution shift, reporting lag, changes in claim handling, data availability, or a mismatch in the outcome definition. Without additional research, the observed degradation cannot uniquely identify its cause.

Research reporting, governance, and the evidence boundary

A completed benchmark report should present the research question, dataset identity and permissions, sample construction, observation and prediction dates, contractual target, eligible features, all splits, baseline and competing pipelines, hyperparameter strategy, metric formulas, uncertainty procedures, computational environment, artifact hashes, and limits of application. Such reporting converts an abstract claim of "transparency" into concrete opportunities for verification.

The 2021 European insurance AI governance guidance discusses robustness and reproducibility as part of its nonbinding, sector-specific account. NIST's voluntary AI risk framework addresses a different, general risk-management level. These documents can inform responsible research reporting but are not independent proof of a model's predictive validity and do not transform a benchmark into regulatory certification.

A scientific record should distinguish four conclusions. First, the computation was reproduced under documented conditions. Second, the proposed evaluation was appropriate for the target and sampling process. Third, the estimate and uncertainty support the stated empirical comparison. Fourth, results carry to another real insurance environment. Each conclusion needs corresponding evidence. Passing the first test is important, but it does not entail the later conclusions.

Open benchmarks also require stewardship. Datasets and code should remain discoverable and versioned when permissions permit, with a method for correcting errors and preserving superseded versions. The FAIR research-data principles explain why reusability involves persistent information about data and methods. The underlying scientific result may need to be revised when a material data correction changes the target, even if every older checksum continues to match its archived file.

The original synthesis of this monograph is therefore a disciplined scope of inference: an insurance benchmark establishes a conditional, time-indexed, contract-indexed comparison whose computational, statistical, and transport claims must be evidenced separately. That conclusion follows from established actuarial probability, modern reproducibility research, documented dataset practices, and the particular observation mechanisms of insurance. It is offered as an analytical conclusion, without claiming a new theory or naming a new normative system.

Conclusion

Insurance predictive modeling becomes scientifically reproducible when researchers make their empirical question, observation process, code, data identity, and evaluation decisions sufficiently explicit for an independent investigator to check them. Open benchmark datasets make parts of this process accessible and are indispensable for transparent method comparison. Their principal strength is inspectable and repeatable experimentation on fixed tasks; their principal limitation is the restricted range of portfolios, contractual settings, observation periods, and variables that they represent.

The monograph establishes that a defensible insurance benchmark requires a precise estimand, admissible as-of features, documented exposure and loss definitions, protected partitions, legitimate baseline models, predeclared scoring rules, uncertainty adjusted for dependence, and verifiable computational provenance. It distinguishes a successful rerun from independent replication, statistical validity, and transfer to another insurance environment. Insurance-specific data issues - contractual coverage, label development, high zero frequency, rare severe losses, repeated exposure units, event dependence, and underwriting selection - determine how that distinction should be applied.

The hypothetical Poisson calculation confirms the algebra of a clearly stated scoring rule and demonstrates how exposure-adjusted forecasts can be independently checked. It does not contribute a real-dataset benchmark or a performance claim. Academic case studies of French motor claims and the open actuarial software ecosystem supply verified starting points for future independent reproduction. General studies of leakage, proper scoring, and structured validation explain why using those resources correctly requires more than running code.

The original contribution is the integrated interpretation of reproducibility as an evidentiary requirement for insurance research, with explicit links between contractual meaning, computational traceability, statistical comparisons, and international transport. The conclusions identify conditions under which evidence is sufficient and boundaries beyond which new observations or independent checks are needed. No claim of new theoretical invention, empirical superiority, or external peer review is advanced.

References

  • Association for Computing Machinery. 2020. "Artifact Review and Badging - Current," version 1.1 policy. ACM artifact review and badging policy.

  • Dutang, Christophe, and CASdatasets contributors. n.d. "CASdatasets: Insurance Datasets." Project documentation, accessed November 14, 2026. CASdatasets project documentation.

  • Dwork, Cynthia, Vitaly Feldman, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Aaron Roth. 2015. "The Reusable Holdout: Preserving Validity in Adaptive Data Analysis." Science 349 (6248): 636-638. DOI and indexed article.

  • European Insurance and Occupational Pensions Authority, Consultative Expert Group on Digital Ethics in Insurance. 2021. "Artificial Intelligence Governance Principles: Towards Ethical and Trustworthy Artificial Intelligence in the European Insurance Sector." Published June 17, 2021; nonbinding consultative guidance. EIOPA official report.

  • Gebru, Timnit, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daume III, and Kate Crawford. 2021. "Datasheets for Datasets." Communications of the ACM 64 (12): 86-92. DOI and journal record.

  • Gneiting, Tilmann, and Adrian E. Raftery. 2007. "Strictly Proper Scoring Rules, Prediction, and Estimation." Journal of the American Statistical Association 102 (477): 359-378. DOI and publisher record.

  • Henckaerts, Roel, Marie-Pier Cote, Katrien Antonio, and Roel Verbelen. 2021. "Boosting Insights in Insurance Tariff Plans with Tree-Based Machine Learning Methods." North American Actuarial Journal 25 (2): 255-285. DOI and publisher abstract.

  • Kapoor, Sayash, and Arvind Narayanan. 2023. "Leakage and the Reproducibility Crisis in Machine-Learning-Based Science." Patterns 4 (9): 100804. DOI and full text.

  • Mitchell, Margaret, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. 2019. "Model Cards for Model Reporting." Proceedings of the Conference on Fairness, Accountability, and Transparency: 220-229. DOI and conference proceedings.

  • National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: National Academies Press. National Academies report summary.

  • National Institute of Standards and Technology. 2023. Artificial Intelligence Risk Management Framework (AI RMF 1.0). NIST AI 100-1, January 26, 2023. NIST publication DOI.

  • Noll, Alexander, Robert Salzmann, and Mario V. Wuthrich. 2020. "Case Study: French Motor Third-Party Liability Claims." SSRN working paper, originally posted April 18, 2018, revised March 5, 2020. SSRN paper and revision history.

  • Roberts, David R., Volker Bahn, Simone Ciuti, Mark S. Boyce, Jane Elith, Gurutzeta Guillera-Arroita, et al. 2017. "Cross-Validation Strategies for Data with Temporal, Spatial, Hierarchical, or Phylogenetic Structure." Ecography 40 (8): 913-929. DOI and journal article.

  • scikit-learn developers. n.d. "Tweedie Regression on Insurance Claims." Official scikit-learn examples documentation, accessed November 14, 2026. Official insurance regression example.

  • scikit-learn developers. n.d. "3.1. Cross-Validation: Evaluating Estimator Performance." Official scikit-learn user guide, accessed November 14, 2026. Cross-validation user guide.

  • scikit-learn developers. n.d. "GroupKFold." Official scikit-learn API reference, accessed November 14, 2026. GroupKFold API documentation.

  • Varoquaux, Gael. 2018. "Cross-Validation Failure: Small Sample Sizes Lead to Large Error Bars." NeuroImage 180 (Part A): 68-77. DOI and indexed abstract.

  • Wilkinson, Mark D., Michel Dumontier, IJsbrand Jan Aalbersberg, et al. 2016. "The FAIR Guiding Principles for Scientific Data Management and Stewardship." Scientific Data 3: 160018. DOI and journal article.

  • Wuthrich, Mario V., and Michael Merz. 2023. Statistical Foundations of Actuarial Learning and Its Applications. Springer Actuarial. Cham: Springer. Open-access monograph. Publisher book and DOI.