RESEARCH DOSSIER
Underwriting Model Calibration under Portfolio Selection
Abstract. Underwriting combines statistical risk prediction with institutional decisions about eligibility, coverage, terms, pricing, and capacity. These decisions determine which insurance exposures subsequently generate observed claim and cost outcomes. A model can therefore exhibit excellent calibration on the historical bound portfolio while having unknown or materially different calibration among applicants outside that portfolio. This monograph develops a rigorous scientific account of the resulting evaluation problem through conditional expectations, exposure-weighted actuarial cost metrics, propensity weighting, partial identification, counterfactual policy evaluation, disaggregated validation, and comparative supervisory governance. It distinguishes ordinary covariate shift from outcome-dependent selection, contract-induced outcome changes, incomplete claims development, and selection through cancellation or nonrenewal. The analysis integrates actuarial auto-calibration research, empirical studies of insurance selection, selective-label and off-policy methods, insurance industry monitoring studies, and official professional and supervisory sources. Transparent synthetic examples demonstrate aggregate cancellation of subgroup calibration errors, the limits of propensity adjustment, the width of identification bounds, and the economics of an acceptance threshold under uncertainty. A model's evidentiary scope follows its population, outcome, contract, time horizon, and historical decision process. Transport to a broader population requires documented overlap and conditional comparability; extension to unobserved actions requires either new information or acknowledged partial identification. The contribution is an original comparative and mathematical synthesis of established methods, with explicit inferential boundaries and a reproducible validation design. It claims no new named theory, carrier-level empirical calibration estimate, or universally optimal underwriting threshold.
Keywords: Underwriting; portfolio selection; selective labels; calibration; auto-calibration; pure premium; exposure weighting; adverse selection; covariate shift; inverse probability weighting; positivity; partial identification; counterfactual evaluation; mortality slippage; model risk; temporal validation; insurance governance.
Introduction and Conceptual Foundations
Research object and institutional setting
An underwriting model predicts an insurance-relevant outcome conditional on information available at a particular decision time. A portfolio is the changing collection of contracts acquired, retained, and administered under a set of underwriting, distribution, and contractual rules. Calibration describes agreement between predictions and the corresponding outcome distribution. The scientific object of this study is the interaction between calibration and the decision process that produces the observed insurance portfolio.
An insurer receives proposals and applications, determines eligibility, requests information when necessary, estimates risk, chooses an available offer or declines the application, and records the applicant's response. A successful offer can become a bound contract. A bound contract can be canceled, renewed, amended, or converted. A claim must occur within the relevant covered exposure, be reported and administered, and develop sufficiently to support a defined cost measure. Each stage changes the information available to an evaluator. Even a perfectly managed claims database is a record of a selected economic process.
Consider a property underwriter who rejects structures with unacceptable flood exposure, offers restricted deductibles to an intermediate class, and accepts other structures at ordinary terms. Subsequent incurred claims provide evidence about the insured structures under their actual terms. They provide no direct record of what the rejected structures would have cost under the ordinary contract. The absence of a recorded indemnity on a rejected property reflects the absence of that insurance obligation; coding the missing counterfactual as a zero loss would mechanically treat rejection as evidence of safety.
A related problem appears in life insurance. Applications routed through medical examinations and those approved through accelerated procedures may differ in health information, age, amount applied for, or willingness to complete additional requirements. The portfolio eventually observed after both pathways integrates those differences. An aggregate observed-to-expected mortality ratio merges risk prediction, underwriting routing, applicant completion, issue decisions, exposure duration, lapse, and measurement of death claims.
The selective-label literature makes this general observation explicit. Lakkaraju, Kleinberg, Leskovec, Ludwig, and Mullainathan (2017) study decisions that determine whether the outcome needed for evaluation is observed in their selective labels analysis. Their motivating judicial example is distinct from insurance; the transferable insight is the dependence of the observed evaluation sample on earlier decisions. Insurance adds contract-specific outcomes, monetary severities, exposure measurement, customer choice, and long development periods. These features make a direct transfer of a binary classification procedure insufficient.
Research questions and definitions
The first question is how selection alters calibration across risk subgroups and time. The second is which temporal, cross-carrier, or portfolio-standardized validation designs can sustain claims about generalizability. The third concerns the conditions under which selection-adjusted predictions can inform acceptance and terms without equating an uncertain prediction with a justified underwriting action.
Let X denote observable proposal-time risk characteristics and let U represent additional information or latent risk that is unavailable in the analytical data. Let A denote the underwriting action, including an offer with specified contract terms or a refusal. Let B equal one when a contract is bound and zero otherwise. Let M identify sufficient outcome observation at the analysis date. A simplified observation indicator S equals B times M; its exact construction is outcome-specific. For a fixed reference contract a and horizon h, Y(a,h) denotes a potential insurance outcome under that contract and defined exposure conditions. The observed outcome corresponds to Y(A,h) where consistency holds. A monetary payment under a contract actually declined is a different quantity from the hypothetical payment had coverage been provided.
The distinct target populations are all eligible proposals at decision time; proposals screened into a quotation channel; offers accepted by customers; bound exposures; mature bound exposures; and the target portfolio created by a proposed future policy. Each is scientifically legitimate when clearly named. An application-level event probability assigns one unit of weight per applicant or proposal; an exposure-year cost rate assigns proportional weight to duration or another appropriate exposure measure. An insurer seeking annual expected cost across a commercial property portfolio may choose insured location-years, building-years, or another unit, but the choice must remain coherent with the loss variable.
A model score p(X) for a binary event is calibrated on population Q when the conditional event mean given the score equals that score, Q-almost surely. A cost prediction m(X) is calibrated on Q when the conditional cost mean given the prediction equals the prediction for the appropriately defined cost unit. These are statistical conditions on distributions. A decision threshold is a separate economic and institutional mapping of estimated outcomes to actions.
Research method and evidentiary design
This study is a documentary and mathematical synthesis, guided by the individual Underwriting scientific design. Sources were selected for identifiable contributions: original scholarly abstracts and accessible articles for estimation and insurance choice, a published actuarial recalibration article and author manuscript, primary insurance practitioner monitoring summaries, dataset documentation, and official regulatory and professional materials. Publisher abstracts support narrowly stated reported findings, not uninspected coefficients, robustness tests, or causal generalizations. The full empirical records of applicants, claims, medical audits, and carrier scoring policies were not available; none of the calculated illustrations is presented as real business experience. The evidence ledger identifies the reviewed material and maps each citation to its scholarly claim and chapter.
The analytical method separates description, identification, estimation, decision, and governance. A descriptive ratio summarizes a specified historical sample. Identification establishes whether a target quantity is determined by observable data and stated assumptions. Estimation introduces statistical error given a chosen identification argument. Decision analysis evaluates possible actions under loss, utility, resource, legal, and capacity constraints. Governance documents responsibility, review, limitations, and monitoring. These distinctions structure the remaining chapters and prevent a computationally precise estimate from being mistaken for evidence about an inaccessible outcome.
Selection-Induced Data Shift
How the observation mechanism changes the population
Write f(x) for the distribution of measured characteristics in a target applicant population and e(x) for the conditional probability that a usable outcome is observed. Then, provided the target population is the source of the observed portfolio, the observed density is proportional to e(x)f(x). Restrictions on age, insured amount, occupancy, geography, previous claims, or distribution channel may cause e(x) to vary dramatically. A scorecard optimized on the observed portfolio minimizes its training loss under this altered distribution.
Conditional comparability determines whether this distributional change also changes outcome relationships. For a fixed contract a, suppose the reference outcome is independent of S given X. Within the support of observed risks, the conditional mean among observed applicants equals the corresponding target conditional mean. Selection has changed the composition but not the conditional response. Yet a score p(X) that compresses several characteristics into one scalar can be calibrated in one mixture and uncalibrated in another if its residual errors differ within a score cohort. If p(X) equals the true conditional mean for all X, calibration holds for any mixture of those X values under the same outcome mechanism. Calibration conditional only on the score is weaker.
Sugiyama, Krauledat, and Müller (2007) formalize covariate shift adaptation for changed input distributions with an invariant conditional response. Importance-weighted validation can adjust evaluation to the target mix within this setting. It does not supply labels for unsupported risk classes, and its covariate-shift assumption is a substantive empirical and institutional condition rather than a mathematical identity.
A shift in the conditional response arises if underwriters see material information U omitted from the recorded model features. Among individuals with identical X, those with lower U-related risk can be preferentially selected. The regression trained on those selected risks estimates the conditional mean under their latent characteristics. Increasing model complexity in X cannot recover the distribution of U among individuals whose outcomes were never observed. The problem also arises through applicant choice: applicants may know about risk characteristics not recorded by the carrier and respond to a quoted premium or contract condition accordingly.
For a binary event, the difference in conditional event means can be expressed as follows. At fixed X, with e = Pr(S = 1 | X), the target mean equals e times the observed conditional mean plus (1 − e) times the unobserved conditional mean. Thus the bias from substituting the observed mean is (1 − e) times the difference between observed and unobserved means. At e = 0 no observed conditional mean exists. A narrow-looking confidence interval around the experienced mean says nothing about the missing conditional mean absent additional assumptions. This simple decomposition locates the selection risk precisely.
Offer, take-up, exposure, and development
It is useful to decompose observation probability sequentially into the probability of an offer given proposal information, the probability of binding given the offer and information, and the probability of adequate observation given binding and the relevant recorded history. This is a chain of conditional probabilities, not an assumption of independence. For example, an applicant willing to accept a high deductible can differ from another applicant with the same estimated expected cost. A policy with a short duration or a poorly matured claim has a different measurement process from an annual fully developed policy. Conditioning on the eventual contract or on a claim that has already occurred can introduce variables unavailable at quotation time.
A missing outcome and a zero outcome must be defined separately. On a bound and completely observed policy, zero covered claims is a meaningful observed value. On a rejected proposal, zero insurer paid claims is an accounting consequence of the rejection. Under the hypothetical reference contract, the outcome remains unobserved. On a bound policy observed before the reporting window has elapsed, a currently zero claim count may represent either no event yet or an event not yet reported. The analyst who merges these categories manufactures a favorable training signal precisely where the evidence is absent.
Economic selection reinforces this issue. Finkelstein and Poterba's (2002) U.K. annuity selection study reports greater longevity among annuitants and larger mortality differences in the voluntary market than in the compulsory market. This is empirical evidence that participation and institutional setting shape the risk distribution. A mortality table estimated among purchasers consequently describes a specific selection environment, subject to its other assumptions and period. It cannot alone identify mortality among all those who could purchase.
The contract can change the outcome as well. Einav and colleagues (2013) investigate selection on moral hazard using employee-level panel data from one firm. Their reported evidence concerns heterogeneous medical-utilization responses and selection based on anticipated responses. The present underwriting implication is methodological: when coverage alters behavior, comparisons across contracts combine selection and contractual effects. A deductible adjustment can change claim reporting and economic incentives even if underlying physical hazards remain stable. A lifetime benefit, policy limit, or coverage exclusion changes the payment function even if the latent event process does not change.
Temporal selection and the feedback cycle
Historical outcomes reflect policy versions in force when proposals were evaluated. A carrier that raises a minimum premium can lose low-cost applicants with attractive competing offers. A change in referral rules can increase the proportion of cases reviewed by experienced underwriters. A distribution partnership can alter geography and applicant wealth. These changes can transform the observed portfolio while all coefficients in the prediction model remain fixed.
Selection also creates feedback. A risk model can influence premium and eligibility; its decisions determine future observed labels; those labels are used to update the risk model. An unsupported group repeatedly assigned a high predicted cost will generate little contradictory evidence if it is systematically excluded. Conversely, the most profitable accepted group can become overrepresented in training and validation. This is a mechanism of endogenous measurement rather than proof that a particular carrier's estimates are biased.
Distinguish three monitoring targets. Proposal drift tracks shifts before the carrier's decision. Offer and binding drift tracks the difference between incoming and insured mix. Outcome drift tracks changes in mature claim relationships under the actual contract. For each month, reporting both the distribution of proposal scores and the distribution of bound scores identifies selection at the observable-score level. Comparing mature calibration in fixed cohorts tests a different property. A dashboard showing only the latter misses application segments that have ceased to enter the portfolio.
A useful counterfactual decomposition holds either the distribution of measured characteristics or the conditional cost function constant while changing the other. The first component estimates composition-related change; the second reflects changing conditional outcomes. In practice the decomposition depends on a reference distribution, contract alignment, common support, and appropriate censoring adjustment. It is not uniquely determined when entire risk cells appear or disappear.
Synthetic composition example
Suppose 10,000 otherwise comparable full-year applicants share a model prediction of 10% claim probability. Half belong to group L, with true event probability 5%, and half to group H, with probability 15%. In the target population the mean is 10%; the score is calibrated in the aggregate despite errors in both groups. Underwriting accepts 90% of L and 10% of H, with selection random within each group. There are 4,500 selected L and 500 selected H. Expected events are 225 + 75 = 300, against 500 predicted events, giving observed-to-expected E[O]/E = 0.60 in the selected portfolio.
An intercept correction that makes every selected prediction 6% restores aggregate agreement on that selected mixture. Applying the 6% probability to the original applicant population gives 600 predicted events versus 1,000 expected events, an observed-to-expected ratio of 1.667. Neither initial aggregate calibration nor portfolio recalibration established subgroup validity. The model's original p = 0.10 was mathematically compatible with one mixture only. This example is an exact expectation calculation, not a stochastic sample or an observed carrier result.
The example also demonstrates what selection did not do. Since selection is random within group, the conditional L and H means remain 5% and 15%; access to both groups allows a correctly specified model incorporating group membership to estimate both. This differs from the more difficult case of unobserved selection within group. Knowing the rate at which a group was accepted is necessary for some adjustments but does not prove independence of selection and potential outcome.
Absolute Cost Calibration
Probability, frequency, severity, and contract cash flows
The target of actuarial prediction must follow the unit of loss. Binary claim occurrence answers whether at least one eligible claim happens during a fixed follow-up window. Claim frequency counts events per unit exposure. Severity describes the amount associated with a defined claim. Aggregate cost includes the sum of claim amounts for an exposure. These quantities have different denominators and may require different likelihoods.
The French motor frequency vignette in CASdatasets documents the separation of policy-level claim counts and exposure from the associated claims table. That structure supports frequency and severity modeling on insured contracts. It does not establish the coverage of declined proposals. A study using such data may credibly analyze risk differentiation within the documented portfolio if the data and assumptions support it, but extending a calibrated result to all applicants would require additional records.
Let T be exposure duration, N be the claim count over T, and C the ultimate cost. A frequency model with rate lambda(X) predicts E[N | X,T] = T lambda(X), when proportional duration is appropriate. A severity estimate nu(X) can represent anticipated cost per claim. Under the relevant compound-count assumptions, and particularly when conditional mean severity does not depend on the realized count, E[C | X,T] = T lambda(X) nu(X). More generally, define nu(X,T) as E[C | X,T] divided by E[N | X,T] for a positive denominator. This ratio is a claim-count-weighted conditional average; it avoids an unjustified independence assumption but no longer automatically implies that severity is constant across durations.
A log-link generalized linear frequency model commonly uses log(T) as an offset. To interpret the resulting annual rate, duration must genuinely scale the expected number of claims after the chosen covariates and exposure definition. Seasonal concentration, newly acquired risks, and cancellation related to unobserved risk can break a simple duration scaling law. Separating annual rate from event probability also matters. A constant Poisson frequency of 0.2 per year yields probability 1 − exp(−0.2), approximately 0.1813, of at least one event over a full year. Frequency and the event probability coincide only approximately for sufficiently rare events.
Aggregate insurer payments depend on the terms of coverage. A simple per-occurrence indemnity for a loss L under deductible d and maximum insurer payment l is min(max(L − d, 0), l), before expenses or special conditions. Under proportional cession, retained losses may be further reduced by reinsurance, subject to actual treaty provisions. A policy with aggregate limits, different occurrence definitions, reinstatements, deductibles per claimant, or exclusions needs a more specific transformation. Thus observed cost calibration under a high-deductible policy cannot be transferred mechanically to full coverage without modeling the ground-up loss and contract response.
Define pure premium as expected covered claims cost per chosen exposure unit. Commercial premium ordinarily includes additional amounts reflecting expenses, taxes where relevant, capital and profit considerations, and distribution arrangements. A loss ratio comparing claims with charged premium is an economic performance metric. It is also affected by price levels and can change when the risk model remains unchanged. A calibrated pure premium supports cost estimation but does not require a particular commercial price.
Calibration hierarchy and measurement
For equal full-year exposures, aggregate observed-to-expected cost is total realized cost divided by total predicted cost. For variable exposures, the denominator is the sum of T times predicted cost rate. A ratio close to one measures global balance, subject to random variation, claim maturity, and consistent units. When the denominator is zero, the ratio is undefined. Low expected totals yield unstable ratios that should be accompanied by counts and intervals.
The calibration literature in predictive medicine offers useful distinctions. Van Calster and colleagues (2019) discuss calibration intercepts and curves and distinguish mean, weak, moderate, and strong forms of agreement. For binary risks strictly between zero and one, logistic calibration fits logit Pr(Y = 1) = alpha + beta logit(p). With beta fixed at one, alpha is a calibration-in-the-large correction on the log-odds scale. With both parameters free, ideal intercept and slope are zero and one respectively. A flexible observed-versus-predicted curve evaluates score-cohort agreement that a single intercept and slope might miss. These definitions are imported as statistical methods, not medical regulatory standards for insurance.
For cost models, Wüthrich and Ziegel (2024) examine isotonic recalibration as a means of achieving empirical auto-calibration through pooling observations by ordered predictions. Their published work and author manuscript distinguish the empirical property from an out-of-sample guarantee. A cost model with an intercept can balance the training total under suitable estimation equations and weights. A flexible recalibration can improve fitted cohort agreement. Both achievements leave future validation and the selection mechanism to be tested.
Let Q be the probability measure used for validation and R = C/T a rate when T is positive. Exposure-weighted calibration is a statement under the distribution formed by weighting each contract by T. Policy-weighted calibration instead gives each contract equal weight. If annual protected properties and short-duration canceled policies have different loss rates, both measures describe different estimands. The exposure-weighted squared error and the policy-weighted squared error therefore need not prefer the same model. A published comparison should state the weighting, outcome scale, and maturity rule before presenting the ranking.
A binary Brier loss is the mean of (Y − p)^2; logarithmic loss is the negative mean of Y log(p) + (1 − Y) log(1 − p). Both are proper for binary predictive probabilities on the distribution of observed outcomes. Proper scoring does not make an unrepresentative distribution representative. In the population, expected Brier loss conditional on X equals q(X)(1 − q(X)) + (p(X) − q(X))^2, where q is the true conditional event probability. Thus minimizing expected Brier loss recovers q if unrestricted, while the realized score combines irreducible event variation and model error. The decomposition explains why a Brier score can change with the risk mix even when conditional predictions remain correct.
AUC, rank correlation, Gini-type measures, and lift diagrams focus on ordering or discrimination, with their exact definitions and weights specified. A strictly increasing transformation of a binary score preserves its ordering and, with unchanged tie structure, its AUC. It can alter predicted probabilities substantially. High discrimination therefore cannot demonstrate absolute calibration, and a calibrated constant prediction can be uninformative for classification. For cost distributions, tail coverage, quantiles, predictive intervals, and portfolio covariance deserve separate validation if the model is intended to support capital or limit decisions.
Monetary subgroup cancellation and claims development
Consider two accepted groups, each containing 500 full-year exposures. Both have predicted pure premium of 600 dollars. For group A the ultimate observed cost rate is 400 dollars; for group B it is 800 dollars. Predicted total cost is 600,000 dollars and actual total cost is also 600,000 dollars. Aggregate O/E is one, while group O/E values are 0.667 and 1.333. A commercial rate differentiated on other grounds might mask or magnify the financial effects. The example demonstrates why global portfolio balance can coexist with substantial cross-subsidization of predicted cost cohorts.
Now suppose the same portfolio is examined before liability claims have matured, with recorded losses of 480,000 dollars and an actuarial estimate of an additional 120,000 dollars. The raw observed-to-expected ratio is 0.80; the developed ratio is one. The second ratio compares prediction to an estimated ultimate outcome, not an observed ultimate value. Its uncertainty includes development assumptions. A calibration report must distinguish paid, case-incurred, reported and estimated ultimate measures and document the valuation date, inflation basis, currency, claim handling expenses and recoveries. Claims reserving estimates must avoid using future information that would not have been available at the model decision date, but development may appropriately use later information when constructing validation labels.
Long-tailed liability, major property catastrophes, and disability duration create additional variation. A correct mean can coincide with poorly estimated tail dependence or reserve emergence. When two large industrial risks share a flood zone, individual conditional expected loss predictions contribute additively to portfolio expected loss, but the joint tail distribution depends on their dependence structure. Individual calibration cannot establish adequacy of aggregate capital or reinsurance protection. The boundary preserves the independent scientific roles of underwriting, pricing, reserving, and portfolio risk modeling.
Recalibration choices
An intercept-only update accommodates global level changes while preserving ordering; a slope adjustment addresses over- or under-dispersed score estimates on the chosen link scale. A flexible regression of outcomes on scores can model nonlinear cohort error when sufficient data support it. Isotonic recalibration imposes monotone predicted means and pools sparse adjacent regions; if the original ordering is materially wrong, monotonicity can hide subgroup-specific reversal rather than solve it. A full refit allows changes in predictor relationships but increases estimation and implementation risk.
These procedures require separate training, recalibration, and final evaluation data. In an out-of-time design, fit the base model on earlier cohorts, estimate the recalibration on the next fully usable cohorts, and evaluate both on a later independent cohort. If the later outcomes mature only partially, report the agreed development method and its sensitivity. Selecting the method using the final test set and reporting its performance on that same set makes the evidence optimistic. A recalibration applied after a portfolio selection change must use the intended target weights and a model version that matches the scored proposals.
Disaggregated Risk Groups
Cohorts, support, and statistical uncertainty
Disaggregated calibration evaluates differences with respect to scientifically relevant and legally permissible strata. Underwriting dimensions include score range, product, distribution channel, referral path, contract terms, risk class, region, issue age in applicable life business, time since inception, and exposure period. Governance evaluation may also require assessing outcomes across protected groups subject to lawful data access and jurisdiction-specific requirements. Variable availability for monitoring does not in itself establish legality or suitability for underwriting.
An observed subgroup with 100 full-year exposures, each predicted to have 2% event probability, has two expected events. Observing four yields O/E = 2. In a homogeneous independent Bernoulli model with p = 0.02, expected count variance is 1.96 and standard deviation is 1.4. Observing four is compatible with meaningful random variation; the point ratio alone does not establish systematic underprediction. Real data may exhibit dependence, heterogeneous risks, and estimated predictions, making a fixed binomial calculation insufficient. Calibration curves should present exposure, expected and observed counts, intervals, and the method used to estimate uncertainty. Sparse intersections should be labeled, pooled with documented assumptions, or withheld from strong conclusions.
A subgroup with no bound exposure has no empirical bound-outcome calibration estimate. Reporting its error as zero because its observed and predicted costs total zero is mathematically and institutionally wrong. Reporting it as poor calibration is also unsupported. Its status is unvalidated under the relevant target. This distinction matters especially near eligibility thresholds, where an individual decision depends on a score extrapolated into the least evidenced region.
Hierarchical partial pooling can stabilize subgroup deviations by estimating shared and group-specific effects. A simple logit calibration model uses an overall intercept and random subgroup deviations; groups with little information shrink toward the overall relationship. The shrinkage assumptions determine how much information is borrowed. They should be validated against known group heterogeneity, and raw counts should remain visible alongside model-based estimates. Shrinkage produces a conditional estimate under a model; it does not provide new independent outcomes for empty groups.
Repeated exploratory tests raise multiple-comparison concerns. Predefine priority groups and materiality thresholds, report interval estimates, and distinguish an investigation signal from a conclusive finding. When many policy periods represent the same insured or broker, bootstrap or covariance estimation should account for that grouping. Weather shocks and catastrophes can create period-level dependence. A random split of rows across training and testing can leak exposure history; entity- and time-aware splits are more defensible for genuine prospective validation.
Temporal and cross-carrier designs
An out-of-time evaluation holds out cohorts that were not available at model development and reconstructs each proposal as seen at its decision date. It preserves policy version, pricing and eligibility rules, source-data timestamps, feature definitions, and the score generated at that time. Rather than re-scoring historical proposals with today's enriched data and treating those scores as past forecasts, it evaluates the version actually deployed or clearly labels a retrospective reconstruction. This ensures the study distinguishes model degradation from retrospective information leakage.
A rolling cohort design monitors recurrence of errors across seasons and policy changes. If mature claims take several years, leading indicators can be tracked at shorter intervals while final outcome metrics mature later. Leading indicators may include drift in records, referrals, non-disclosure audits, override patterns, and the rate at which applications are withdrawn. Their function is early diagnosis; their relationship to actual losses is a separate validation question.
Cross-carrier validation needs common outcome definitions, contract terms, hazard dates, currency and inflation basis, exposure measures, underwriting coding, and relevant data provenance. If carrier A covers comprehensive losses while carrier B covers collision only, an apparently high O/E for B could result from aligning predictions to the wrong outcome. A model's cross-carrier rank may depend on which risks each institution accepts and the other carrier's competitive position. Standardizing covariate distributions can address some differences only where common support and conditional comparability are credible. Report performance on all observed external business, on the overlap population, and where possible on a prespecified target mix; these quantities should never be conflated.
A reference portfolio sampled from participating carriers is usually not a random sample of the total market. A leave-one-carrier-out design helps reveal sensitivity to institutional context but still inherits the participating carriers' collective selection. Data-sharing restrictions and differences in underwriting information may prevent full harmonization. Where lawful data cannot be pooled, standardized local summaries and comparable evaluation code are possible alternatives, with privacy safeguards and explicit treatment of heterogeneity.
Evidence from accelerated life underwriting
Life insurance offers a particularly clear example of the distinction between an audit proxy and an observed outcome. Seeman and Herzog (2024) describe random holdouts and post-issue audits for accelerated underwriting, using additional evaluation before issuance or medical records after the offer or decision. Their practitioner report describes more than 33,000 monitored lives across 30 accelerated programs as of year-end 2023. The study uses misclassification-based monitoring to estimate future mortality implications given limitations in realized mortality experience. This is substantive primary industry evidence about monitoring methods, but its proxy estimates should not be labeled as realized death-claim rates.
The two audit timings generate different observation mechanisms. A randomized holdout for additional examination can alter the applicant's willingness to continue and can affect the eventual underwriting outcome. A post-issue audit observes a selected issued population and relies on available records. Agreement with a fully underwritten risk class is not equal to calibrated long-term mortality: the benchmark class itself is underwritten and approximate. A proxy is useful when its association with the eventual target is understood and tested as claims accumulate.
Klein's (2024) accelerated underwriting practices survey describes a 2022 survey with 24 responding direct companies and seven reinsurers. It documents changes in program operation and compares methods for estimating mortality slippage. Those data are responses from participating organizations, with estimates dependent on assumptions and practices. They do not establish a uniform industry-wide deterioration or validate all accelerated models. For evaluation design, the most important information is the coexistence of changing eligibility, algorithms, data sources, and monitoring procedures. Each can change the observed cohort.
For a mortality model, expected deaths may be computed by summing individual age-, duration-, and contract-relevant probabilities over intervals at risk. Expected benefit costs require weighting by the insured benefit and accounting for policy status and coverage. A lapse-related reduction in observed deaths cannot automatically be interpreted as lower risk per original applicant. Hazard estimation needs a clear treatment of censoring and exposure exit. If evaluating mortality under a sustained policy, informative lapse requires additional assumptions or evidence; if evaluating actual portfolio claims, lapse can be part of the target process. A study must state which meaning is intended.
Counterfactual Evaluation
Identification and inverse probability adjustment
The core counterfactual research question concerns the mean under a clearly specified reference insurance action a for a named target population. Let S indicate observation under that action; assume consistency, conditional exchangeability between S and the potential outcome given X, and positive observation probability e(X) wherever the target has positive mass. The assumption of exchangeability concerns the outcome under the reference contract and the availability of its label; it does not follow from the observational design.
Under these assumptions, E[S Y/e(X) | X] = E[Y(a) | X]. The proof uses the conditional probability of observation multiplied by the mean of the compatible observed outcome, then divides by that probability. Taking expectations over the target X distribution identifies the target mean. The same reasoning applies to E[Y(a) | p(X) in a score band] by restricting the expectation and using the target band denominator. A Horvitz–Thompson-type estimator divides weighted observed outcomes by the number of target units; a self-normalized ratio divides by the sum of weights. The ratio often moderates scale fluctuations, but is not exactly unbiased in every finite sample.
Rosenbaum and Rubin (1983) establish the role of the propensity score for adjusting observed treatment-assignment differences. In underwriting, e(X) may describe the probability of observing the specified outcome through acceptance, binding, and adequate measurement. A model for acceptance alone is inadequate when take-up or observation also depends on variables related to risk. The original study supplies an adjustment principle for measured covariates, not evidence that actual underwriting selections satisfy it.
When the desired target comes from a different proposal distribution, selection weighting may be combined with the ratio of target to source covariate densities. Estimating these ratios requires data about target features and the observation mechanism. A single density-ratio estimator can directly map the observed source X distribution onto the target X distribution under conditional invariance. The analyst should avoid multiplying distinct estimates that both correct the same selection mechanism, which would double-adjust the data.
Propensity probabilities may be estimated from all recorded decisions by a statistical model, or obtained from documented randomized assignments. Underwriters can override deterministic eligibility rules; if the override criteria are omitted, estimated probabilities may obscure unmeasured selection. Good propensity discrimination is not the objective. Calibration of the selection probabilities, covariate balance after weighting, common support and the reasonableness of the assignment model are relevant diagnostics. Known randomized audit probabilities provide clearer design information than a classifier trained to reproduce policy decisions.
Synthetic weighting and effective sample size
Return to the earlier two-group example. The 4,500 selected L exposures have e = 0.9, while 500 H exposures have e = 0.1. Inverse weights restore expected group totals of 5,000 each. Weighted event counts are 225/0.9 + 75/0.1 = 250 + 750 = 1,000 and the weighted denominator is 10,000. The adjusted observed event rate is 10%, exactly reproducing the synthetic target expectation. This calculation works because selection within each group was stipulated to be independent of its outcome.
The effective sample size commonly used to summarize unequal positive weights is (sum w)^2 / sum(w^2). Here it is 100,000,000 divided by 4,500/(0.9^2) + 500/(0.1^2), which equals 1,800. The nominal 5,000 observed exposures do not provide equal-weight precision equivalent to 5,000 independent cases under this weight summary. This quantity is a diagnostic for weight concentration rather than an exact variance formula; outcome heterogeneity, clustering and estimated propensities still matter.
If H selection probability falls to zero, no finite weight can restore H outcomes. Trimming large weights reduces variance while usually changing the target or introducing bias. A report should specify whether it targets the full population, a trimmed population, an overlap population, or a stabilized approximation. Comparing several clipping thresholds can reveal fragility, but choosing thresholds based on favorable results conceals the underlying support limitation.
Missing-not-at-random outcomes and partial identification
Suppose a score band contains 1,000 applicants, of whom 700 have observed full-year reference-contract outcomes. There are 35 events. Among the observed, event probability is 5%. If the 300 missing potential outcomes are binary and no assumption about them is justified, the target event rate is bounded by 35/1,000 = 3.5% and (35 + 300)/1,000 = 33.5%. The bounds are attainable by setting all missing outcomes to zero or one respectively. They are bounds on the completed finite population, not confidence intervals for a superpopulation parameter.
If external evidence supports an assumption that the missing group event probability lies between 8% and 20%, a sensitivity interval for the combined population is 5.9% to 9.5%: 3.5% plus 0.3 times each endpoint. Those numbers illustrate assumed ranges rather than evidence that the true missing group belongs to them. If the model prediction is 6%, the unrestricted bound cannot establish calibration. The restricted sensitivity scenario makes calibration compatible with some assumptions but not others.
For a nonnegative bounded insured-payment outcome with maximum M, the same logic yields a lower mean equal to the sum of observed payments divided by the target count and an upper mean adding M for every missing unit. The bound requires a genuine maximum for the specified horizon and cost variable; a per-occurrence limit does not cap annual aggregate cost when multiple claims are possible. With unbounded nonnegative costs, no finite upper bound follows from missingness alone. Additional actuarial assumptions about the tail, contract aggregate limit, or external risk information are needed.
Sensitivity analysis can express missing-outcome departure through an additive probability difference or a log-odds shift conditional on X. For positive cost means, a multiplicative excess factor for missing risks may be more natural. A grid should justify the ranges, calculate resultant estimates, and show the decision consequences. If all credible scenarios favor the same conservative action, the decision is robust to that specified uncertainty. If scenarios flip the decision, acquiring relevant data or restricting the decision scope has value.
Outcome models and doubly robust estimation
An outcome regression m(X) estimates the conditional mean from the observed compatible sample. Averaging its predictions over the complete target proposal distribution is a direct standardization estimator. It can be more stable than inverse weighting when high-dimensional propensity scores are small, but relies strongly on the outcome model. Extrapolation outside observed support has no empirical calibration check, even if the regression yields smooth finite predictions.
The augmented inverse-probability estimator combines both approaches. For a mean, its empirical expression is the average over all target proposals of m_hat(X) + S/e_hat(X) times the residual Y − m_hat(X), with the residual contribution defined only when S equals one. Under consistency, exchangeability, positivity and suitable nuisance estimation, the conditional expected residual correction restores the target mean when e is correctly specified even if m is not, or disappears in expectation when m is correct even if e is misspecified. This is the standard double-robust property, not a guarantee under latent confounding, zero support, or arbitrary misuse of model training data.
Dudík, Langford, and Li (2011) investigate doubly robust policy evaluation with partially observed action-dependent rewards. An insurance application takes the historical action log seriously: the recorded reward or loss corresponds to the action chosen. Candidate policies may offer different terms, request further data, or decline. For an action-specific value estimate, a reward function and historical action propensity must refer to that action and its target contract. A single binary selection propensity cannot evaluate multiple differentiated offers without additional structure.
Cross-fitting is one way to reduce overfitting in nuisance models: train propensity and outcome regressions on folds outside each evaluation observation, then construct the weighted or augmented contributions on held-out data. Folds must honor temporal and entity structure; using future cohorts to predict historical selection can invalidate a prospective test. In finite samples, very small propensities and heavy-tailed losses can destabilize the augmented estimator even when the theoretical identification assumptions are satisfied.
Acceptance economics and the limits of off-policy estimates
Let the expected contribution from an accepted reference contract be premium P, less expected covered claims m, expenses c, and a stated capital or capacity charge k. Ignoring further interactions and using a risk-neutral contribution target, acceptance is economically attractive relative to a zero-value rejection if P − m − c − k exceeds zero. Other decision criteria may include reinsurance availability, market conduct, legal requirements, service obligations, reputation, or portfolio concentration. A positive expected contribution is not a sufficient universal criterion.
In a synthetic example, take P = 900 dollars, c = 160 dollars, k = 90 dollars, and m = 600 dollars. The expected contribution is 50 dollars. If credible selection sensitivity puts m between 550 and 750 dollars, the contribution ranges from 100 to minus 100 dollars. The break-even expected cost is 650 dollars. Actual acceptance requires a decision about uncertainty and constraints; a central estimate alone hides the action's fragility.
For candidate actions that include different deductibles or premiums, the effect of the offer on take-up enters the target value. At decision time, the expected value per proposal may be modeled as probability of binding under action a multiplied by expected contribution conditional on binding, minus proposal-level acquisition or investigation costs where appropriate. The composition of binders can vary by action, so multiplying a new take-up rate by the old conditional loss can be incorrect. The action-dependent expected value must align with the joint behavioral and cost mechanisms.
A randomized additional-information audit can improve evidence about a comparison risk class without randomly insuring otherwise prohibited risks. A prospective controlled change to an eligibility policy requires legal, ethical, underwriting, and capital approval; it may be appropriate only within a predefined admissible range. Observed outcomes under a lawful exploration design can improve identification near thresholds. A study should never treat experimenting with applicant access as a purely statistical option. Nonintrusive alternative data, where lawful and relevant, can inform otherwise inaccessible risk, but proxies still require validation against the insured outcome.
Governance of Recalibration
Professional and supervisory contexts
The Actuarial Standards Board's ASOP No. 12 governs U.S. actuarial professional services within its risk-classification scope, including the association of risk characteristics with expected outcomes and considerations such as applicable law. The standard's published web record identifies the original effective date in 2006 and a May 2011 revision metadata date. It is relevant to the professional review of classification systems rather than an international law or a prescription for a specific statistical estimator.
The same board's ASOP No. 56, revised in December 2019 and effective October 1, 2020, addresses actuarial services involving models and the suitability of data and assumptions for their intended purpose. Its documented guidance supports evaluating whether data correspond to the modeled circumstances, including actual experience and appropriately adjusted other evidence. These principles are consistent with explicit disclosure of selection and target-population limitations.
At the international level, the IAIS Insurance Core Principles as revised in December 2024 identify underwriting and reserving risk within the broader insurer risk management system and connect that system to enterprise risk management for solvency purposes. The principles and ComFrame operate within a supervisory framework and are implemented through applicable regimes. They do not prescribe the particular propensity-weighting estimator used in this study. A carrier's model governance should connect calibration deterioration to risk appetite, portfolio exposure, solvency and controls rather than treat the scorecard as a disconnected analytic asset.
For artificial intelligence used in insurance, EIOPA's August 6, 2025 AI governance opinion is addressed to national supervisors and explains a proportionate, risk-based approach within insurance-sector legislation. This study uses the official description for that general supervisory orientation; it makes no clause-specific claim about a proprietary or separately uninspected text. Insurance entities should evaluate their actual legal duties under the relevant EU and national regime and the characteristics of the deployed system.
In the United States, the NAIC's AI insurance guidance records adoption of its Model Bulletin in December 2023 and describes expectations for responsible AI use and compliance with applicable laws. A model bulletin is a state-supervisory instrument for adoption and implementation; publication by the NAIC alone does not establish identical enforceable obligations in every state.
New York's final Circular Letter No. 7 dated July 11, 2024 addresses specified insurers' use of artificial intelligence and external consumer information in underwriting and pricing, with sections concerning fairness and governance. The letter identifies both quantitative and qualitative assessment and board-level oversight considerations. It supplies a jurisdiction-specific illustration of why error analysis, risk-characteristic justification and responsibility for use matter in addition to aggregate model accuracy. The study does not generalize New York requirements to every insurer or legal system.
Model ownership and the decision record
A calibration owner should define the population and outcome for which the model is approved. The record includes model version, reference contract scope, exposure units, decision date, training and evaluation periods, data availability, development assumptions, selection policy, covariate support, and the extent of the final evaluation. A change of target from the existing bound portfolio to all applicants requires a specific identification and evaluation justification even if the architecture remains the same.
The decision system and the statistical model require distinct version histories. An unchanged predictor can have a different institutional effect when a threshold, exception process, deductible schedule, or referral arrangement changes. Versioning only the model artifact makes it impossible to attribute calibration deterioration to a rule change. A versioned decision log should preserve the score received, action taken, available actions, offer terms, reasons, override category, consent or permissions when applicable, data snapshot and subsequent binding and exposure status. Access controls and retention should reflect relevant law, contracts, privacy and legitimate purpose.
Calibration monitoring should integrate three lines of inquiry: statistical adequacy for the target; performance of the underwriting decision and its portfolio consequences; and conduct and compliance assessment. A stable aggregate O/E can coincide with unfair exclusion or poor access, and a conduct metric can signal unequal effects without alone establishing statistical miscalibration. The inquiries require compatible evidence but have independent criteria.
Recalibration triggers and remediation
A scientific calibration review starts with data reconciliation. Confirm that claims labels and predictions refer to the same contract, gross or retained basis, monetary units, date, exposure and loss-development convention. Verify denominator changes, cancellations, handling of claim reopenings, mapping of risk classes, and score version. A changed currency conversion or reporting lag can imitate model drift. Correction of a data defect restores the intended definition; it should not be recorded as a change to underlying risk.
If genuine global calibration drift remains, examine frequency and severity separately and inspect group-level variation and proposal-to-bound changes. An intercept revision can be evaluated when errors are consistent with a shift in level. If score ordering or subgroup relationships have changed, test more structured recalibration or refitting. If the target includes new unsupported regions, an updated score alone cannot repair evidence absence. The model may need a restricted purpose, additional information, more cautious decision boundaries, or a specifically documented uncertainty analysis.
Predefined materiality thresholds can guide escalation and resource allocation, but statistical significance and business relevance should both be evaluated. An O/E exceedance based on a single large catastrophe might require capital review, but its interpretation as conditional mean deterioration depends on the event model and time horizon. A small statistically significant error across many low-value exposures can have substantial aggregate monetary implications. A practical threshold is therefore bound to both uncertainty and consequences.
Validation independent of the model development process should assess the revised system on a frozen later sample, review the decision logs, and confirm contract and population alignment. Independent here means organizational or procedural separation within the relevant governance framework, not external scholarly peer review of this monograph. A rollback plan should identify the last supported model and rules, observation cutoffs, and how exceptions are handled during restoration. Governance should not require an obsolete previous model to be used where it is itself materially unsafe or unsupported.
Vendor models require similar review. A vendor's reported score accuracy on a pooled portfolio says little about the target carrier without provenance, outcome definitions and representativeness. A carrier should establish the permitted uses, documentation, version control, relevant validation evidence, and recourse when results are unsupported. When source data or proprietary logic is unavailable, the scope of independent verification narrows. Model selection cannot turn vendor confidentiality into proof of calibration.
Analytical Synthesis, Evidence Gaps and Reproducibility
Integrating actuarial accuracy and the underwriting process
The preceding analysis supports three substantive conclusions. First, absolute calibration is an attribute of predictions relative to a named outcome distribution, and the bound portfolio is itself a product of institutional selection. Second, calibration at the score or aggregate level does not establish correctness across applicant subgroups, unsupported regions, or changing contractual actions. Third, statistical adjustment can extend inference only through justified identification conditions; it cannot transform the consequences of denial into measured hypothetical insurance experience.
These conclusions concern the application of established statistical and actuarial methods. The distinctive synthesis is that an underwriting model connects a prediction target to a measurement process partly controlled by the institution using the prediction. In ordinary supervised learning the observation process can sometimes be treated as an external sample design. In insurance, the prediction itself may shape eligibility, price and terms, and thus the insured experience by which the model is judged. Recording a score without recording the policy that used it obscures the mechanism of future validation.
This interpretation explains several apparently contradictory outcomes. A carrier can reduce actual claims totals by declining risks while losing predictive knowledge about the excluded group. A new model can improve O/E through more conservative pricing or changed selection while leaving conditional risk estimates worse. An intervention can improve absolute calibration yet decrease underwriting profit because of adverse take-up. An accurately calibrated individual cost model can support a portfolio with excessive concentration. Each outcome follows from a different quantity and comparison. A valid evaluation states these quantities separately.
The analysis further distinguishes economic self-selection from statistical missingness. Adverse selection describes the relation between risk information or preferences and participation or choice; selective labels describe the observability consequences of decisions. An eligible applicant may select a competing contract without the focal insurer having any denial decision. Conversely, an underwriting exclusion may create selective labels even if the applicant possesses no private risk information. Both can alter the observed population and can coexist, but causal diagnosis changes the remedy.
A reproducible data and evaluation specification
A defensible research dataset separates proposal, decision, contract exposure and claim entities. The proposal table includes a stable entity identifier, proposal time, versioned features and score. Decision records include permitted actions, actual action, terms, referrals, reasons and assignment probability where known from a designed process. Contract records include binding, coverage dates, exposures, limits, deductibles and changes. Claim records include event and reporting dates, amounts on a specified valuation basis, and relevant coverage status. An audit table records sampling probabilities, benchmark evaluation and completion or withdrawal. Each join is tracked: repeated proposals must not inflate unique-person denominators, and one-to-many claim rows must not multiply exposure totals.
Before evaluation, define the estimand in written form. An example is expected ultimate gross contractual claims cost per policy-year under the specified reference contract among proposals satisfying the lawful eligibility definition at initial quote time in a given quarter. Another is realized annual mortality deviation in currently in-force accelerated contracts at the agreed observation date. These are different questions. The first may involve counterfactual labels; the second is a portfolio outcome that can be estimated from properly observed exposure.
The proposed reproducibility sequence is:
- Freeze the code, source data dictionary, model and underwriting rule versions, target outcome and horizon.
- Reconcile proposal, decision, contract and claims counts; report missing predictors, no offer, no binding, exposure termination and inadequate maturity separately.
- Quantify selection and covariate overlap at each stage, including target and observed counts per priority group.
- Compute observed-portfolio global ratios, calibration curves, discrimination and uncertainty on the explicitly defined sample.
- Compute target-standardized results only for the specified identification design, state overlap and nuisance-model diagnostics, and compare the unadjusted, standardized and weighted estimates.
- Apply missing-outcome bounds and sensitivity scenarios wherever exchangeability is doubtful or support fails; describe which target quantities remain unidentifiable.
- Use frozen later and, where feasible, external carrier cohorts for the final assessment; record monitoring and governance implications independently.
- Report code version, statistical uncertainty, cost and exposure definitions, limitations, and which decisions are justified by the evidence.
This sequence is an operational synthesis of existing methods rather than a newly proposed named framework. It offers testable outputs that an independent analyst could reproduce with lawfully available data. No individual carrier has supplied such records for this monograph.
Falsification, competing explanations, and research gaps
The covariate-shift explanation would be weakened if conditional calibration deteriorates in stable, well-observed risk strata after outcome and terms are harmonized. A loss-development explanation would be weakened if the deviation remains after sufficient maturation and independently credible development estimates. A pricing-selection explanation would be weakened if take-up and contract composition remain consistent across offer versions and no meaningful behavioral difference is observed. A measurement-change explanation would be weakened if independent original records reproduce the same target labels and prediction inputs. No single observational test can exclude all unmeasured confounding; their collective value lies in discriminating plausible mechanisms.
A negative control may help identify a data pipeline change by examining an outcome or field expected to be unaffected by the underwriting model while sharing the data infrastructure. The validity of that control depends on its causal and operational relation to the event under study. A before-and-after comparison around an eligibility rule change may provide useful information, but simultaneous tariff, broker and hazard changes can confound it. Near a known threshold, a regression-discontinuity design might estimate a local causal effect under continuity and absence of precise manipulation; it would not automatically estimate risk throughout the rejected population, especially when rejection leaves the desired insurance outcome unobserved.
Potential empirical extensions include common-support comparisons of multiple insurers with aligned policy terms, prospective stratified random additional-information audits, and structured follow-up of legally accessible independent event measures. Each needs a pre-specified target and data-access approval. A carrier might investigate whether audit risk classes predict later realized mortality, whether propensity scores are calibrated against known randomized routing probabilities, how often subgroup calibration errors recur after a change in policy, and which target populations have sufficient overlapping exposure. The design should register failures and uncertainty as informative results rather than optimizing selection solely to yield acceptable metrics.
The empirical evidence reviewed here is bounded. The U.K. annuity and health-plan studies concern particular populations and reported research contexts. The accelerated-underwriting evidence is audit and survey information, not a statistically representative estimate of all carriers' ultimate losses. The mathematical methodology applies across the named settings only where its assumptions hold. Insurance loss tails, correlated catastrophes, mortality improvement, regulatory restrictions, and changes in consumer behavior require specific modeling. No universal effect size for selection bias can be inferred from these sources.
Numerical reproducibility and checks
All monetary amounts and counts presented in the hypothetical examples are invented to expose logical properties of the methods. The two-group example can be reproduced from group sizes 5,000 each, probabilities 0.05 and 0.15, and selection probabilities 0.9 and 0.1. The balance example uses 500 exposures in each group at cost rates 400 and 800 and predicted rate 600. The missing-label calculation uses total 1,000, recorded 700, events 35, and missing outcomes 300. The threshold example uses premium 900, expenses 160, capital charge 90, and expected cost 600 with a scenario interval from 550 to 750. These specifications are complete for the arithmetic shown. They do not emulate the stochastic dependence of an actual insurance portfolio.
For each empirical application an analyst should reproduce the evaluation metrics from fixed data snapshots, compare code with the mathematical definition and document uncertainty estimation. An error in target definition is generally more consequential than an additional decimal place of numerical accuracy. The scientific claim that the target is identified must precede the claim that its estimate is precise.
Conclusion
Underwriting model calibration is a distribution-specific property of predicted insurance outcomes. The distribution examined by an insurer ordinarily results from a sequence of eligibility, offer, customer response, contract exposure, claims reporting, and observation decisions. The model can participate in this sequence, creating an endogenous relationship between prediction and available validation evidence.
The selected portfolio can be a legitimate and well-defined target for actuarial evaluation. Its calibration should be measured on the intended probability, count, cost or predictive distribution, with transparent weighting, maturity, and contract definitions. Evaluating absolute cost requires more than a loss ratio or discrimination statistic. Global balance can hide subgroup deviations; empirically successful recalibration on one portfolio can misstate another portfolio's cost when risk mixtures change.
Transport to broader applicant populations requires explicit conditions connecting observed and unobserved outcomes. Covariate shift allows adjustment within supported regions under conditional invariance. Propensity weighting and doubly robust methods rely on measurable selection and positivity, among other assumptions. Where unobserved selection or unavailable risk classes defeat identification, honest bounds and sensitivity analysis provide more defensible results than assigned pseudo-outcomes. The economically relevant decision then combines predicted risk, terms, expected contribution, uncertainty, capacity, legal constraints and customer access.
Temporal testing, cross-carrier harmonization, additional-information audits and outcome-development monitoring extend the evidence base in different ways. They have distinct inferential limits. Their value is greatest when evaluated under frozen model and rule versions and when proxy labels are distinguished from realized costs. Professional and supervisory guidance reinforces the need for models appropriate to their intended purpose, documented judgment and accountable governance, subject to the applicable jurisdiction.
The original contribution is the integrated treatment of population definition, conditional calibration, contractual cost, selective observation, uncertainty and underwriting economics. These established analytical components jointly explain why a portfolio's successful observed experience does not, by itself, validate the full decision boundary that produced it. An institution's ability to justify that boundary depends on the outcomes it observes, the populations it can compare, the assumptions it can defend, and the limits it makes visible.
References
Actuarial Standards Board. (2005, revised May 2011). Risk classification (for all practice areas) (Actuarial Standard of Practice No. 12, Doc. 132; effective May 1, 2006). Source.
Actuarial Standards Board. (2019). Modeling (Actuarial Standard of Practice No. 56, Doc. 195; effective October 1, 2020). Source.
CASdatasets project. (n.d.). Frequency analysis of a French motor third party liability dataset [Software documentation]. Source.
Dudík, M., Langford, J., & Li, L. (2011). Doubly robust policy evaluation and learning. In Proceedings of the 28th International Conference on Machine Learning. Author manuscript, arXiv:1103.4601. Source.
Einav, L., Finkelstein, A., Ryan, S. P., Schrimpf, P., & Cullen, M. R. (2013). Selection on moral hazard in health insurance. American Economic Review, 103(1), 178–219. DOI: 10.1257/aer.103.1.178. Source.
European Insurance and Occupational Pensions Authority. (2025, August 6). Opinion on artificial intelligence governance and risk management. Source.
Finkelstein, A., & Poterba, J. (2002). Selection effects in the United Kingdom individual annuities market. The Economic Journal, 112(476), 28–50. DOI: 10.1111/1468-0297.0j672. Source.
International Association of Insurance Supervisors. (2024, December 5 revision). Insurance Core Principles and ComFrame: Online tool. Source.
Klein, A. (2024, May). SOA accelerated underwriting practices survey. Product Matters!, Society of Actuaries. Source.
Lakkaraju, H., Kleinberg, J., Leskovec, J., Ludwig, J., & Mullainathan, S. (2017). The selective labels problem: Evaluating algorithmic predictions in the presence of unobservables. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. DOI: 10.1145/3097983.3098066. Source.
National Association of Insurance Commissioners. (n.d.). Artificial intelligence (insurance topic; description of December 2023 model bulletin). Source.
New York State Department of Financial Services. (2024, July 11). Insurance Circular Letter No. 7: Use of artificial intelligence systems and external consumer data and information sources in insurance underwriting and pricing. Source.
Rosenbaum, P. R., & Rubin, D. B. (1983). The central role of the propensity score in observational studies for causal effects. Biometrika, 70(1), 41–55. DOI: 10.1093/biomet/70.1.41. Source.
Seeman, L., & Herzog, K. (2024, August). Accelerated underwriting: Mortality slippage study and monitoring best practices. Product Matters!, Society of Actuaries. Source.
Sugiyama, M., Krauledat, M., & Müller, K.-R. (2007). Covariate shift adaptation by importance weighted cross validation. Journal of Machine Learning Research, 8(35), 985–1005. Source.
Van Calster, B., McLernon, D. J., van Smeden, M., Wynants, L., & Steyerberg, E. W., on behalf of the STRATOS topic group. (2019). Calibration: The Achilles heel of predictive analytics. BMC Medicine, 17, 230. DOI: 10.1186/s12916-019-1466-7. Source.
Wüthrich, M. V., & Ziegel, J. (2024). Isotonic recalibration under a low signal-to-noise ratio. Scandinavian Actuarial Journal, 2024(3), 279–299. DOI: 10.1080/03461238.2023.2246743. Source.