RESEARCH DOSSIER
Algorithmic Fairness Measurement in Insurance Predictions
Abstract. Algorithmic fairness measurement in insurance investigates whether predictions and consequential decisions display systematic, statistically demonstrable differences among relevant populations, what those differences mean, and how confidently they can be interpreted. Its objects include event probabilities, claims frequency, conditional severity, expected insurance loss, risk classes, quotes, coverage decisions, claims screening, and intervention thresholds. The analysis must connect each measured disparity to a population, observation process, prediction target, protected-attribute definition, exposure unit, and insurance product. This monograph develops the formal foundations of group-specific calibration, error distributions, decision parity, proxy discrimination, intersectional auditing, small-sample inference, data selection, label bias, deterministic quote testing, and longitudinal monitoring. It establishes through transparent constructed calculations that global mean calibration can coexist with offsetting group miscalibration and that equal true-positive and false-positive rates need not imply equal predictive values when event prevalences differ. Research evidence comprises peer-reviewed actuarial and machine-learning studies, historical statistical sources, 2026 insurance-audit preprints, actuarial professional standards, and official international supervisory publications. The central interpretive finding is that fairness statistics acquire scientific meaning through the estimand and data-generating mechanism that support them. The text provides replicable audit specifications, contrasts evidential and regulatory meanings, and identifies limitations of statistical claims under selection, censoring and proxy-group misclassification. The monograph offers original analytical synthesis without proposing new named theories or reporting an original population-level empirical discovery.
Keywords: Algorithmic fairness; insurance predictions; statistical validation; actuarial classification; demographic parity; conditional parity; equalized odds; calibration; predictive parity; Brier score; underwriting; insurance pricing; proxy discrimination; protected attributes; intersectional audit; risk modeling; claims selection; missing labels; error uncertainty; regulatory supervision.
Introduction
Insurance prediction is an empirical claim about uncertain future events. A claim-probability score, a predicted loss amount, a mortality hazard, an expected lapse, or a fraud alert describes a different object. Each has its own time horizon, contractual boundaries, event-generating process, units and affected people. A fairness comparison is consequently meaningful only after the analyst establishes which prediction or decision it describes, which population was observed, which populations were omitted, and which dimensions of group difference the study is intended to measure.
Insurance scholarship supplies an appropriately specific foundation. Lindholm and colleagues (2022), in their discrimination-free insurance pricing analysis, examine the problem of direct and indirect discrimination in insurance prices. Xin and Huang (2024) connect insurance pricing fairness criteria to model construction and regulatory questions. The Society of Actuaries Research Institute (2026) discusses fairness metrics in life insurance. The research field now supports rigorous differentiation among statistical properties, contractual outcomes and legal norms.
This dossier concentrates on measurement. It investigates how a scientific claim that one group receives more false positive predictions, less accurate expected-loss estimates, more severe score-to-outcome deviations, or more unfavorable quotes can be established and tested. The normative desirability of particular insurance risk classes, the allocation of solidarity subsidies, and the interpretation of discrimination statutes are related subjects; their conclusions cannot be inferred mechanically from a metric.
The analysis proceeds from a precise data model to score properties, error-rate identities, intersecting groups, observation bias and audit interpretation. It uses mathematics and constructed examples to distinguish an observed gap from an estimated causal effect. Original regulatory sources are described by jurisdiction and document type. Peer-reviewed claims are differentiated from emerging preprint findings. The use of a source link marks evidence for the associated specific assertion and is supplemented by full attribution in References.
Protected and Proxy Attributes
The statistical unit and the research estimand
Let i denote an applicant, policy, insured life, or policy-period observation, and let t denote evaluation time. Write X for measured predictors, A for a protected or audit-relevant characteristic, E for exposure, Y for the observed loss or outcome, S for a model score and D for a decision influenced by S. For a binary claim event, Y takes values 0 and 1. For an amount, Y is nonnegative and may be heavy-tailed. Define P as the population in scope, H as the horizon, and O as the observation and reporting regime. The object actually estimated is conditional on (P,H,O), even when those conditions are omitted from informal model descriptions.
A person, policy, household and exposure-year are distinct statistical units. One person may hold several policies; one household may generate correlated risks; policies may have different terms and periods; one storm can cause correlated claims across the same territory. A policy-count denominator, person-count denominator and exposure-weighted denominator generally produce different group comparisons. The first responsibility of an audit is therefore to match the unit and denominator to the decision and prediction.
For binary risk, define m(x,a)=Pr(Y=1 given X=x,A=a,P,H,O). A deployed algorithm may compute S=f(X) without access to A, while an auditor estimates model behavior conditional on A. For positive annualized losses, expected loss per exposure is m_L(x,a)=E(Y/E given X=x,A=a,P,H,O), when E is positive and an annualization or rate ratio is appropriate. These conditional quantities are not automatically observable for applicants denied coverage, since their claims outcomes under the proposed policy were never generated or recorded.
Insurance pricing adds further layers. The quoted premium may depend on expected loss, expenses, capital charges, reinsurance costs, subsidies, discounts, taxes, legal caps and commercial strategy. A premium gap is a final-price statistic. A model loss gap is a score statistic. The analyst should document the calculation between them, especially if a retention or marketing algorithm modifies the price after an actuarial indication. Measuring fairness of the quoted premium using only the underlying expected-claims model can overlook the actual consumer-facing decision.
Group definitions and comparison baselines
The variable A requires an auditable definition. Depending on product and legal setting it may refer to age, sex, race or ethnicity, disability, location, or a relevant intersection of attributes. The scientific category is a measurement unit; legal protection and permissible processing are determined by jurisdiction. An audit should report whether A is self-reported, administrative, inferred, or missing, and whether an unknown category is excluded, redistributed or retained.
A relative disparity depends on the reference group. Comparing one group to the largest demographic category, the most favorable category, the whole remaining population, or another specifically authorized comparator yields different results. For a multi-group attribute, pairwise comparisons and distances from an agreed institutional benchmark can be more transparent than a selective single ratio. Every reported metric requires an explicit direction: whether a positive value indicates more claims, more adverse flags, more insurance approvals, or a higher price.
A further distinction concerns individual and group measures. Comparing similar-risk applicants is a conditional exercise; evaluating the aggregate proportion of approvals is an unconditional exercise. Conditional analysis must state which factors constitute legitimate risk adjustment. Conditioning on a variable affected by historic exclusion can conceal a disparity; omitting a lawful risk predictor can make an actuarially explicable difference appear anomalous. This tension is part of the target definition and is not solved by adding every available covariate.
Protected information in model operation and evaluation
The absence of A in deployed model code does not prove that predictions are independent of A. Distributions of X may differ by A, and X can carry information about protected status. Conversely, an authorized evaluation may need protected-group data even when operational pricing is barred from using it. Lindholm and colleagues (2022) discuss this tension within their analysis of protected characteristics.
A scientifically defensible implementation separates operational data privileges from audit data privileges, with the permitted basis determined by law, contract and privacy governance. For example, an audit team may analyze aggregated group error without giving a live pricing engine person-level group attributes. Whether any such arrangement is legally permissible must be established for the jurisdiction and use case. The mathematical point is that evaluating conditional behavior requires a scientifically adequate group measurement or an appropriately qualified proxy.
Proxy dependence and feature provenance
A proxy feature Z can encode information about A through geography, occupation, household composition, credit activity, health-data availability or another history of unequal exposure. A raw correlation between A and Z provides a screening signal. It does not establish that Z has a direct causal effect, that its use is prohibited, or that removing it will eliminate group disparity. More informative analysis compares models under justified feature sets, documents the origin and function of Z, and tests whether the resulting outcome gap survives clinically or actuarially legitimate adjustments.
Lindholm and colleagues (2024) establish the distinction between proxy discrimination and group disparity. Their results show why avoiding one type of proxy dependence is not equivalent to satisfying every group-parity criterion. A group parity measure examines a distribution of outputs or decisions; a proxy inquiry examines how sensitive information reaches and influences the system. An audit that treats the two as synonyms mistakes different properties for one another.
Predicting A from X can measure leakage, but model recoverability of A need not be the same as its actual causal contribution to pricing. Feature attribution depends on interaction and correlation assumptions. A protected characteristic may be poorly predicted in the population as a whole while interactions with a small set of factors produce a locally important adverse effect. A complete audit combines data provenance, sensitivity, conditional comparisons and institutional interpretation.
Inferred group attributes and misclassification
Where direct labels are unavailable, analysts may reconstruct group identity using names, geographical information, or probabilistic models. Let A be actual membership and A* the inferred category. The matrix M_ab=Pr(A*=b given A=a) describes a classification mechanism. Estimating a disparity conditional on A* generally measures a mixture of underlying groups, and naive standard errors omit the uncertainty induced by the reconstruction.
As a constructed illustration, assume two groups have equal population shares and an outcome mean difference δ=100 units. Assume independent, symmetric misclassification with probability e=0.20 in each direction. Then the observed difference between groups classified by A* is (1-2e)δ=60. This exact attenuation depends on equal priors and symmetric nondifferential error. It is not a universal correction for inferred status in insurance data.
Xin, Hooker, and Huang (2026) analyze a more realistic setting in their proxy-race audit preprint. They report that regression estimates can be attenuated or amplified when inference errors vary with geographical and socioeconomic factors. Their study is an emerging preprint; the immediate methodological lesson is that an inferred group label is a fallible measurement. Auditors should validate its confusion matrix, report uncertainty and investigate whether misclassification is related to pricing or outcome residuals.
When probabilities of category membership are used, their weights should be checked for calibration and the statistical procedure should propagate the underlying uncertainty. Sensitivity analyses can describe the effect of plausible misclassification matrices on the reported disparity. An impressive point estimate based on uncertain labels should never be presented with the precision of a directly observed protected attribute.
Group-Specific Prediction Metrics
Distinguishing risk scores from decisions
For binary Y define group prevalence π_a=Pr(Y=1 given A=a) and decision rate q_a=Pr(D=1 given A=a). The meaning of D=1 has to be fixed. It might signify a high-risk alert, a denial, a quotation issued, approval of insurance, manual claims review or fraud investigation. Each decision has different consequences and different interpretations of higher rates. A risk alert is often adverse for the policyholder; an approval is favorable.
Unconditional decision parity evaluates whether q_a is equal across groups. Useful descriptive measures are the difference q_a-q_b, the ratio q_a/q_b when the denominator is positive, and standardized comparisons using an agreed reference distribution. These quantify distributions under the observed underwriting and data collection process. They do not demonstrate a causal effect of A or a breach of law without additional institutional and evidential work.
An audit may instead compare decisions conditional on legitimate factors L. Then its estimand is Pr(D=1 given A=a,L=l) relative to another group at the same l. Conditional comparison may reduce differences due to lawful and genuinely predictive risk, yet can also control away mechanisms related to discrimination if the chosen variables are themselves consequences of historical exclusion. Which L values are appropriate is a research and governance decision.
Error rates and threshold effects
Define TPR_a=Pr(D=1 given Y=1,A=a) and FPR_a=Pr(D=1 given Y=0,A=a). False-negative rate is 1-TPR; true-negative rate is 1-FPR. Equal opportunity requires equality of TPR for the relevant groups, and equalized odds requires equality of TPR and FPR. Hardt, Price, and Srebro (2016) formalize equal opportunity in supervised learning.
Threshold-based error rates reflect the particular operating threshold, even if continuous score ranking is identical. An insurer that varies referral or rejection thresholds by business channel can change the measured FPR and TPR without retraining the probability model. A threshold audit must therefore record threshold versions, decision overrides and whether adverse outcomes are caused by the score or by other business rules.
False-positive rate, false-negative rate, precision and recall measure different properties. A fraud model with many alerts in a low-prevalence population may have a high true-positive rate but many false accusations. A missed high-risk claim may matter for insurer claims adequacy while a spurious adverse underwriting flag may matter for consumer access. The relative harm of these errors belongs to the decision context and cannot be recovered from a single AUC.
Predictive values and comparative discrimination
Positive predictive value is PPV_a=Pr(Y=1 given D=1,A=a). Negative predictive value is NPV_a=Pr(Y=0 given D=0,A=a). A model can have equal TPR and FPR across groups and different predictive values if base event prevalences differ. Reporting PPV therefore supplies information unavailable in error-rate equality statements.
AUC evaluates whether a randomly selected positive example tends to receive a higher score than a negative example within the relevant group. It does not identify calibration at a probability value or the effect of the deployed threshold. Aggregate AUC can be high while an important subgroup has low sensitivity. Conversely, group-specific AUC differences may result from changes in outcome severity, label selection or covariate distributions, not only from algorithmic bias.
Comparison should include absolute event rates and sample counts. An odds ratio may sound extreme when the reference event is rare, while a percentage-point gap may describe a large number of affected applications. Scientific interpretation requires absolute and relative effects, uncertainty, and a decision-relevant scale. Neither an arbitrary ratio threshold nor an unadjusted significance test is a universal scientific definition of insurance fairness.
Calibration of probabilities
For a probability score S, group calibration means Pr(Y=1 given S=s,A=a)=s at relevant score levels. In data, calibration is estimated through bins, smoothing, or other conditional procedures. A group can be calibrated on average and substantially miscalibrated in a narrow high-risk tail. Sparse support creates unstable local estimates. Meaningful reports include group-specific score distributions, event counts and interval estimates.
Define calibration-in-the-large C_a=E(Y-S given A=a). The condition C_a=0 is one expectation identity; it is weaker than calibration at each predicted score. The observed-to-expected count ratio is another summary, subject to difficulties when expected events are scarce. In count or pure-premium problems the same principle applies after fixing consistent exposure and loss units.
Guo and colleagues (2017) study modern neural-network calibration and temperature scaling on image and document datasets. Their results support examining recalibration of classification scores, but they do not certify any method on insurance severity, delayed claims, censored survival or changing policyholder populations. Recalibration fitted on old experience needs independent evaluation under the intended insurance horizon.
Proper scores and loss-model validation
The squared-probability or Brier score is BS_a=E[(Y-S)^2 given A=a], following the probability forecast verification literature associated with Brier (1950). It evaluates a probabilistic forecast under the outcome distribution. Its group comparison mixes properties of the model and the difficulty of the group's prediction problem. A lower Brier score by itself is not proof of fairer treatment.
Logarithmic score is another proper scoring rule for probabilities in the relevant interior domain. It heavily penalizes very confident errors, so reports should specify clipping and numeric handling of near-zero scores. For count, severity and pure-premium models, validation should use distribution-appropriate residuals, predictive deviance, interval coverage and expected-versus-observed losses. A single score inappropriate for the response family creates artificial group discrepancies.
A severity model can predict means accurately while missing the upper tail of one group's claim distribution. A frequency model can fit total claim counts while underestimating low-exposure policyholders. A mortality model can appear accurate among fully observed lives and perform poorly after censoring is considered. These are distinct scientific tests requiring different error constructions and observation rules.
Exposure and aggregation
For losses Y_i and expected losses μ_i at compatible units, group O/E may be computed as ΣY_i/Σμ_i. If μ_i is annualized per unit of exposure, its expected aggregate component must be multiplied by E_i before summation. If μ_i already represents total expected cost over each policy term, multiplying by E_i again double-weights exposure and changes the estimand.
Record-weighted, person-weighted and exposure-weighted group averages can disagree. Suppose one group has numerous short-duration policies and another has fewer annual contracts. Policy counts overweight the first group relative to annualized risk exposure. A valid fairness claim states which average matters for the product or decision and makes alternative denominators available for sensitivity analysis.
Insurance losses are often dependent. An earthquake, wildfire, cyclone, pandemic, or litigation wave may generate correlated events. Standard errors based on independent records would be too narrow if a single catastrophe affects many claims. Clustering by geographic event, policyholder, employer or other shared exposure is therefore part of statistical fairness inference, not an optional afterthought.
Calibration and Error Tradeoffs
A mathematical identity behind incompatible parity criteria
Let π_a be group event prevalence, t_a true-positive rate and f_a false-positive rate of a fixed binary alert. Then q_a=t_aπ_a+f_a(1-π_a). When q_a>0, PPV_a=t_aπ_a/q_a. These are direct consequences of total probability and Bayes' rule. If two groups have different π and share the same t and f, their PPV values generally differ. Equalized odds can thus hold without predictive parity.
Take hypothetical groups with π_A=0.20 and π_B=0.10. Suppose both have t=0.80 and f=0.20. Then q_A=0.32, q_B=0.26, PPV_A=0.50 and PPV_B=0.08/0.26≈0.3077. The alert marks a high-risk or adverse event and is not an approval. The calculation proves a logical coexistence of equal error rates and unequal positive predictive values in this constructed example.
Chouldechova (2017) studies conflicting group fairness conditions in recidivism assessment, while Kleinberg, Mullainathan, and Raghavan (2016) analyze trade-offs in risk scores. Their incompatibility results concern particular mathematical definitions and exclude special cases. They do not establish a universal impossibility of fairness. Insurance transfer requires explicit recognition of the event being predicted, score use, base rate and product-specific decision.
Demanding unconditional parity, equalized odds, perfect calibration and predictive parity simultaneously can therefore lead to contradictory model specifications. An insurer may require accurate expected claims for reserving and a separate review rule with limits on adverse errors. Measuring those properties individually permits a meaningful trade-off analysis without changing the meaning of a loss prediction to satisfy an unrelated parity statistic.
Constructed portfolio calibration counterexample
Consider two hypothetical groups of 200 independent policy periods each. Assign all observations in A the forecast probability S=0.16, and all in B the forecast probability S=0.14. Let A record 40 events and B record 20. Observed group rates are 0.20 and 0.10. Predicted events total 32 in A and 28 in B. The whole portfolio has 60 predicted events and 60 observed events.
Aggregate calibration-in-the-large is exactly zero: mean forecast 0.15 and mean observed rate 0.15. Group residual means are +0.04 and -0.04. Their contrast is 0.08. The aggregate result therefore conceals systematic group deviations of opposite signs. The example isolates the aggregation property and makes no assertion about a real insurer or legal discrimination.
For constant group scores, BS=π(1-p)^2+(1-π)p^2. In A, 0.20×0.84²+0.80×0.16²=0.1616. In B, 0.10×0.86²+0.90×0.14²=0.0916. Their equally weighted portfolio mean is 0.1266. Different Brier scores reflect different event prevalence as well as the fitted risk estimates and cannot be interpreted as pure measures of group harm.
Treating the two groups' outcomes as independent Bernoulli observations with fixed forecast probabilities, the estimated standard error of the residual contrast is sqrt(0.20×0.80/200+0.10×0.90/200)=0.03536. A normal approximate 95% interval centered at 0.08 runs from approximately 0.0107 to 0.1493. This interval ignores model-estimation error, dependence, group reconstruction, selection and temporal variation. It is a transparent mathematical illustration of one inference procedure rather than an insurance audit result.
Repair, trade-offs and protected-data constraints
A model may be recalibrated on a common population, recalibrated separately by group, retrained with better features, or supplemented by a distinct policy decision rule. These actions have different legal and statistical implications. A model recalibrated by protected status may improve group error but create new concerns about direct attribute use. A decision rule that adjusts acceptance thresholds can alter access without changing underlying expected-loss estimates.
Fairness remediation should therefore retain distinct ledgers for predicted risk, quoted price, coverage decision and subsequent outcome. If the problem is a claim-reporting artifact, recalibrating the model may merely learn a biased observation pattern more precisely. If the problem is a vendor proxy, post-hoc output parity may conceal continued use of objectionable information. The diagnosis determines which intervention can address the cause.
An empirical remedy is supported when independent data show improvement in the pre-specified metric, with unchanged definitions of exposure and outcome and uncertainty calculated under the actual sampling design. A favorable training-set fairness score is weak evidence of out-of-sample improvement. Changes to loss adequacy, risk selection, consumer consequences and legal restrictions should be documented alongside the targeted metric.
Intersectional and Small-Sample Tests
The significance of overlapping groups
A population-wide comparison may conceal disparities concentrated among groups defined by several interacting attributes. An aggregate fairness statement, such as equal mean predicted loss for two broad demographic groups, can coexist with unequal predictions for a particular age range, underwriting channel or geographical subgroup. The research on fairness gerrymandering across subgroups by Kearns and colleagues (2018) supplies the statistical foundation: satisfying selected constraints at an aggregate level does not establish the same constraints for every meaningful subgroup.
The practical insurance manifestation is especially important because predictive models commonly include interactions. An occupation may influence claims differently across types of insurance; a neighborhood measure may behave differently across vehicle use; a health feature may have a different relationship with mortality at different durations. Fairness analysis must therefore distinguish a property conditional on a broad protected category from the corresponding property for intersections. A result of zero mean group residual does not imply zero residual for each subgroup contained within it.
Hébert-Johnson and colleagues (2018) develop multicalibration across overlapping populations. Their work offers a mathematical response to the limitations of a small number of coarse calibration checks. The relevant property is approximate calibration across a specified, computationally identifiable class of subpopulations. Such calibration does not automatically guarantee decision parity, lack of proxy dependence, or legal compliance. Its usefulness depends on the richness of the class, sample support and the reliability of the measured outcome.
A scientifically responsible intersectional program identifies the protected and audit-relevant dimensions whose combinations have a defensible interpretation. It then states whether comparisons are confirmatory or exploratory. The audit should record its population, denominators, product mix, group category construction, threshold choices and meaningful disparity tolerance. Selecting a subgroup only because its measured gap is extreme can produce an optimistic impression of the underlying signal.
Reliability with few events
Suppose five adverse events occur among twenty independent policy periods in one hypothetical group. The estimated event probability is 0.25. The usual plug-in binomial standard error is sqrt(0.25×0.75/20), approximately 0.0968. The size of this uncertainty is substantial relative to the estimate itself. With sparse observations, conventional normal intervals can perform poorly; Wilson or exact binomial methods may be more appropriate for proportions. A statistical procedure remains conditional on the assumption of independent, comparable outcomes.
Insurance observations frequently create greater uncertainty than a binomial illustration suggests. A cluster of claims may arise from one storm, a business may insure multiple vehicles under a shared fleet exposure, and several apparent claims may arise from one underlying incident. For large severity risks, a small number of extreme claims can dominate total loss. Effective independent information can be much smaller than the nominal count of rows in a database.
A group may also have apparently abundant policy records and very few informative observations for the specific outcome being evaluated. One thousand life policies with one observed death provide quite different evidence from one thousand independent deaths. A fairness report should identify total records, exposure, number of relevant events, and observation maturity. It should provide uncertainty intervals even when a point estimate appears to meet a selected tolerance.
Hierarchical modeling and partial pooling can stabilize subgroup estimates by borrowing information from related strata. These methods impose assumptions about how different groups are related. Excessive pooling may smooth away a genuinely unusual subgroup. For that reason, an audit should preserve raw group summaries alongside pooled estimates and show the sensitivity of conclusions to the pooling structure. A statistically smoother estimate is not itself proof of equality.
Multiple comparisons and selective discovery
With many outcomes, groups, thresholds and time periods, the analyst can calculate hundreds or thousands of comparisons. A collection of hypothesis tests at a conventional nominal level has a nontrivial opportunity to report chance findings. Family-wise error procedures, false-discovery-rate methods and hierarchical modeling can address different inference goals, but the choice must match the design and assumptions. Descriptive monitoring may justifiably report raw intervals without turning every interval into a hypothesis test.
Confirmatory contrasts should be defined before inspecting group outcomes when possible. Exploratory searches should disclose how many configurations were examined, their selection rules, and the reasons for pursuing a finding. A result originally selected from an exploratory search should be evaluated on a later or independent sample before being described as replicated.
An audit of an evolving portfolio also faces temporal multiplicity. Weekly fluctuations can trigger a sequence of apparent breaches even when the underlying process is stable. A monitoring framework can distinguish a persistent pattern from noise by specifying intervals, alert thresholds, a sequential testing convention and a documented follow-up procedure. The appropriate monitoring cadence depends on the volume and maturation of relevant insurance outcomes.
Weighting, the reference population, and Simpson-type reversals
Insurance measures often change after aggregation because products and risk exposures have different weights. For example, suppose a model has similar conditional error within comparable risk bands across two groups, yet one group has more high-risk policyholders. The unadjusted mean residual, alert rate or price may then differ. Conversely, parity in the pooled portfolio can conceal persistent conditional disparities with opposite weights.
Before declaring a contradiction among results, the audit should examine their estimands. Person-weighted mean price, policy-weighted mean price, exposure-weighted mean loss and premium-weighted claim ratio are distinct quantities. A company may legitimately analyze all four, provided the question answered by each is clear. A change of weights can reverse which group appears to have a larger numerical gap without changing any individual's prediction.
The defensible report includes observed and standardized comparisons. Standardization may use a common age or risk distribution when that distribution is scientifically and legally appropriate. Its reference population and strata must be documented, as standardization is an explicit choice about the target comparison rather than a mechanical correction.
Consequences of low statistical power
An inconclusive small-sample test provides evidence of uncertainty. It does not demonstrate parity. A wide interval may include both a minor difference and a potentially substantial disadvantage. If the group is consequential and the uncertainty material, the justified response may include targeted data-quality work, longer observation, a larger audit frame or appropriately authorized additional measurement.
Reporting should distinguish a null-hypothesis test, a confidence interval, an equivalence test and a substantive tolerance. Equivalence requires an explicit acceptable margin and sufficient precision to exclude disparities beyond that margin. The statement that “no significant difference was found” supports a weaker conclusion than “an equivalence margin was met.” Such distinctions are fundamental to any insurance audit being used to support an institutional decision.
Feature and Outcome Bias
Observed insurance losses arise through selection
Historical loss data are generated through a sequence of decisions. An applicant seeks insurance; underwriting determines eligibility and terms; a price is offered; the applicant may purchase; exposure occurs; an event may happen; a claim may be reported, adjusted and settled. These stages jointly determine whether a label appears in the dataset. The observed variable Y is thus not simply a transparent measurement of risk independent of insurance operations.
Let R=1 indicate that the outcome is observed. A model trained on recorded policies may estimate E(Y given X,A,R=1) when the scientific or commercial target is E(Y given X,A) for all relevant applicants. These differ whenever selection R conveys information about outcome risk after the measured features are considered. A model may be well calibrated among purchasers while systematically overstating risk among applicants who declined an unaffordable quote; the latter population contributes no subsequent policy loss experience.
Under appropriately specified missing-at-random assumptions, a response observation probability Pr(R=1 given X,A) and suitable positivity can support inverse-probability-weighted estimates. Those assumptions need a substantive account of how missing labels arise. When selection depends on the unobserved outcome or unmeasured causes of that outcome, observational data may establish only bounds or sensitivity results. Adding predictors to a flexible model cannot guarantee identification of an otherwise unobserved counterfactual.
A predictive fairness claim covering “all applicants” therefore requires more than test-set accuracy among policyholders. The sampling frame, missing outcomes, rejected applicants and voluntarily uninsured consumers should be identified. This condition has particular importance in insurance access studies because the model and price may influence who enters the portfolio in the first place.
Label construction and claims maturation
Insurance labels can differ between groups because of administrative practice. Claim reporting may depend on awareness, access to documentation, perceived claim value and expected treatment. An insured who faces more obstacles in reporting a valid loss may have fewer recorded claims even with equivalent underlying loss experience. A predictor trained on those records may learn differences in reported behavior rather than differences in loss incidence.
Claims are also observed at different development ages. A recent claim may be open, partially paid or subject to litigation. Comparing paid loss for two groups without harmonizing development may confuse delayed settlement with underlying claim severity. Insurance fairness validation should state the valuation date, development adjustment, inflation and coverage basis, and whether claims outcomes are paid, incurred, ultimate or thresholded.
Fraud models create a further feedback loop. A confirmed fraud outcome may be available only after investigation, while the model helps determine which claims are investigated. Uninvestigated claims cannot automatically be relabeled as verified nonfraud. Doing so creates selective labels and can make the historical investigation policy appear accurate because the system determines what gets verified. Randomly sampled quality review, subject-matter adjudication and explicit selection analysis may improve knowledge, subject to legal safeguards.
The Actuarial Standard of Practice on Data Quality establishes professional expectations for assessing the suitability, limitations and use of data within the standard's applicable scope. Its relevance to fairness work lies in forcing the analyst to explain data provenance, inconsistencies, selection and the effect of limitations. It does not supply a universal formula that converts deficient group labels into valid group comparisons.
Feature measurement and imputation
Inputs used in insurance models include reported driving activity, location, telematics, income-related commercial data, medical histories, property condition, claims history, third-party scores and inferred categories. These inputs can exhibit systematic gaps. Missing records may be coded differently across distribution channels; geographic coverage may depend on the vendor; the meaning of a score can change between product versions.
A common failure is to assign a numerical placeholder that the model interprets as a real observation. For example, missing annual mileage coded as zero can lead to an artificial low-risk classification. Missing health assessments may correlate with healthcare access rather than health status. Group-dependent imputation can create residual differences even when protected status is omitted from the deployed model.
A fairness audit should report each consequential feature's provenance, missingness, imputation logic, units, plausible range and drift. Group-specific missingness and outlier rates are useful screening measures. An analyst should also distinguish a feature that is accurately measured but normatively controversial from one that is factually unreliable. Those are separate reasons to question its use.
The Actuarial Modeling standard addresses model selection, intended purpose, validation, use and communication in applicable actuarial practice. In the fairness context, it supports keeping model version, feature transformations, data cutoffs, validation criteria and known limitations traceable. The standard is a professional practice instrument within its applicable jurisdiction and circumstances; it should not be represented as an international statutory fairness code.
Time variation, transfer and recalibration
Group disparities may change when risk distributions, insurance products, data vendors or loss experience change. A model may remain fixed while the applicant mix shifts. An aggregate prediction score may appear stable while particular groups experience deteriorating calibration. A group with greater exposure to a severe event year may have large observed loss changes unrelated to a permanent difference in underlying risk.
Temporal assessment distinguishes covariate shift, shifts in outcome conditional on features, changed reporting and threshold revision. These should be tested separately where the data allow. Recalibration using the same observations on which final performance is reported creates optimistic estimates. A forward-time validation design tests whether an updated calibration or decision rule generalizes to the future insurance population.
Portability also has limits across countries and products. A mortality-risk outcome in a life portfolio, paid property-loss amount, health utilization measure and claims fraud adjudication indicator require different statistical assumptions. The mathematical definitions of calibration and error can be portable, while the underlying event, permissible risk classification and data-generating process differ materially.
Conditional association and causal interpretation
Regression adjustment can measure a residual association between group membership and predicted outcomes after selected covariates are controlled. That association is not automatically a causal effect. Appropriate conditioning depends on the chronology and causal roles of those covariates. If price influences purchase, adjusting for the purchased-only portfolio can change the association between group membership and risk even when the same underwriting score is applied.
A causal graph can describe the assumptions: group attributes and background conditions affect measured features; features and model results influence offered price; price affects purchase and exposure; claims outcomes are observed for purchasers who report. In that graph, selection into the insured sample may act as a collider and induce association not present in the applicant population. A descriptive group residual and a counterfactual effect of the pricing algorithm therefore have different identification requirements.
An insurer seeking to diagnose discrimination may need a well-defined counterfactual intervention, such as changing one suspect feature in matched applicants or comparing quotes before and after a rule revision. Such designs demand attention to which inputs can coherently be intervened upon and which other variables would respond. A mechanical one-feature perturbation is useful for sensitivity analysis but is not, without a causal account, proof of the effect of social identity.
Deterministic prices and design-based variance
Insurance quotation engines often provide a deterministic price for a fixed set of inputs. Suppose an auditor samples consumer profiles and queries an insurer's engine. Repeatedly submitting an identical profile may reproduce an identical quote. The statistical uncertainty then lies in how profiles are sampled, in estimated counterfactual costs or controls, and in dependence among compared profiles; it need not arise from an independent random premium-setting disturbance.
Huang and Hooker (2026) highlight this issue in fairness testing for algorithmic pricing. Their preprint argues that naive residual-based regression standard errors can be inappropriate for deterministic quote designs, derives corrected variance expressions and reports an empirical Illinois auto-insurance quotation analysis. The authors' empirical findings should be treated as preliminary until independently replicated; the underlying statistical warning is directly relevant to study design.
A paired-quote experiment should specify sampled applicant profiles, which risk covariates remain fixed, which features are perturbed, the time of quotation and whether the change produces a legally and actuarially coherent comparison. An observational premium regression on existing customers measures a different distribution from a controlled matched-profile exercise. Both may be useful, but their causal interpretation and variance estimators differ.
If many profiles share the same ZIP code, household, insurer or model version, clustering can materially change estimated standard errors. The evaluation unit and the stochastic sampling mechanism should be established before inferential software is selected. The automatic output of a regression command is no substitute for documenting how randomness and dependence enter the research design.
Fairness Audit Interpretation
Establishing the exact scope of a claim
A valid fairness finding identifies its prediction target, insured population, observation horizon, group definitions, metric, reference group, exposure denominator and model or decision version. It reports an effect estimate, an interval or other uncertainty characterization, and an explanation of the relevant data-generating assumptions. It should also state whether it concerns observed associations, risk-adjusted comparisons or an identified causal contrast.
The phrase “the system is fair” is too broad to function as the conclusion of a single statistical test. A more precise conclusion states that a particular group-specific calibration error was estimated below a pre-specified tolerance within a defined portfolio and period, or that an adverse threshold error difference exceeded the chosen evidentiary criterion. Such statements are replicable and remain open to revision if the target population or outcomes change.
The distinction between statistical evidence and normative judgment is especially important when a disparity concerns a genuinely risk-relevant feature. An observed group gap in projected claim costs may reflect unequal exposure, reporting or risk; a gap in final quote may additionally reflect pricing rules. Measurement establishes which difference occurs. A legal or ethical determination requires the applicable obligations and the consequences of the practice.
United States insurance oversight
The National Association of Insurance Commissioners approved an insurer AI model bulletin in December 2023. An NAIC model bulletin is a policy template and supervisory instrument used by member jurisdictions; it does not establish that every state has enacted an identical rule. A United States fairness study must establish the state, line of business, document status and adoption or enforcement context before assigning legal consequences to a measured result.
New York State's 2024 insurance AI circular letter provides a concrete regulatory discussion of underwriting and pricing. It addresses actuarial validity, unfair discrimination, use of external consumer data, quantitative adverse-impact assessments, governance, monitoring and less discriminatory alternatives. Its regulatory expectations can motivate several complementary statistical tests; their legal interpretation remains connected to New York's regulatory powers and the letter's stated scope.
A regulator may consider descriptive distributions, risk-adjusted mean price differences, odds ratios, adverse-decision rates and proxy effects as complementary evidence. Each test requires its own denominator and control variables. No universal adverse-impact threshold can substitute for the substantive legal context and the factual character of the insurance decision. A purported international conclusion drawn solely from New York guidance would exceed the primary source.
International supervisory comparisons
The International Association of Insurance Supervisors presented its AI supervision application paper in July 2025. It discusses insurance AI through supervisory themes and Insurance Core Principles, including governance, robustness, transparency and fairness. Its application-paper status must be distinguished from national legislation. It provides a comparative supervisory framework rather than a single globally enforceable statistical definition.
The European Insurance and Occupational Pensions Authority issued its 2025 insurance AI governance opinion. The opinion addresses sectoral supervision under relevant EU insurance law and distinguishes its scope from AI systems governed by specific AI Act high-risk or prohibited-system provisions. A fairness comparison involving EU insurance should therefore identify which regime and which date govern the particular product and system before interpreting an audit as a compliance test.
The National Institute of Standards and Technology's AI Risk Management Framework 1.0 offers a voluntary, cross-sector risk-management structure. It is useful for documenting risk context, measurement, management and governance, but the voluntary framework does not itself decide the legality of an underwriting practice. Its contribution to insurance research is methodological rather than a new insurance regulatory mandate.
Across jurisdictions, mathematical quantities such as TPR, FPR and calibration can share formal definitions. The choice of protected categories, permitted use of those categories, permissible risk factors, remedies and institutional accountability may diverge. A comparative study can therefore report consistent mathematical estimands while explicitly separating the legal interpretation in each country.
An implementable research audit protocol
A reproducible audit begins by documenting the product, portfolio, covered population, decision and prediction horizon. It identifies outcome definitions and event maturities, recording whether claims are reported, paid, incurred, ultimate, adjudicated or censored. It then locks the observation unit, exposure denominator, group attribute measurement and reference comparison before selecting performance metrics.
The next step specifies which models, features, quotes and decisions are being compared. Versioned source data, transformations, missingness rules, vendor contributions, training periods and applied thresholds must be retained. Group membership inferred from other data requires its own validation and uncertainty analysis. Where applicants outside the insured portfolio lack labels, the audit states the consequent identification limits.
Metrics should be chosen with their units and purpose explicitly linked to the insurance question. Classification audits may report TPR, FPR, PPV, AUC, calibration plots and decision rates. Frequency and severity models require observed-to-expected experience, appropriate deviance or residual diagnostics, tail behavior and coverage. Each report should include overall and group-specific estimates, major intersections, event counts, exposure and sampling uncertainty.
For primary comparisons, the audit specifies a material disparity tolerance, inferential procedure, relevant multiple-testing treatment and a method for preserving statistical independence between training and validation. It records the effects of alternative reference groups, weighting schemes and model versions. It tests whether findings persist across future periods, product cohorts or other defensible validation samples.
A final stage interprets discrepancies with domain experts. Potential mechanisms include an omitted legitimate risk feature, missing or delayed claims, an inappropriate exposure denominator, imputed protected characteristics, an unstable small group, a pricing override, or genuine disparate outcomes requiring further review. Each alternative can generate a specific follow-up test. The report should record the evidence for the explanation selected and any uncertainty that remains material.
A complete audit archive includes a machine-readable variable dictionary, code or mathematical definitions, source and model version identifiers, expected and observed counts, all planned comparisons, estimator choices, intervals, sensitivity results, important institutional decisions and reproducible citations. The scientific result is the bounded, supported conclusion rather than an unqualified certification of the entire algorithm.
Reinterpreting the constructed examples
The two-group calibration illustration above has perfect portfolio-wide mean calibration and opposing group residuals. It demonstrates that averaging can hide conditional error. A real audit would first examine claim development, exposure, product mix and group outcome measurement, then test the finding on independent experience. Adjusting premiums immediately on the basis of the constructed 0.08 contrast would introduce an intervention unsupported by the hypothetical data.
The separate equalized-odds example shows two different positive predictive values under equal TPR and FPR. It demonstrates the importance of base prevalence and metric selection; it cannot identify whether a particular insurance alert is institutionally desirable. Its interpretation depends on false-positive and false-negative consequences and the actual function of the decision.
The proxy-label illustration shows how a known classification error can attenuate a difference under narrowly specified assumptions. The recent regression audit distortion study explains why more realistic inferred labels may also amplify an effect. The responsible inference is to model and disclose label uncertainty rather than treat reconstructed protected status as observed truth.
Finally, the deterministic quote example establishes that inference depends on what has actually been sampled or randomized. An estimate of mean premium difference across sampled profiles and an estimate of adverse claim-event prediction error are separate research objects with separate sources of variance. Their numerical outputs cannot be compared or combined until the units, groups and stochastic designs are reconciled.
Integrative research findings
The literature and mathematical analysis support four related conclusions. First, group disparities are indexed by clearly specified estimands and observation processes. Second, the familiar fairness criteria answer different mathematical questions and selected criteria can be incompatible under unequal event prevalences. Third, outcome bias, proxy-label errors, exposure choice and small-cell uncertainty may dominate the apparent precision of a model-level comparison. Fourth, the institutional significance of the result is determined by the insured decision, insurance contract and applicable legal framework.
This synthesis has methodological consequences. A single aggregate “fairness score” loses necessary information about accuracy, access, proxy dependence and consequential decisions. A defensible research record instead makes it possible to inspect the precise group comparison, evidence and uncertainty associated with each claim. It treats a significant association as a statistical result and asks additional questions before proposing a substantive remedy.
The original contribution of this dossier is the explicit integration of these conditions into a coherent account of insurance prediction fairness measurement. It offers a structured analytic synthesis, mathematical demonstrations and operational research specifications. It claims neither a newly discovered empirical relationship nor an additional named theory, principle or doctrine.
Conclusion
Algorithmic fairness measurement in insurance predictions is the study of how models, labels and decisions behave across defined populations under specified statistical and institutional conditions. The principal quantities include group-conditioned calibration, errors, predictive values, exposure-normalized loss discrepancies, decision rates, proxy effects and uncertainty. Their values can be reproduced and interpreted only when the insured population, outcome definition, observation horizon, comparator and model version are known.
The formal derivations establish that error parity and predictive-value parity need not coincide across populations with different event prevalences. The constructed loss-forecast example establishes that pooled mean calibration can conceal opposite group errors. The small-sample, selection and proxy examples show how apparently simple disparities can be uncertain or measure a different population from the one the auditor intends to describe. These results follow from established probability and statistical principles applied to insurance settings.
Insurance data create a particularly demanding measurement environment through underwriting selection, buyer response to price, varying coverage terms, exposure differences, delayed claims, correlated catastrophes and claim-investigation feedback. Those mechanisms determine what observed labels mean. Valid performance tests therefore examine data provenance and observation alongside predictive accuracy. Recent 2026 preprints offer additional research directions on deterministic pricing variance and inferred protected-group status; their substantive empirical findings remain subject to independent verification.
The governing scientific conclusion is that measurement quality is established by the traceable relationship between the claimed disparity and its estimand, data, method, assumptions and consequences. A responsible audit maintains multiple complementary metrics, states the meaning of each comparison, quantifies uncertainty, tests robustness and describes regulatory context accurately. The monograph supplies a complete conceptual, formal and methodological treatment of that task. It reports no original insurer dataset analysis, no external peer review and no public publication.
References
Hardt, Moritz, Eric Price, and Nati Srebro (2016). “Equality of Opportunity in Supervised Learning.” Advances in Neural Information Processing Systems 29. Equality of Opportunity in Supervised Learning.
Chouldechova, Alexandra (2017). “Fair Prediction with Disparate Impact: A Study of Bias in Recidivism Prediction Instruments.” Big Data 5(2): 153–163; DOI 10.1089/big.2016.0047. Fair Prediction with Disparate Impact: A Study of Bias in Recidivism Prediction Instruments.
Kleinberg, Jon, Sendhil Mullainathan, and Manish Raghavan (2016). “Inherent Trade-Offs in the Fair Determination of Risk Scores.” arXiv:1609.05807, preprint. Inherent Trade-Offs in the Fair Determination of Risk Scores.
Lindholm, Mathias, Ronald Richman, Andreas Tsanakas, and Mario V. Wüthrich (2022). “Discrimination-Free Insurance Pricing.” ASTIN Bulletin 52(1): 55–89; DOI 10.1017/asb.2021.23. Discrimination-Free Insurance Pricing.
Lindholm, Mathias, Ronald Richman, Andreas Tsanakas, and Mario V. Wüthrich (2024). “What Is Fair? Proxy Discrimination vs. Demographic Disparities in Insurance Pricing.” Scandinavian Actuarial Journal 2024(9): 935–970; DOI 10.1080/03461238.2024.2364741. What Is Fair? Proxy Discrimination vs. Demographic Disparities in Insurance Pricing.
Xin, Xi, and Fei Huang (2024). “Antidiscrimination Insurance Pricing: Regulations, Fairness Criteria, and Models.” North American Actuarial Journal 28(2): 285–319; DOI 10.1080/10920277.2023.2190528, online 2023. Antidiscrimination Insurance Pricing: Regulations, Fairness Criteria, and Models.
Krafcheck, Eric, Igor Balnozan, and Fei Huang (2026). “Fairness Metrics for Life Insurance.” Society of Actuaries Research Institute, February 2026. Fairness Metrics for Life Insurance.
Huang, Fei, and Giles Hooker (2026). “Fairness Testing for Algorithmic Pricing.” arXiv:2605.11614, May 12, 2026; preprint. Fairness Testing for Algorithmic Pricing.
Xin, Xi, Giles Hooker, and Fei Huang (2026). “How Proxy Race Distorts Regression-Based Fairness Audits.” arXiv:2603.17106, March 17, 2026; preprint. How Proxy Race Distorts Regression-Based Fairness Audits.
Hébert-Johnson, Ursula, Michael Kim, Omer Reingold, and Guy Rothblum (2018). “Multicalibration: Calibration for the (Computationally-Identifiable) Masses.” Proceedings of Machine Learning Research 80: 1939–1948. Multicalibration: Calibration for the (Computationally-Identifiable) Masses.
Kearns, Michael, Seth Neel, Aaron Roth, and Zhiwei Steven Wu (2018). “Preventing Fairness Gerrymandering: Auditing and Learning for Subgroup Fairness.” Proceedings of Machine Learning Research 80: 2564–2572. Preventing Fairness Gerrymandering: Auditing and Learning for Subgroup Fairness.
Guo, Chuan, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger (2017). “On Calibration of Modern Neural Networks.” Proceedings of Machine Learning Research 70: 1321–1330. On Calibration of Modern Neural Networks.
Brier, Glenn W. (1950). “Verification of Forecasts Expressed in Terms of Probability.” Monthly Weather Review 78(1): 1–3; DOI 10.1175/1520-0493(1950)078<0001:VOFEIT>2.0.CO;2. Verification of Forecasts Expressed in Terms of Probability.
New York State Department of Financial Services (2024). “Insurance Circular Letter No. 7 (2024): Use of Artificial Intelligence Systems and External Consumer Data and Information Sources in Insurance Underwriting and Pricing.” Final New York State circular letter, July 11, 2024. Insurance Circular Letter No. 7 (2024): Use of Artificial Intelligence Systems and External Consumer Data and Information Sources in Insurance Underwriting and Pricing.
National Association of Insurance Commissioners (2023). “NAIC Members Approve Model Bulletin on Use of AI by Insurers.” Official NAIC press announcement, December 4, 2023. NAIC Members Approve Model Bulletin on Use of AI by Insurers.
European Insurance and Occupational Pensions Authority (2025). “Opinion on Artificial Intelligence Governance and Risk Management.” Official EIOPA Opinion, August 6, 2025. Opinion on Artificial Intelligence Governance and Risk Management.
International Association of Insurance Supervisors (2025). “Application Paper on the Supervision of Artificial Intelligence.” Official IAIS application-paper release, July 2, 2025. Application Paper on the Supervision of Artificial Intelligence.
Actuarial Standards Board (2016). “Actuarial Standard of Practice No. 23: Data Quality.” Revised December 2016; effective April 30, 2017. Actuarial Standard of Practice No. 23: Data Quality.
Actuarial Standards Board (2019). “Actuarial Standard of Practice No. 56: Modeling.” Revised December 2019; effective October 1, 2020. Actuarial Standard of Practice No. 56: Modeling.
Tabassi, Elham, for the National Institute of Standards and Technology (2023). “Artificial Intelligence Risk Management Framework (AI RMF 1.0).” NIST AI 100-1, January 26, 2023; DOI 10.6028/NIST.AI.100-1. Artificial Intelligence Risk Management Framework (AI RMF 1.0).