Insurance by BogdanovaThe Encyclopedia of Insurance and Risk Science

RESEARCH DOSSIER

Human Underwriter Consistency and Inter-Rater Reliability

Abstract. Human underwriters interpret evidence, classify exposures, and exercise delegated judgment under changing underwriting mandates. Their decisions can diverge in risk classification, eligibility, price, coverage, and referral even when submissions are comparable. This monograph examines the sources and consequences of disagreement, distinguishes agreement, reliability, validity, and policy conformity, and develops an evidence-grounded audit design for insurance institutions. Direct underwriting literature, statistical research, and professional and supervisory documents are synthesized with transparent access limitations. Illustrative calculations demonstrate chance-corrected agreement, prevalence sensitivity, continuous reliability, and uncertainty. The analysis addresses authority, specialist expertise, training, selective labels, procedural fairness, and accountable escalation. Findings support risk-sensitive calibration and independent blinded comparison rather than a universal agreement threshold. No insurer-level dataset or industry-wide dispersion estimate is claimed.

Keywords: insurance underwriting; inter-rater reliability; human judgment; agreement; noise audit; risk classification; blinded case review; calibration; underwriting authority; fairness; selective labels

Introduction and Conceptual Foundations

Underwriting is the institutional decision through which an insurer transforms an application into a proposed allocation of coverage, obligations, price, and capacity. The human underwriter receives evidence, interprets it within a product and an underwriting mandate, and decides whether a risk may be assumed, what terms are warranted, and whether further evidence or authority is required. The outcome may therefore include a risk class, quotation, exclusion, deductible, coverage limit, documentation request, referral, deferral, or declination. Each is a distinct decision variable. A study that records only an issued premium omits important variation in access to insurance and in the substance of the contract.

Human underwriter consistency is the degree to which evaluations made under comparable information, decision rights, policy versions, and portfolio constraints lead to equivalent decisions or justifiably equivalent ranges of decisions. Inter-rater reliability is the reproducibility of ratings across reviewers under a specified sampling and measurement design. Repeatability concerns the same person's decisions on repeated presentations. Agreement concerns the proximity of realized ratings; reliability often concerns the proportion of variation associated with differences among cases rather than measurement error. Validity concerns whether the judgment corresponds to the construct or objective that the insurer legitimately seeks to evaluate. Policy conformity concerns whether the decision is authorized and grounded in the applicable rules. These properties must be measured separately.

The comparisons are conditional. A property underwriter acting under a catastrophe accumulation limit cannot automatically be compared with a colleague considering an identical building in a different portfolio. Two life underwriters examining identical medical data can legitimately reach different interim recommendations if one has authority to approve a class and the other must refer it. The object of this investigation is the residual variation in judgments after relevant circumstances are aligned or measured. It includes the important interaction between a reviewer and particular cases. It excludes price competition between unrelated insurers as a measure of within-insurer consistency.

Three questions organize the research. How much of dispersion can be attributed to a stable reviewer tendency, a case-specific interpretation, a transient occasion, or an institutional process? When should differences be retained as evidence of expertise, changed information, or a deliberately authorized alternative, and when should they be treated as avoidable errors? Which forms of measurement, training, governance, and follow-up can reduce material decision discrepancies while retaining informed judgment?

The method is a structured research synthesis, not a systematic review or pooled estimate. Searches on November 19, 2026 combined insurance underwriting, same-case decisions, ambiguity, cognitive bias, life expectancy estimates, inter-rater reliability, agreement statistics, reporting guidance, selective labels, and underwriting supervision. Direct insurance studies and reports are distinguished from general statistical methods and institutional documents. The accessible text or indexed record supporting each claim is identified in the accompanying evidence ledger. Where only an abstract or indexed excerpt was accessible, substantive interpretation is confined to it. Hypothetical numerical computations are a mathematical demonstration, not a dataset or estimate of actual market performance. The study focuses on decision reproduction; broader underwriting optimization and the design of actuarial tariffs remain separate topics.

A sound institutional starting point is the relationship between risk evidence and expected outcomes. The Actuarial Standards Board's ASOP No. 12 (2005, revised 2011), section 3.2.1, makes this relationship explicit for actuarial risk classification and relates equitable rate differences to material differences in expected cost. This professional standard does not define an insurance-wide inter-rater threshold. Its relevance is that reproducible treatment should still have an evidence-based economic justification. A consistently applied rule can be wrong; a statistically accurate prediction can be used in an unauthorized decision; a rule-compliant judgment can have materially adverse effects that deserve independent examination.

The unit of analysis

A case is a versioned submission for a specified product, coverage request, valuation date, jurisdiction, insured or exposure unit, and decision stage. A decision is a reviewer's output conditional on that case, the instruction set, and the information made available. The case identifier represents the complete evidence snapshot rather than merely the applicant's identifier. Renewals, endorsements, separate layers, or amended submissions should be separate versions linked to the same underlying risk cluster. Repeated presentation of one version permits estimation of repeatability; comparison of materially changed versions measures responsiveness to evidence.

The population of reviewers also requires definition. A reliability study may concern a named panel, a particular underwriting office, or the wider population of qualified staff eligible for a class. Reviewers with materially different authorities should be compared on common judgment tasks or separately within authority groups. An estimand based on independent qualified reviewers does not automatically describe a workflow involving a senior sign-off, a queue, or an underwriting committee. A useful study defines whether it intends to characterize the average qualified reviewer, the worst credible disagreement within a team, or the reliability of the institutional decision taken after the full process.

The significance of conditional equivalence

Conditional equivalence allows equality of treatment without demanding identical actions across changing economic states. A risk may be acceptable at one limit and unacceptable at another. The same technical risk assessment may result in different offered premiums because of taxes or lawfully approved commercial actions. The same expected loss can be associated with different tail risk. The audit must identify the dimension that is allowed to change and record its justification.

The practical consequence is that agreement is interpretable only against an explicit decision specification. The specification states the information set, permitted considerations, risk tolerance, required authority, allowable terms, and materiality of differences. If a classification rule leaves ambiguous the meaning of a particular exception, repeated disagreements may indict the rule itself. Penalizing the individuals before resolving that ambiguity confuses administrative reproducibility with procedural compliance.

Human Decision Variability

Direct insurance evidence and the limits of generalization

Kunreuther, Meszaros, Hogarth, and Spranca (1995) report in Ambiguity and underwriter decision processes that commercial property-and-casualty underwriters assigned higher premiums to risks characterized by ambiguity about event probability or loss magnitude. The accessible abstract identifies ambiguity as a relevant decision input. It does not supply an inter-rater coefficient or prove that the different premiums were erroneous. Ambiguity can warrant a margin, and differences in the chosen margin can also reveal divergent interpretations of the insurer's risk appetite. The distinction must be investigated in a controlled case design.

Xu's (2020) life settlements underwriting comparison examines estimates from four U.S. medical underwriting providers, exploiting a normalized mortality measure and life settlement samples from 2011–2016. The available abstract describes relative conservatism that changes across cohorts and potential selection by intermediaries. This is provider-level evidence in a secondary insurance-related market: organizations may differ in models, evidence processing and incentives. Its results motivate investigation of case-by-reviewer effects and selection, but cannot be interpreted as evidence about interchangeable individual employees within a primary insurer.

The study of Lo, Patel, and Calder (2015), a P&C prediction survey, is an especially useful counterweight to stereotypes about professional groups. Four prediction tasks in one company did not produce evidence supporting the expected systematic difference between underwriters and analysts in the directions hypothesized. The subjects were forecasting problems, not replicated underwriting acceptance files. The reported result neither establishes equivalence between professional groups nor rules out variation among individuals. It demonstrates the importance of testing actual decisions rather than attributing a fixed judgmental defect to a job title.

Kahneman, in his 2022 interview with Frueh, describes in an interview on judgment noise an exercise in which individuals at one insurer priced common cases. He reports variability close to half compared with executives' expectations of about one tenth. The account is a primary testimony about an investigation, but its public narrative is not a replicable dataset with a fully specified estimator. Its value lies in the demonstration that managerial intuitions can underestimate differences revealed by independent evaluations. The figures must not be transformed into an industry rate of error or a probability of incorrect decisions.

Pui and Feddersen (2022), in their underwriting judgment investigation, combine case-like questions with practitioner feedback and benchmarking and report a costly gap between anticipated and observed costing dispersion. Their reported 27% and 79% are study-specific descriptive comparisons, with limited publicly available design details. The results are not pooled with Kahneman's report because the populations, tasks, and calculation definitions have not been harmonized. Their discussion of referral framing suggests a testable mechanism, not an experimentally established intervention effect.

Together these sources support a modest but consequential conclusion. Underwriting variability can appear in pricing, forecasting, and medical inference, but its magnitude and meaning depend on the population and task. The available records do not identify a representative worldwide estimate of disagreement in insurance acceptance, nor establish what fraction of variation is avoidable. The strongest defensible contribution is a design capable of resolving the question within specified institutions.

Systematic and case-dependent variability

Consider an illustrative continuous judgment in comparable monetary or transformed units. A conventional crossed random-effects representation is y(i,j,t) = mu + c(i) + r(j) + u(i,j) + e(i,j,t), where c is a case effect, r a persistent reviewer effect, u a case-by-reviewer interaction, and e an occasion-specific residual. This is a familiar measurement model applied to the underwriting task. It is not a newly proposed theory or a fitted model. Fixed effects for measured information differences, product, policy edition and relevant experience may be introduced, provided the design supports estimation.

A conservative reviewer might set a higher indicated cost across most cases; that pattern contributes to r. A reviewer who places extra emphasis on a particular industrial process could deviate on a subset of manufacturing cases while agreeing elsewhere; that pattern contributes to u. Fatigue, an interrupted reading session, or temporary uncertainty can contribute to e. The terms are sources of variance, not established causes. Causal attribution requires an additional design and observed or randomized covariates.

Under mutually independent, centered random components with finite variance, the variance of a single observation is the sum of component variances. For distinct independent reviewers assessing the same case on independent occasions, the case effect cancels in the paired difference and the variance of that difference is twice the sum of reviewer, interaction and residual variances. For repeat assessments by the same reviewer, both the case and stable reviewer-by-case effects cancel, leaving twice the occasion variance. These calculations require independence across the compared occasion errors and stable case snapshots. Correlated team behavior, shared prompts, or learning across sessions invalidate that simple decomposition.

If a study has only one observation for each case-reviewer pair, the interaction and occasion-level noise cannot be separately estimated without additional assumptions. Repeated ratings of a randomized subset provide information about the residual. If all complex cases are assigned to specialists and all simple cases to juniors, reviewer and case difficulty are confounded. The solution is an overlapping, authorized crossed design, not the retrospective comparison of employees' raw acceptance rates.

Information, thresholds and workflow

Human decision dispersion can begin before a risk estimate is formed. An application contains narrative representations and structured data; summaries may disagree with underlying documents; an inspection may report a mitigation measure without evidence of its operation. Reviewers may locate different parts of the file, make different assumptions about missing facts, or interpret words such as adequate, recent and exceptional differently. A protocol should record information requests, unresolved evidence and the policy provisions cited. This makes it possible to distinguish differences in what was seen from differences in what was inferred.

The next stage is the assessment of uncertainty. An individual may give a point estimate and a range, or identify an uninsurable hazard. Point estimates alone hide the difference between low expected loss with severe tail uncertainty and moderate expected loss with well-understood variance. A standard decision form should separate estimated quantities from commercial adjustments and decision reasons. Requiring decimal precision when the underlying evidence is sparse creates a misleading appearance of certainty.

The last stage is conversion of assessment into action. An institution can establish consistent estimates yet inject discretion at the threshold controlling binding authority, risk class assignment, or referral. Consider two reviewers with indicated losses of 98 and 102 on a scale with an approval threshold of 100. Their relative difference is small, but their final actions may diverge completely. Conversely, a large numerical difference far from any relevant threshold may yield the same action. Thus one numerical agreement measure is insufficient for assessing access or capacity consequences.

Short-term commercial objectives may also shift decisions. A renewal retained at an approved discount and a new risk declined for accumulation reasons need not reflect inconsistent risk estimates. Production metrics should preserve technical estimates and commercial adjustments as separately recorded fields so that reviewer variation is not confused with legitimate business discretion.

Blinded Comparison Protocols

Study design and sampling

The core audit is a blinded, version-controlled common-case exercise, with independent assessments before discussion. The objective is to observe the decision that each qualified reviewer would make under the same available evidence and instructions. Blinding concerns the previous decision, other reviewers' outputs, and subsequent losses. Personally identifying information can be masked when ethically and legally permissible without destroying relevant risk facts. A blind exercise must not delete legitimate inputs and then penalize reviewers for uncertainty that the exercise created.

Define the target population and document the sampling frame before collecting results: time interval, product, geography, new business or renewal, application completeness, authority tier and submission channel. Sample both the central distribution of ordinary submissions and rare decisions with significant consequences. A pure random sample of abundant standard approvals may underestimate the difficulty of marginal risks. A deliberately enriched sample of borderline cases, on the other hand, overstates operational disagreement if its frequencies are treated as market frequencies without weighting.

For a stratified case design, record the inclusion probability of each stratum and report both stratum-specific results and operationally standardized results. If 10% of submissions are borderline but half the sample is borderline, the overall target must be reweighted to the intended population. For pairwise agreement, weight case-level agreement indicators before computing the aggregate; the marginal distributions used for chance correction must reflect the same target weighting. Report the enriched-sample coefficient as a distinct stress-test quantity.

An economically practical design uses a common case core evaluated by all eligible reviewers with a larger balanced incomplete block of additional cases. The overlap should permit estimation of reviewer contrasts. Case allocation must be randomized within eligible reviewer-product groups or otherwise justified. Every isolated cluster of reviewers and cases weakens the ability to disentangle reviewer from case effects. For retest analysis, present a subset in independently randomized order after a prespecified interval that minimizes recall while avoiding actual policy drift. Reuse the same frozen evidence, rules and market constraints.

The exercise should reflect actual decision rights. A junior reviewer cannot legitimately bind a risk beyond delegated authority. Ask for an underlying assessment and a proposed action separately from the final authorized action. The first can be compared across expertise levels; the second measures compliance with the role. Keep institutional referral decisions distinct from final authorization by the referral authority.

Data structure and protocol registration

A minimum record includes case ID and version, submission population and sampling stratum, timestamp and reviewer pseudonym, qualification and authority tier, product and jurisdiction, guideline version, evidence packet hash, input completeness, technical estimate and uncertainty, eligibility or action, proposed terms, explicit reason codes, information requests, referral requirement, exception rationale, and whether an independent review was completed. Record time spent, aborted ratings, and why an assessment is missing. The linkage key should be protected separately from the analytic dataset.

Use a pre-specified estimand and a written analysis plan: which reviewers are interchangeable; whether case mix is weighted; whether the rating is nominal, ordinal, or continuous; acceptable absolute discrepancies or decision-cost tolerances; how missing assessments are classified; which contrasts are exploratory; and the planned uncertainty interval. The same frozen case set makes paired comparison of two tools possible, but randomized presentation order and suitable washout are needed to control learning.

Kottner and colleagues' (2011) GRRAS reporting guidance emerged from clinical measurement research and proposes reporting issues covering agreement and reliability studies. Adapting the reporting objective to insurance entails documenting the exact submission versions, policy and authority settings, the reviewer sampling frame, study timing, statistics and uncertainty. It does not transform a clinical reporting guideline into a binding insurance regulation.

Nominal outcomes

For two reviewers and mutually exclusive outcomes, report the cross-tabulation, observed agreement P_o, the expected agreement P_e under the product of their observed marginal category probabilities, and Cohen's kappa, (P_o - P_e)/(1 - P_e). The coefficient is associated with Cohen's 1960 paper; the methodological exposition in Zapf et al. (2016) independently documents its two-rater, nominal role and the broader family of coefficients. An undefined denominator when P_e equals one indicates a degenerate comparison rather than a perfect calibrated result.

A fully synthetic binary audit illustrates the calculation. The same 100 files are reviewed by A and B with the instruction that only an accept or a decline is feasible. The data are hypothetical:

Reviewer A / Reviewer B Accept Decline Total
Accept 70 10 80
Decline 5 15 20
Total 75 25 100

Observed agreement is (70 + 15)/100 = 0.85. Expected agreement is (0.80 x 0.75) + (0.20 x 0.25) = 0.65. Kappa is 0.20/0.35 = 0.5714286. A's acceptance rate is five percentage points higher. Ten cases are accepted by A alone and five by B alone, providing the direction of discordance. Category-specific positive agreement is 140/(140 + 10 + 5) = 0.9032 for acceptance and 30/(30 + 10 + 5) = 0.6667 for declination. Neither quantity determines who made a valid decision.

The practical implication is visible in the fifteen affected applications, not just the coefficient. A file-level comparison of reasons can establish whether the ten A-only acceptances arise from unauthorized exceptions, evidence overlooked by B, different perceptions of materiality, or unresolved ambiguity. Collect this information without showing either reviewer the other decision before independent recording.

Prevalence sensitivity matters. Consider a second hypothetical table of 100 cases: both accept 94, A-only accept 3, B-only accept 3, both decline zero. Observed agreement is 0.94, the acceptance marginal for each reviewer is 0.97, and expected agreement is 0.9418. Kappa is approximately -0.0309, even though the absolute agreement is high. The six discordant cases represent all decisions involving at least one declination. A manager who celebrates the first statistic alone misses this failure to reproduce the rare decision, and one who interprets the second as a universal failure ignores the highly concentrated population. Gwet's (2008) high-agreement reliability study identifies this methodological problem as a subject of direct investigation; alternative chance baselines and raw cross-tabulations are sensible sensitivity checks rather than devices for selecting the most flattering score.

Multiple reviewers and ordinal outcomes

For more than two raters, a nominal analysis may use an appropriate multirater coefficient. Fleiss' K uses the observed average within-case pair agreement and a pooled category distribution; its sampling assumptions and chance model differ from Cohen's two-specific-rater coefficient. In a complete equal-ratings design the within-case agreement is the sum across categories of n_c(n_c - 1) divided by m(m - 1), where m is the number of ratings and n_c the count in category c. Averaging these quantities over cases yields an observed pair agreement. Different reviewer structures may require different coefficient choices.

Krippendorff's alpha expresses disagreement as 1 - D_o/D_e using an appropriate distance function and coincidence structure. Its eligibility for incomplete data does not mean that informative missingness disappears. Zapf and colleagues (2016) compared Fleiss' K and alpha through simulations and a case study; their missingness simulations used completely random deletion. A referral-only reviewer who omits every complex case therefore creates a different problem from random omission. Confidence intervals and sensitivity to missingness mechanisms must reflect the real design.

Risk classes such as preferred, standard, substandard and decline may form an order within a specified underwriting product, but referral and deferment are different procedural states, not automatically intermediate risk classes. A weighted kappa can allocate partial credit to ordered differences with preregistered weights. For example, a four-class linear agreement weight is 1 - |a-b|/3; a quadratic agreement weight is 1 - ((a-b)/3)^2. Expected weighted agreement uses the product marginal cells under the same weights. These weights express analytical choices about class distance, not a measured insurance cost. A one-class difference can have nonlinear financial significance, especially when the transition eliminates eligibility. For competing coverage options without a defensible order, use nominal comparisons or an explicit cost matrix.

Continuous judgments and absolute agreement

A technical premium estimate, expected loss cost, or quantitative risk score can be compared through absolute differences, relative differences, rank association, or variance-based reliability. The appropriate object is defined before computation. Pearson correlation may be perfect even if one reviewer always adds a large surcharge. Koo and Li (2016) distinguish ICC model, single-versus-average measurement and agreement-versus-consistency in their ICC selection guideline. A two-way random-effects, single-rating, absolute-agreement estimand is suitable when the objective is interchangeability among eligible reviewers drawn from a defined population under a crossed design. A fixed panel or a multi-reviewer final decision calls for a different estimand. The 2017 erratum to the guideline specifically identifies an error in Table 3; the variance ratio below is derived from the stated independent random-effects assumptions rather than copied from that table.

Under the simplified random-effects model earlier, the single-rating reliability ratio for absolute agreement is case variance divided by the sum of case, reviewer, interaction and occasion variances. Suppose the components on a common numeric scale are 100, 16, 9 and 4. The ratio is 100/129 = 0.7752. The corresponding judgment variability between two independent reviewers on the same case has standard deviation sqrt(2 x 29) = 7.6158 units. A repeated judgment by the same reviewer under unchanged conditions has difference standard deviation sqrt(2 x 4) = 2.8284 units. These are algebraic consequences of the stated illustrative model, not estimated statistics for an insurer.

If case variance increases from 100 to 400 while the other components remain fixed, the ratio rises to 400/429 = 0.9324 even though the paired reviewer spread remains 7.6158. Therefore an ICC improvement may arise from increased case heterogeneity without improved practical agreement. Compare reliability ratios on a stable case mix and report absolute dispersion on the original business scale.

For paired numeric estimates with differences d_i = A_i - B_i, report the mean difference, standard deviation and a graph of differences against paired means. Bland and Altman (1986) introduce method-comparison analysis in clinical measurement. In a simple independent-pairs setting with roughly normally distributed differences and stable variance, descriptive limits of agreement are mean difference plus or minus 1.96 standard deviations. They are estimates of a distribution of differences, not confidence intervals for the mean, and their own sampling uncertainty should be reported. Repeated or clustered files, outliers or heteroscedasticity require adaptations.

For positive premiums, a log ratio can be preferable when variation scales with the amount; back-transform limits as ratios, and identify the direction. A symmetric relative absolute difference 2|A - B|/(A + B) is useful when both amounts are positive and comparable; zero plus zero is undefined. A review of the premium alone must verify that limits, deductible, coverage and period are identical. Otherwise the apparent price disagreement can be a difference in the insurance product.

Uncertainty, missingness and materiality

A confidence interval for the observed agreement or reliability parameter describes sampling uncertainty under the design. It does not validate the decisions or certify that each application received suitable treatment. Use case-cluster resampling when cases are independent but multiple ratings occur within cases. If generalizing to new reviewers as well as new cases, a hierarchical model or an appropriate crossed resampling scheme is required; resampling cases alone conditions on the observed reviewers. With rare categories, report actual event counts and avoid high-dimensional intervals that hide sparse cells.

A sample-size calculation should follow a target precision for each important raw disagreement rate and reliability estimate. For illustration, with independent binary case-level disagreement indicators and an anticipated disagreement probability of 0.15, the normal approximation for half-width 0.05 at 95% confidence is 1.96 squared times 0.15 times 0.85 divided by 0.05 squared, about 196 cases, rounded upward. This is a planning approximation only; weighting, clustering, rare stratum objectives, dropout and small-sample performance may require a much larger design. A power analysis or simulation should reproduce the actual assignment structure.

Missing decisions fall into two categories: a recorded, policy-relevant inability to decide (which may be a meaningful outcome) and missing measurement (which requires a reason and analysis). A controlled experiment may prohibit further evidence gathering to isolate interpretation; if so, it measures decision behavior under an artificial constraint. A naturalistic validation round should allow permitted requests and record the costs of delay and collection as outcomes.

A material disagreement is one that exceeds a prespecified decision-relevant tolerance. An institution may prioritize unauthorized binding, inconsistent rejection or coverage exclusion, limits beyond capacity and unexplained changes in risk class. The goal is to bound the rate of consequential disparities with uncertainty, not to optimize one coefficient. No generic label such as substantial agreement establishes that all these risks are within tolerance.

Authority and Specialist Expertise

Authority is the institutionally specified right to make a decision that commits the insurer or advances a submission. Technical expertise is the capacity to interpret evidence relevant to risk. The two can overlap but are different. A skilled engineer may provide an assessment without binding authority; a senior underwriter may have authority without possessing the specialist knowledge required for an unusual industrial process. A reliability program should observe the decision chain rather than collapsing all roles into interchangeable reviewers.

At Lloyd's, the official Underwriting Profitability principle identifies controls, monitoring and reporting aligned with risk profile and business plan. The page establishes the importance of underwriting control as a market oversight objective. It does not mandate a particular coefficient or numerical threshold. A corresponding institutional research question is whether reviewer dispersion increases when authority language is ambiguous or when escalation triggers conflict with operational incentives.

Expertise-conditioned agreement

To compare a specialist and a generalist, first ask whether both were expected to make the same inference under the same evidence. The generalist may properly detect uncertainty and request specialist review; the specialist may properly provide the more specific estimate. In this setting identical final selections could indicate inappropriate overreach, while a distinct referral is correct. Compare the shared evidence-screening task, the referral decision, and the authorized final disposition separately.

Specialist review is also not an infallible reference. An expert panel can use an obsolete assumption, or discount a rare risk identified by a minority reviewer. A study should distinguish a reference decision derived from a proven rule from an adjudicated expert consensus when no uniquely correct outcome is observable. Record the evidence, independence and authority of the reference panel. When multiple valid outcomes exist, define an admissible decision set and evaluate membership in that set. A decision may differ from a reference choice without being outside the admissible set.

A common error is to evaluate each reviewer against a reference panel that includes the reviewer. This creates incorporation bias. Leave the evaluated reviewer out of the reference process or use an independent panel. Reviewers should have the same evidence as the panel when the assessment concerns original decision correctness; later information should be used only for a separate hindsight analysis. If a case is excluded because adjudicators cannot reach consensus, report the exclusion rate, since it may identify the most difficult cases.

Escalation, decision aggregation and accountability

A manager can resolve conflicting judgments through a prescribed escalation rule. Independent initial assessments should be retained before an expert or committee reconciles them. A discussion can improve the final result by pooling relevant evidence; it can also induce conformity to hierarchy or an initial estimate. Retaining both the pre-discussion and post-discussion decisions measures the change while preserving information about latent variability.

For quantitative continuous forecasts, averaging independent centered errors reduces their variance as the number of reviewers grows. Under identical error variance sigma squared and common correlation rho, the variance of the mean of m errors is sigma squared times (rho + (1-rho)/m). Correlation is the limiting factor: when all reviewers reproduce the same incorrect model output, averaging offers no correction of that shared error. In practical underwriting, decisions that concern exclusions, legally restricted classifications or authority boundaries cannot be averaged as if they were numerical forecasts. They require admissibility checks and an accountable decision maker.

The economically appropriate referral strategy weighs potential decision loss, investigation cost, delay, and customer impact. A second independent review of every file may deliver little benefit for standard cases decided by explicit rules. Cases near a material boundary, involving inconsistent evidence or exceptional capacity, can merit a higher review intensity. Estimate the effect of referrals prospectively, including cases that end in withdrawal, not only the business ultimately bound.

Audit evidence should specify the owner of the final decision, whether mandatory specialist input was obtained, the applicable authority and guideline edition, and the rationale for exceptions. Otherwise disagreement may be eliminated on paper by overwriting the earlier decisions rather than resolving the institutional source. An authority control should remain objectively auditable even when substantive risk expertise entails discretion.

Organizational sources of reviewer variation

Queue assignment, workload and availability of expert support can generate variation that looks personal. A reviewer assigned difficult files at peak hours may use a higher referral rate than a colleague with routine work and immediate access to a specialist. A hierarchical model can estimate conditional contrasts under adequate overlap, but adjusted associations are not automatically causal. Random allocation among qualified reviewers or randomized queue strategies in a nonbinding experiment can isolate particular effects.

Incentives require similar scrutiny. A reviewer rewarded for volume has an incentive to treat documentation or exclusions differently from one evaluated solely on reported early loss ratio. Identifying a difference is the beginning of the investigation. The insurer must distinguish business appetite set by management from individual conduct outside the mandate; protect legitimate disclosure of policy ambiguities; and assign correction to the level of the organization that controls the source of the variation.

Training and Calibration

Training for reproducible judgments is a cycle in which independent decisions are compared, evidence-based explanations are examined, shared standards are clarified, and the effect of interventions is measured on previously unseen cases. Agreement obtained by memorizing a benchmark file measures recall. Agreement obtained by passing new files under frozen instructions measures transfer. Training quality therefore depends on the separation of development, testing and monitoring samples.

A calibration cycle

Begin with baseline assessment on blinded cases spanning routine, borderline, and unusual risks. Decompose disagreement by judgment component and case type. A high rate of inconsistent documentation requests calls for a different intervention from systematic overestimation of loss cost. A policy reviewer should inspect ambiguous wording, and technical specialists should examine shared assumptions. The adjudication process should identify the correct or admissible output, its supporting evidence, and the confidence in that reference.

Translate recurring disputes into unambiguous guidance supported by realistic examples. Include contrasting cases with one legitimately changed factor, as well as superficially similar cases whose treatment should coincide. Specify what evidence is required, which options remain within discretion, when additional information is appropriate, and when a referral is mandatory. Do not eliminate a justified alternative simply because the majority made a different choice.

Conduct training on development cases and collect independent decisions on a held-out set. Preserve the reference standard and scoring plan before examining the results. A parallel control group or appropriately randomized staggered implementation helps estimate the incremental effect of training. If the same staff assess both conditions, randomize order or use a design that accounts for carryover: new knowledge from the intervention cannot simply be erased during the control period.

The practitioner work of Pui and Feddersen (2022) also discusses feedback and judgment prompts. Its suggestions motivate an intervention that separates individual uncertainty, evidence interpretation, and team calibration. The effectiveness of each device in a given insurer remains an empirical question. Changes in the composition of applications, pricing manuals or staffing during a before-and-after comparison can produce changes in apparent reliability without any training effect.

A practical program records a versioned case library with: original submission snapshot; evaluation context; admissible answers and their rationales; known uncertainties; independent adjudication; and documented policy version. When policy changes, retain the old examples for historical audit but create a new version for future training. Publishing the answer key to all employees destroys the independence of future tests using those cases. A rotating protected hold-out bank, with fresh cases collected under the same sampling strategy, can prevent this.

Calibration of probabilities and confidence

Agreement between two reviewers is distinct from calibration against outcomes. For a probability forecast p of an explicitly defined event within a fixed observation horizon, the Brier score is the average of (p - outcome)^2. Reliability diagrams compare predicted probabilities with empirical event frequencies in comparable exposure groups. A forecast that consistently assigns 0.1 to groups in which the event occurs 30% of the time may be reproducible yet systematically miscalibrated.

In underwriting, a risk class generally summarizes a distribution of future outcomes and contract obligations. A single individual's observed claim cannot determine whether the initial estimate was correct. The appropriate test uses a cohort at sufficient exposure and maturity, consistent coverage conditions, allowance for censoring, and adjustments for changes in claims reporting. Long-tail lines require development-sensitive evaluation; life insurance mortality depends on exposure duration and competing risks. Economic profitability further depends on premiums, acquisition costs, capital, expenses and reinsurance, rather than solely initial predicted claim probability.

Ask reviewers to document uncertainty without requiring them to invent precise probabilities where evidence is insufficient. For genuinely probabilistic forecasts, monitor calibration by probability band and score proper predictions; for ordinal risk decisions, examine transitions and the validity of class assignment. For reasons that are required to be factually accurate, audit evidence matches. Confidence judgments can be evaluated by repeated assessments, but subjective certainty is not an objective certificate of decision validity.

Selection and feedback

The observed claim history is determined partly by prior acceptance and customer take-up. Withdrawn or rejected applicants generally do not furnish comparable insured claim observations to the original insurer. Lakkaraju and colleagues (2017) describe the selective labels problem in evaluating decision makers and algorithms when the observed outcomes depend on human choices. This selection is central to underwriting calibration: the losses on the bound portfolio cannot directly determine the accuracy of every decline.

Risk assessment and commercial terms can themselves change outcomes by altering selection, behavior, deductibles, limits and the insurer's payment obligation. Evaluating a decision policy's effect therefore requires a defined counterfactual or a justified proxy rather than an unadjusted comparison of accepted cohorts. Randomized experiments involving binding decisions may be unacceptable or illegal; nonbinding case experiments, independent evidence, permitted randomized audits, and transparent observational sensitivity analyses offer safer sources of evidence. No method eliminates the identifying assumptions simply by providing a numerical estimate.

A reviewer with a low reported loss ratio might have selected well, restricted access excessively, written a favorable mix, experienced benign catastrophe activity, or had a portfolio too immature for claims to emerge. A robust monitoring report therefore includes application-level disposition, policy take-up, exposure maturity, case mix, forecast performance, and downstream customer or contract consequences. Loss ratio remains useful but insufficient for ranking individuals' judgment quality.

Measuring the effect and retaining diversity

Before an intervention, specify minimum meaningful improvements in the rate of material discordance and the maximum tolerable deterioration in accuracy, compliant decision-making, turnaround, fairness or customer access. Measure both intended and unintended consequences. A uniform checklist can clarify common evidence requirements while creating systematic mistakes if it excludes rare but relevant hazards. A model recommendation may reduce variability while creating anchoring: distinguish a blind human assessment from a human assessment with model assistance.

A useful evaluation compares independent human, model-only, and assisted-human assessments on the same eligible evidence, with randomized assistance and a separately constructed reference where feasible. Evaluate disagreement with the human benchmark as a diagnostic, not proof of model error. A model that reproduces a flawed historical decision consistently can appear excellent by agreement alone. Conversely, an override may improve validity if it identifies evidence missing from the model.

The intended outcome of training is reliable application of justified standards and a repeatable treatment of uncertainty. A persistent disagreement that exposes a real gap in the guidance is productive research evidence. The intervention should refine the institutional rule or evidence pathway, retaining the original dissent and its justification so later audits can distinguish learning from enforced conformity.

Fairness and Consistency Measures

Fairness in underwriting concerns the justification, legality and consequences of differentiated treatment. Consistency asks whether materially comparable cases receive comparable judgment under the same institutional framework. A consistent discriminatory rule remains discriminatory. An inconsistent rule can harm applicants without producing an easily detected aggregate disparity. A credible audit therefore observes case-level comparability and distributional outcomes as independent dimensions.

Matched cases and lawful risk factors

A matched-case exercise holds legitimately relevant information constant while varying a cue under study. For example, compare synthetic business applications with the same insured activity, documented loss history, coverage request and risk mitigation evidence, differing in an irrelevant presentation feature. A difference in judgment may reveal sensitivity to the cue; randomization protects against systematic differences in assigned reviewer. The exercise must be reviewed for legality, ethics, use of personal data, and realism before applying it in a live underwriting environment.

The effect of changing a protected characteristic cannot always be defined by naïvely editing a field. Correlated features may make the resulting scenario inconsistent or alter legally permitted risk information in some jurisdictions and products. Specify the target intervention and the relevant legal context, and distinguish a valid paired experiment from observational disparity measures. Use synthetic or authorized deidentified data; obtain specialist legal and privacy review before processing protected attributes or their proxies. The research design here does not create a legal permission to collect any attribute.

The statistical analysis can report differential disagreement, differential referral or additional-document requests, rates of adverse actions, risk-class transitions, and variation in processing delay across relevant groups where lawful and feasible. Standardize by legitimate risk factors and common product and authority scope. Present both marginal and conditional results; they answer different questions. Conditioning on an intermediate variable affected by earlier decision-making may conceal part of a disparity. A causal claim requires explicit assumptions and an appropriate design.

A low overall disagreement rate may conceal poor performance in an uncommon group. Conversely, very large percentage differences with few cases may have substantial uncertainty. Use denominators, interval estimates and suppression or aggregation rules for small cells under privacy requirements. Report intersections only where supported by the sample and appropriate governance. Predefine multiple comparisons or distinguish exploration from confirmatory inference. Avoid treating a statistical significance threshold as a legal finding.

Fairness objectives and regulatory scope

ASOP No. 12 provides a US actuarial professional context for evaluating expected-cost relationships, within the scope of actuarial practice. It does not override insurance statutes or imply that every predictive characteristic is permissible. Economic relevance, legal permission, and equitable procedure need separate review. A cost relationship cannot by itself settle the legality of a risk classification.

In US life insurance, the NAIC's accelerated underwriting overview (updated April 2026) describes use of external data, analytics, and movement to traditional underwriting when available evidence is inadequate. It identifies 2024 committee guidance addressing data sources and unfair discrimination. NAIC guidance is a supervisory resource within a state-based system; applicable state enactments and examination practices must be checked before any legal conclusion. A human reviewer receiving only more complex referrals from an automated pathway cannot be compared against all automated approvals without accounting for selection.

New York's Department of Financial Services issued Circular Letter No. 7 on July 11, 2024. Its defined scope encompasses AI systems and external consumer data used in underwriting and pricing for specified regulated entities. Paragraphs 10 through 12 address prohibited discrimination, actuarial support and proxy assessments, and the final letter explains risk-based governance. This is a specific jurisdictional example of oversight of technology-assisted decisions, rather than a global legal rule for all human underwriting. It motivates testing whether manual reviewers who use the same model or data can identify errors, make consistent permitted overrides, and maintain a documented justification for adverse outcomes.

Internationally, the IAIS ICPs and ComFrame overview distinguishes global supervisory principles from the framework specifically addressing internationally active groups. The latter was updated in December 2024. An insurer operating across markets should maintain local legal and product-specific decision matrices rather than imposing a single worldwide risk-class definition. Jurisdictions differ in permissible inputs, product regulation and consumer remedies. These sources establish the institutional scope of oversight; the design of the empirical agreement audit is the author's analytical application.

Customer access and procedural accountability

The underwriting action affects the applicant even without an issued policy. An unexplained disparity in evidence requests can create extra costs and abandonment, while a different deductible or exclusion can alter the value of coverage. Monitor time to decision, documentation burden, referral paths, appeal frequency and corrected adverse actions. Equal acceptance alone is inadequate if the protection or opportunity to obtain it varies without legitimate explanation.

An applicant-facing reason should correspond to the actual decision basis, not merely to a standardized code that happens to fit. The internal record must distinguish risk evidence, policy authority, commercial constraints and any model output relied upon. The reason can then be reviewed against legal disclosure requirements for the relevant jurisdiction. An appeal process should be independently routed, with corrected evidence preserved and the original and revised decisions separately recorded.

The objective is to make comparable decision circumstances, justified differences and material human variability visible. Consistency controls can support fairness but cannot substitute for a substantive legal and actuarial evaluation of the rule that is being consistently applied.

Analytical Synthesis, Evidence Gaps and Reproducibility

Interpreting the evidence as a whole

The insurance studies accessible to this research are heterogeneous. The historical ambiguity field study evaluates pricing under uncertainty. The life settlements analysis evaluates differences across medical underwriting providers. The P&C prediction survey tests a professional-group hypothesis in one organization. The two practitioner narratives address judgment dispersion in specific same-case or case-like exercises. Reliability research supplies the statistical means of evaluating classification and measurement. Professional standards and oversight sources define institutional considerations. These different kinds of evidence cannot be combined into one estimate of the underlying population parameter.

The principal authorial conclusion is that underwriting dispersion should be evaluated as a conditional distribution of decisions and their consequences. A raw comparison of prices cannot identify the underlying decision mechanism because the price may depend on differences in coverage and authority. An agreement coefficient cannot identify whether the decisions are valid because reference evidence is independent of concordance. A review of accepted business cannot characterize all decisions because acceptance and take-up determine observed outcomes. These are distinct identification problems. They require the study to preserve multiple decision layers.

A decision audit therefore begins with the common information and rules, observes independent assessment, estimates conditional variation, distinguishes authority and permitted discretion, adjudicates material disagreement, evaluates outcome and access consequences with appropriate limitations, and tests corrective intervention on a held-out design. These are established research and governance practices integrated in an underwriting-specific sequence. The work makes no claim to a novel scientific law, named method, or validated proprietary model.

Why minimizing dispersion alone is insufficient

Let a true conditional target for an illustrative technical estimate be theta, and let two independent estimates have common expected signed error b and error variance v. Mean squared error for a single estimate is b squared plus v. The expected squared difference between the two estimates is 2v if their errors are independent and the mean bias is shared. Thus reducing the squared difference can lower random error while leaving shared bias unchanged. In an illustrative institution where every reviewer underestimates the same rare exposure by 20 units, perfect agreement can coexist with mean squared error of 400. This result follows from the standard bias-variance decomposition.

The institution should distinguish three questions: is the reviewer variation avoidable under the available evidence; do decisions accord with defensible evidence and legal or policy constraints; and what is the consequence of remaining variation for risk transfer, access and solvency? These questions correspond to different evidence and controls. If no reliable reference estimate exists, useful conclusions can still be drawn about unauthorized decisions and documented unexplained dispersion; substantive superiority cannot be inferred merely from popularity of an answer.

For a nominal decision, define a decision-specific loss table L(action, reference state) where the reference state has been constructed with explicit evidence and limitations. Such a table may place a larger cost on unauthorized acceptance or unlawful rejection than on a minor class difference. Its weights are an institutional valuation, subject to legal constraints, not objective market constants. Expected disagreement cost can be estimated only if the reference or a defensible distribution of it is available. In a no-reference audit one can report the expected cost of inconsistency under stated hypothetical weights, not the expected cost of actual error.

Decision audit and institutional interpretation

Observed audit pattern Plausible explanation Information needed before attributing error
Scores align; final actions diverge Threshold or authority differences Frozen rules, role and exception records
Wide numeric spread; identical actions Similar positions relative to decision threshold Contract terms, capacity and cost tolerance
High agreement; poor cohort calibration Common biased evidence or model Mature outcome data and selection analysis
Low rare-category agreement Scarcity, unclear definition or uneven expertise Category counts, paired reasons and specialty strata
Post-training agreement rises Learning, memorization or conformity Unseen cases, independent comparison and validity checks
Different group referral rates Risk mix, access barrier or data quality Lawful matched analysis and workflow records

These patterns are possible diagnostic interpretations, not observed frequencies. A single file can display several of them. The table provides an audit trail from observed pattern to a question that can be tested. It avoids the fallacy of assigning fault to a person on the basis of a coefficient alone.

The economics of correcting disagreement are asymmetric. A loss from inappropriate acceptance may arise through claims and excess capacity use; inappropriate rejection may cause lost positive expected contribution and harm to applicant access. Excessive referral can delay a valuable contract or burden applicants. In a competitive market, a quote that is too high can lose business, while a quote that is too low can attract risks on unattractive terms. These mechanisms justify treating intervention as an economic and procedural design problem, not simply as a statistical exercise.

A practical audit can prioritize the rate of disagreements that would have resulted in materially different valid or invalid contracts. Use case-specific exposure and authorized limits to define a severity measure; report raw rates as well, since monetary weighting alone can obscure harms to smaller applicants. Tolerances should be set before data collection based on the product, legal perimeter and risk appetite. A common institution-wide score may support monitoring, but cannot replace the underlying exposure-specific report.

Research gaps and empirical requirements

The literature located does not answer how large the within-insurer human acceptance-dispersion distribution is across markets. Further research requires qualified reviewers in multiple independent organizations, comparable frozen submissions, transparent category and premium definitions, repeated judgments, detailed authority and policy records, and population-aware case sampling. Cross-company comparison should harmonize outputs that share a legitimate meaning while preserving differences in coverage, model and risk appetite. A meta-analysis would need compatible estimands and uncertainty, which the retrieved reports do not provide.

Prospective trials of calibrated guidance, assistance tools, blind independent peer review, and feedback should assess not only agreement but procedural compliance, subgroup impact, turnaround and predictive outcomes. Existing judicial or clinical reliability methods are transferable as statistical methods, not as domain-specific intervention effect sizes. Investigation of informative missingness and rejected risks needs careful ethical design and explicit identification assumptions.

To study decision drift over time, freeze a stable anchor set while adding genuinely new samples that represent current business. The stable set detects scoring changes and the new samples detect shifts in the actual application mix. Some differences are expected when policy or economic states change, so the historical rule version must be preserved. Compare identical rules when studying human drift and separately evaluate the deliberate change in decision criteria.

Reproducible protocol specification

An empirical replication should publish or securely archive a protocol defining the estimand, inclusion criteria and reviewer eligibility, case allocation and overlap, frozen case versions and rulebooks, timing and blinding, variable dictionary, adjudication procedures, primary and secondary measures, treatment of missingness, interval estimation, and any prespecified intervention assignment. Archive analysis code, package versions, pseudonymized case keys, and reproducibility receipts within the applicable legal and information security perimeter. If actual submissions are confidential, a shared synthetic example and non-disclosive aggregate outputs can establish computational reproducibility without revealing private applicant information.

The illustrative binary counts and random-effects variances supplied above are sufficient to reproduce the arithmetic. The examples deliberately change marginal frequencies and case variance while preserving or nearly preserving other quantities to show why agreement and reliability cannot be read in isolation. The numerical demonstrations do not require external observations and do not estimate an insurer's actual performance.

The full research record distinguishes accessible primary documents and abstracts from indexed-only material. Several publisher links were unavailable to the direct retrieval mechanism; the corresponding claims have been confined to the available abstract or metadata, and this is recorded in the evidence ledger. No independent insurer files, unpublished underlying noise-audit data, or external peer review were obtained. These boundaries determine the permissible conclusions. They do not establish that additional empirical research has been exhausted.

The final scholarly judgment is that evidence about human underwriter consistency must remain tied to the cases, reviewers, context, outcome definition, and sampling design under which it was obtained. When the purpose is governance, the most informative results connect measured differences to rule clarity, authority, valid evidence, and the actual contractual consequences of a decision. A reproducible audit has scientific value even when it finds a legitimate difference rather than an error.

Conclusion

Human underwriting consistency is the reproducibility of justified insurance decisions within specified information, policy, product, authority, and portfolio circumstances. Inter-rater reliability measures a defined aspect of that reproducibility. Its interpretation requires separate evaluation of validity, conformity, access, and economic consequence.

The research evidence supports the existence of consequential variation in insurance-related judgment tasks, but the available studies are too different in design and level of aggregation to establish a common international prevalence or universal benchmark. A credible local investigation uses independent blinded case reviews, overlapping case allocation, retained initial decisions, repeated assessments, explicit statistical estimands and uncertainty intervals. It includes borderline and rare decisions, and distinguishes a stress-test sample from the operational population.

Contingency tables and category-specific agreement quantify nominal actions; weighted analyses quantify ordered class differences under declared weights; continuous agreement analyses identify numerical discrepancies and systematic offsets; and crossed designs can separate stable reviewer differences from case-dependent and occasion-dependent variability under stated assumptions. Rare-class prevalence, case heterogeneity, selection, missingness and authority can make superficially comparable coefficients represent different questions.

Calibration and training should be directed to sources of material disagreement, evaluated independently on new cases, and linked to predictive, economic and procedural outcomes. Professional and supervisory standards supply context for risk classification and control obligations, while local law determines the permissible use of particular factors and methods. Consistency supports fairness when the rule and its application are substantively justified.

The authorial synthesis establishes a practical research and governance conclusion: an insurer learns more from understanding the origin, legitimacy and consequence of disagreements than from pursuing a single measure of identical judgment. The long-term objective is an underwriting institution that consistently identifies relevant differences, treats equivalent circumstances in equivalent ways, corrects unjustified variation and retains accountable expert judgment where the evidence warrants it.

References

  • Actuarial Standards Board. (2005, revised 2011). Actuarial Standard of Practice No. 12: Risk Classification (for All Practice Areas), section 3.2.1, as displayed in the official standard. Official or published source

  • Bland, J. M., & Altman, D. G. (1986). Statistical methods for assessing agreement between two methods of clinical measurement. The Lancet, 1(8476), 307–310. DOI: 10.1016/S0140-6736(86)90837-8. Source record

  • Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1), 37–46. DOI: 10.1177/001316446002000104. Source record

  • Erratum to “A Guideline of Selecting and Reporting Intraclass Correlation Coefficients for Reliability Research” [J Chiropr Med 2016;15(2):155-163]. (2017). Journal of Chiropractic Medicine, 16(4), 346. DOI: 10.1016/j.jcm.2017.10.001. Source record

  • Gwet, K. L. (2008). Computing inter-rater reliability and its variance in the presence of high agreement. British Journal of Mathematical and Statistical Psychology, 61(Pt 1), 29–48. DOI: 10.1348/000711006X126600. Source record

  • International Association of Insurance Supervisors. (2024). Insurance Core Principles and Common Framework for the Supervision of Internationally Active Insurance Groups (ICPs and ComFrame), overview and December 2024 update. Official or published source

  • Kahneman, D., & Frueh, S. (2022, Spring). Try to Design an Approach to Making a Judgment; Don’t Just Go Into It Trusting Your Intuition. Issues in Science and Technology, 38(3). Official or published source

  • Koo, T. K., & Li, M. Y. (2016). A Guideline of Selecting and Reporting Intraclass Correlation Coefficients for Reliability Research. Journal of Chiropractic Medicine, 15(2), 155–163. DOI: 10.1016/j.jcm.2016.02.012. Source record

  • Kottner, J., Audigé, L., Brorson, S., Donner, A., Gajewski, B. J., Hróbjartsson, A., Roberts, C., Shoukri, M., & Streiner, D. L. (2011). Guidelines for Reporting Reliability and Agreement Studies (GRRAS) were proposed. International Journal of Nursing Studies, 48(6), 661–671. DOI: 10.1016/j.ijnurstu.2011.01.016. Source record

  • Kunreuther, H., Meszaros, J., Hogarth, R. M., & Spranca, M. (1995). Ambiguity and underwriter decision processes. Journal of Economic Behavior & Organization, 26(3), 337–352. DOI: 10.1016/0167-2681(94)00041-C. Source record

  • Lakkaraju, H., Kleinberg, J., Leskovec, J., Ludwig, J., & Mullainathan, S. (2017). The selective labels problem: Evaluating algorithmic predictions in the presence of unobservables. Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 275–284. DOI: 10.1145/3097983.3098066. Source record

  • Lloyd's. (n.d.). Principle 1: Underwriting Profitability. Principles for Doing Business at Lloyd's. Official or published source

  • Lo, J., Patel, N., & Calder, A. (2015). Judgmental Topics In P&C Companies: Findings from a Prediction Survey. Variance, 9(1), 101–113. DOI: 10.66573/001c.120853. Source record

  • National Association of Insurance Commissioners. (2026, April 7). Accelerated Underwriting. Official or published source

  • New York State Department of Financial Services. (2024, July 11). Insurance Circular Letter No. 7: Use of Artificial Intelligence Systems and External Consumer Data and Information Sources in Insurance Underwriting and Pricing. Official or published source

  • Pui, A., & Feddersen, M. (2022, November 3). So, You Think You Can Underwrite? Actuaries Digital, Actuaries Institute. Official or published source

  • Xu, J. (2020). Dating Death: An Empirical Comparison of Medical Underwriters in the U.S. Life Settlements Market. North American Actuarial Journal, 24(1), 36–56. DOI: 10.1080/10920277.2019.1585881. Source record

  • Zapf, A., Castell, S., Morawietz, L., & Karch, A. (2016). Measuring inter-rater reliability for nominal data – which coefficients and confidence intervals are appropriate? BMC Medical Research Methodology, 16, 93. DOI: 10.1186/s12874-016-0200-9. Source record