Insurance by BogdanovaThe Encyclopedia of Insurance and Risk Science

RESEARCH DOSSIER

Insurance Risk Expert Judgment and Qualitative Assessment

Abstract. Insurance risk decisions routinely concern exposures that are materially consequential before they become statistically estimable. Unobserved cyber accumulation, changing legal interpretations, operational control failures, climate-transition interactions, claims emergence and concentrated dependencies all require judgments about mechanisms, plausible outcomes, uncertainty and action. This monograph examines expert judgment as an evidentiary and organizational component of insurance enterprise risk management. It distinguishes risk identification from probabilistic estimation, qualitative classification from measurement, agreement from accuracy, and expert advice from accountable acceptance of risk. It critically synthesizes international supervisory requirements, risk-assessment standards and methodological research on elicitation, combining expert distributions, risk matrices, inter-rater reliability and retrospective forecast verification. A fully hypothetical insurance assessment is used to demonstrate how ordinal rankings can conceal differences in loss magnitude, how scenario-specific probability statements can be audited, and how changes in information affect decisions. The central finding is that the defensibility of an expert assessment depends on the traceable relationship among a precisely stated question, relevant expertise, evidence provenance, elicitation procedure, uncertainty expression, decision authority and subsequent testing. Qualitative methods have a legitimate, enduring role in insurance when their inferential limits are observed and material uncertainty is explicitly carried into governance. No new theory or empirical insurer data are claimed. Comparative conclusions are bounded by the cited jurisdictions, documents and methodological literature.

Keywords: Insurance risk; expert judgment; expert elicitation; qualitative assessment; enterprise risk management; insurance solvency; ORSA; risk matrices; ordinal scales; calibration; model uncertainty; inter-rater agreement; decision governance; retrospective validation; emerging risks; risk appetite; insurance regulation.

Introduction

The insurance enterprise is organized around promises that extend through uncertain future states. Prices, reserves and capital estimates generally depend on models and historical observations; institutional judgments determine whether exposures have been identified, whether past data are representative, whether models cover material risks, which scenarios merit attention and what action is warranted when the evidence is incomplete. In this setting, expert judgment is a substantive input to the assessment of risk, and its methodological quality can influence underwriting commitments, reinsurance selection, claims preparedness, liquidity planning and the security of policyholder obligations.

The IAIS insurance supervisory framework locates enterprise risk management within the integration of risk identification, measurement, reporting, appetite, capital and the own risk and solvency assessment (IAIS, 2024, ICPs 8 and 16). In the European Union, Solvency II Article 45 requires an undertaking's own assessment of its overall solvency needs and material departures from assumptions underlying the regulatory capital calculation. These instruments identify why judgments about exposures beyond established models can be materially important. They do not supply a universally valid recipe for converting a committee's verbal risk descriptions into probabilities.

This research addresses three questions. First, which types of expert assertion are being made when an insurer describes a risk as material, probable, severe, emerging or controlled? Second, under which processes can expert assessments become sufficiently reproducible, challengeable and useful for an insurer's enterprise-level decisions? Third, what evidence supports their eventual validity, especially when outcomes are rare, observations are delayed and the exposure itself changes?

The method is a source-grounded critical synthesis. The normative evidence consists of international principles, enforceable or supervisory jurisdiction-specific texts and voluntary standards, distinguished by legal force. Methodological evidence includes primary research on probabilistic elicitation, pooling, risk-matrix limitations, coder reliability and predictive scoring. Comparative application is analytical: the manuscript does not claim a representative insurer survey, regulatory inspection, commissioned interview, or observed improvement in solvency. Numerical illustrations below are invented solely to expose logical and decision properties. The research cutoff is November 14, 2026; changes after that date are outside the assessment. Insurance law and regulation must always be applied to the correct legal entity and temporal version.

The argument follows the six-part research architecture approved for this dossier. The immediate concern is the quality and governance of judgments about insurance risk. Related monographs address the separate sciences of ORSA, capital, actuarial validation, underwriting and risk behavior.

Definitions, History, and Documentary Foundation

The object of judgment and its insurance setting

Expert judgment is a reasoned assessment offered by a person with relevant domain knowledge about a proposition, quantity, relationship or possible outcome for which the available evidence does not fully determine an answer. Qualitative risk assessment is the systematic description, classification or evaluation of risks through categories, narratives, causal pathways, comparative priorities or decision-relevant judgments. Structured expert elicitation is the deliberate collection of such judgments under a specified protocol designed to clarify questions, expose assumptions and reduce avoidable distortions.

These definitions separate at least five outputs that insurance committees routinely mix. A detection claim asserts that an exposure exists or is plausible; a mechanism claim concerns how it might develop; a probabilistic claim assigns likelihood to a specified event over a specified horizon; a consequence claim describes losses or other impacts conditional on an event; and a decision claim recommends acceptance, mitigation, transfer, monitoring or escalation. Each output has different grounds and tests. A plausible catastrophe scenario can be valuable even when its probability is unresolved. A highly calibrated probability forecast can be operationally inadequate if the loss definition excludes relevant liabilities. A board's determination that a risk exceeds appetite is a governance decision applied to an evidentiary assessment.

Insurance expertise is plural. Underwriters know policy terms, selection, deductibles and exposure concentrations; claims specialists understand reporting lags and legal or judicial developments; actuaries address loss distributions and reserves; risk engineers inspect physical hazards and controls; cyber specialists analyze attack mechanisms; investment and liquidity specialists assess stress transmission; and compliance specialists interpret jurisdictional duties. Each expert sees part of the system. The task is to elicit relevant knowledge while preserving ownership of the final enterprise decision.

A recurring ambiguity concerns the word risk. An insurer may mean event frequency, expected claims, tail loss, earnings volatility, solvency strain, failure to meet obligations, or some composite of these. The assessment must specify the protected objective, insured population, currency, gross or net exposure, geographic and legal scope, relevant dependencies, horizon and valuation convention. Without these parameters two plausible-looking judgments can refer to different risks altogether.

History of structured judgment and regulatory developments

Professional estimates have accompanied insurance underwriting since its commercial emergence. The distinct modern scientific contribution is the formal examination of how those estimates can be stated, elicited, compared and evaluated. Cooke's Experts in Uncertainty (1991) surveyed the foundations of elicitation, scoring and combining expert probability assessments, and described a classical performance-weighted approach. Clemen and Winkler's combining expert distributions (1999) explained both mathematical and behavioral pooling and cautioned that no combination process universally dominates.

The modern elicitation literature insists on careful framing rather than treating an unstructured interview as a quantitative instrument. The European Food Safety Authority's expert knowledge elicitation guidance (2014), developed for food and feed safety rather than insurers, documents a process from question specification and expert selection through elicitation and reporting. O'Hagan's scientific elicitation review (2019) analyzes how structured protocols, including SHELF, address cognitive biases when distributions are required. These are cross-disciplinary methodological sources whose principles require adaptation to insurance loss definitions, incentives and exposure dependence.

The supervisory trajectory has similarly broadened insurance risk assessment. International ICPs and ComFrame provide a global reference for supervisors, whereas national regimes define binding obligations. The EIOPA ORSA guidelines were originally published in 2015, and their documentation reflects subsequent file and consultation updates; a landing-page file date cannot automatically be treated as the date of a new substantive legal rule. In Australia, APRA CPS 220, in force from July 1, 2019, requires an integrated risk management framework covering material risks and specifies ultimate board responsibility. The United States' NAIC ORSA materials describe group-level risk assessment practices through state-regulatory implementation rather than a single uniform national insurance statute.

An analytical consequence follows from this history: the vocabulary of enterprise risk assessment has become more integrated, but methodological practices still vary because evidence, institutions, lines of business and supervisory duties differ. An insurer's risk register should therefore identify the provenance and status of a judgment, rather than equating the existence of a governance form with verified predictive performance.

Documentary hierarchy and international comparison

The IAIS principles express standards and guidance for supervisors and insurers through their supervisory context. ICP 16's ERM requirements treat risk identification and interdependencies as components of ongoing enterprise assessment. ISO 31000:2018 is international risk-management guidance, confirmed as current in its publisher's record as of the research cutoff, and is not a certifiable insurer-specific prudential regulation. IEC 31010:2019 supplies guidance on selecting assessment techniques, including planning, verification and validation. The distinction between a voluntary general standard, supervisory expectation and legally enforceable rule is indispensable to any comparative finding.

The EU provides a specific legal illustration. Delegated Regulation Article 306 requires ORSA reporting of qualitative and quantitative results, methods and assumptions. Delegated Regulation Article 233 includes qualitative indicators for assessing whether an internal model covers material quantifiable risks in its scope. Qualitative judgments thus participate in an enforceable reporting and model-coverage context, although the precise duties must be interpreted with the operative legislation, entity type and competent authority's requirements.

By comparison, APRA's general insurance risk guidance illustrates supervisory guidance that accounts for institutional size, complexity and risk profile. A similar word such as “material” in one jurisdiction should not be presumed to implement the same numerical threshold or delegated authority as in another. The present study accordingly compares roles and analytical requirements, not an invented universal rulebook.

The evidentiary status of an assessment

A usable insurance judgment has several separate quality dimensions: relevance to a defined decision; traceability to records or domain reasoning; coherence with probability and accounting conventions when applicable; replicability under the stated procedure; independence from improper incentives; and performance against later evidence where outcomes can be tested. None of these dimensions can substitute for the others. Board approval supplies authority, not empirical accuracy. A high agreement coefficient supplies evidence of shared coding, not of correct causal knowledge. A numerical score supplies an operational ordering only to the extent that the scale and aggregation rule justify it.

This separation supplies the foundation for the later chapters. The unit of scientific examination is the documented assessment-and-decision episode: what was known, what was judged, what procedure was used, what uncertainty remained, who decided, and what later observations could change the conclusion.

Expert Elicitation Approaches

Question formulation and expert selection

Every elicitation begins by specifying the object with more precision than a generic “rate the risk.” A useful question identifies the conditioning scenario, time interval, population, measurement unit and the information available at the assessment date. Compare “How likely is a major cyber event?” with “Given the insurer's policy forms and exposure inventory at the assessment date, what is the probability that aggregate gross cyber-related insured claims from a specified common-vendor outage exceed USD 100 million during the next twelve months?” The second question defines an observable event, though an operational definition of “related” and the portfolio boundary are still required.

The SHELF elicitation resources provide structured documents and software for obtaining distributions from groups under limited data. Applying such practice to insurance requires careful separation of exposure evidence from experts' personal experience. The panel should represent distinct causal knowledge rather than merely distinct job titles. A concentration risk question may require exposure aggregation, contract interpretation, IT dependencies, operations, scenario modeling and reinsurance recovery expertise. Those perspectives will not have equal familiarity with every parameter.

Selection criteria should be documented before participants see the answer sought by senior management. Domain qualifications, direct data access, past forecasting experience where available, conflicts, dependence on shared evidence and institutional incentives should be recorded. Experts should be allowed to abstain from questions beyond their domain; forced numerical answers create false precision. When a material perspective is absent, the gap belongs in the assessment rather than being hidden by averaging the available panel.

Structured individual and group elicitation

Unstructured group meetings invite dominance, anchoring and premature convergence. A defensible protocol begins with individual judgments before discussion, records each rationale and uncertainty range, presents relevant common evidence and allows participants to revise their assessments with visible reasons. An independent facilitator distinguishes clarification from advocacy. The final record preserves persistent disagreements, dissenting scenarios and the evidence on which revisions depend.

The EFSA guidance on expert elicitation lays out a reproducible lifecycle, while O'Hagan's 2019 analysis explains why distribution elicitation requires a protocol sensitive to cognitive heuristics. The distinction between structured group discussion and a Delphi-style anonymous iterative questionnaire is important. The first can expose and reconcile technical reasoning; the latter can reduce certain interpersonal pressures but may also conceal the sources of genuine disagreement. Neither should impose consensus as an assumed scientific outcome.

Probability elicitation can ask experts for specified quantiles, median and plausible extreme values rather than immediately requesting a complete parametric distribution. It should then check whether these summaries are mutually coherent and whether fitted distributions represent intended tail behavior. For example, a 90th percentile is not a “90% worst-case loss”: it is a threshold exceeded with 10% probability under the elicited distribution. In catastrophe insurance the distinction is consequential because tail losses, attachment points and liquidity stress may dominate decisions.

A good elicitation report gives the exact question, assumptions, participating roles, chronology of independent and deliberative stages, elicited points or intervals, resulting distributions or categories, competing narratives and sensitivity of the final decision to alternative credible judgments. It documents whether the panel addressed parameter uncertainty, model structure or ambiguity about the event definition. Documentation of these alternatives is especially important when loss experience later becomes available.

Aggregation, dependence, and probability coherence

The aggregation of multiple opinions is a second scientific decision. Clemen and Winkler (1999) analyze mathematical and behavioral aggregation. For a common event with expert distributions Fi(x)F_i(x), a linear pool may be defined for illustration as

Fpool(x)=∑i=1mwiFi(x),wi≥0,∑iwi=1.F_{\mathrm{pool}}(x)=\sum_{i=1}^{m}w_i F_i(x),\qquad w_i\geq 0,\quad\sum_iw_i=1.

This definition yields a mixture of distributions, not a claim that the experts are independent or their information collectively adds up to new observations. Equal weighting expresses one transparent rule. Performance weighting requires relevant, sufficient calibration evidence and a pre-specified method. Cooke's classical model discussion gives a historically influential approach based on calibration and information. Applied without representative calibration questions, performance weights can themselves become another unverified judgment.

There is a further issue when experts rely on a shared catastrophe model, vendor dataset, media report or claims committee. Their forecasts may be statistically and epistemically dependent. Simply treating them as independent observations can overstate the information acquired. A panel of five who independently interpreted one flawed exposure extract is not five independent confirmations of the underlying portfolio. Correlated evidence should be recorded and, where material, represented through shared assumptions or alternative scenarios.

Combining marginal distributions for different losses does not establish their joint distribution. If property claims, liability claims and business-interruption recoveries respond to the same catastrophe, the dependence structure affects aggregate risk and capital. A committee may identify the causal dependence qualitatively before there is enough information to estimate a copula or aggregate tail precisely. The technically defensible response is an explicit dependence assumption, scenario sensitivity and a decision analysis that reports the uncertainty; numerical exactness should not be manufactured from a verbal correlation label.

Translating elicitation into an insurer decision

Elicited distributions or risk narratives should be tested against the decision's alternatives and constraints. Suppose a loss-control intervention is expected to reduce the chance of a significant event. The insurer must evaluate the intervention's cost, the distribution of retained losses, reinsurance and liquidity effects, the relevant time horizon and the plausible range of expert estimates. The decision may change little across credible distributions, or it may cross a risk-appetite threshold under a small alteration in the tail. Both are valuable findings.

This is where qualitative assessment contributes independently of probability elicitation. It can identify the causal pathway that links a third-party failure to several product lines; expose a contractual ambiguity that makes modeled recoveries unreliable; or record that event frequency is presently indeterminate. Those judgments influence what a decision-maker should investigate and monitor. The general NIST risk-assessment guide, written for information systems, demonstrates the wider logic of preparing, conducting and maintaining risk assessments rather than treating a score as a one-time answer.

The resulting organizational product should be a decision record containing the assessed exposure, question, knowledge owners, source documents, principal assertions, alternatives, constraints, limitations, conditional judgments and authorized action. Subsequent chapters examine the frequent mistake of translating this richer record into an unexplained color or ordinal number.

Qualitative Scoring Scales

The meaning and limits of ordinal classification

Qualitative scales are useful when the immediate question is whether management should investigate, escalate or prioritize a possible exposure under limited evidence. A scale such as low, moderate, high and critical establishes an ordered set of descriptions if evaluators share its definitions. The categories do not automatically specify the distance between adjacent levels, nor do they assign a probability distribution or monetary loss.

The IEC 31010 assessment techniques provide a general methodological setting for selecting risk-assessment techniques according to purpose, available information, decision consequences and uncertainty. Insurer implementation requires a mapping from the enterprise objective into operational definitions of likelihood and impact. An outcome may affect liquidity, own funds, the ability to meet policyholder obligations, business continuity and conduct obligations in different units and at different times. These are related but conceptually separate consequences.

The difficulty begins when an ordinal label is treated as if it were an interval or ratio measurement. If “high” is coded 4 and “moderate” 3, the difference of one represents a coding choice, not necessarily an equal increment of risk. Multiplying ordinal labels adds another assumption: that numerical category positions have meaningful ratios and that their product captures the insurer's preference over losses and chances. A monotonic renumbering can preserve all original ordering information while changing the products and, sometimes, the implied priority. A numerical ranking that depends on arbitrary category labels needs explicit decision justification.

Cox's risk matrix critique (2008) established mathematically important problems of limited resolution, ambiguous categorization and potential rank reversal. Duijm's risk matrix design review (2015) examines both weaknesses and practical measures for improving matrix construction and use. Together they support a narrower, useful conclusion: a risk matrix can structure conversation and screening, while decisions that depend on fine comparisons, capital or optimal resource allocation require information and rules beyond the color grid.

Risk matrices, thresholds, and interpretive error

A risk matrix should specify what is held constant. Frequency or probability may refer to one policy, one contract, one event across the entire group or at least one qualifying loss in a year. Severity may be insured gross loss, net retained loss, incremental claims reserve, maximum liquidity need or impaired own funds. A matrix crossing a policy-level occurrence rate with group-level impact produces a category that lacks a coherent interpretation.

Consider a three-category probability scale over one year: Rare is below 5%; Possible is from 5% through 20%; Likely is above 20%. Suppose the impact categories for a single gross insured-loss event are Minor below USD 10 million, Major from USD 10 million through USD 100 million, and Extreme above USD 100 million. The category edges are hypothetical conventions, not industry standards. They require definitions for events exactly at the bounds, multiple occurrences, currency, changing exposure and net recoveries.

Now compare two mutually exclusive stylized event definitions assessed one at a time. Exposure A has a probability of 5% and a conditional loss of USD 12 million; exposure B has a probability of 19% and a conditional loss of USD 90 million. Both occupy the “Possible–Major” cell. Under a simple Bernoulli fixed-severity simplification, annual expected loss is USD 0.60 million for A and USD 17.10 million for B—a factor of 28.5 difference within the same color. That is a demonstration of category compression, not a real insurer estimate. No claim is made that expected loss alone determines which risk is more important; liquidity, tails and policyholder impacts may alter the decision.

A second example exposes the importance of deciding what the product score is supposed to mean. An event with annual probability 3% and loss USD 120 million has a simplified expected loss of USD 3.6 million. Another event with probability 12% and loss USD 20 million has expected loss USD 2.4 million. Numbering the three likelihood and impact categories 1–3 gives them scores of 1×3 = 3 and 2×2 = 4. The matrix ranks the second event higher although expected loss is higher for the first. A cautious insurer might nonetheless place greater priority on either event given its capital constraints; the mathematical lesson is that the implicit rule of a category-product matrix is not equivalent to expected financial loss.

In both cases, uncertainty about the inputs should be visible. If the 19% estimate might reasonably fall anywhere from 6% to 30%, its location may cross an ordinal boundary. Reporting only the initial category hides that fragility. The relevant report states the scenario, uncertainty and sensitivity of the resulting decision, then explains any risk appetite criterion used to determine escalation.

Mapping severity, time horizon, and risk appetite

An insurer's severity measure should align with the decision. For underwriting portfolio selection, gross and net insured losses may matter separately. For claims preparedness, operational surge capacity and settlement delays may be critical. For enterprise solvency, future own funds and loss absorption matter. For group liquidity, collateral calls and reinsurance payment timing may dominate the short-run profile. A “severe” classification is therefore always relative to an explicitly protected capacity or obligation.

Supervisory materials explain the setting in which such distinctions become material. The IAIS ERM framework connects risk limits, appetite, capital and material exposures. In the EU, the ORSA supervisory report rule includes methods and assumptions as well as qualitative and quantitative results. Neither text converts a board-approved color threshold into an invariant financial measurement.

The time horizon must be stated. One-year insured event frequency is different from cumulative occurrence probability over a ten-year period; renewal behavior, exposure evolution and dependence complicate any mechanical extrapolation. A climate scenario in 2040 or a latent liability emerging after many years may be sufficiently decision-relevant for product design and asset allocation even if it does not fit an annual matrix. EIOPA's climate scenario guidance for ORSA illustrates the supervisory importance of forward-looking materiality analysis.

A practical qualitative scale can remain in the enterprise risk register while carrying a companion narrative explaining the event definition, potential loss pathway, gross-to-net qualification, uncertainty confidence, evidence age, controls, owners and escalation criterion. Management can then see whether the label represents a measured probability, an expert-suggested range, a plausible stress case, or an issue requiring evidence gathering. It should never be assumed that identical colors make distinct risk types economically equivalent.

Hypothetical insurance assessment

The following case is intentionally hypothetical and contains no observed firm data. A multiline insurer identifies a shared third-party claims-processing platform as a possible source of correlated interruption, service delays, litigation and cyber-related claim emergence. The asset inventory is incomplete; policy wording may create contingent business interruption liability; historical company-wide loss counts are zero; and the cloud provider's incident frequencies are only partially comparable. The decision is whether to prioritize an independent dependency audit and a recovery capability program during the next planning cycle.

A panel comprising the chief risk function, senior claims operations, cyber underwriting, legal counsel and a reliability engineer reviews three explicit pathways: an outage with operational loss alone; an outage with insurer contractual liability; and an outage combined with a correlated external cyber event. The assessor first records what each expert knows, which source is shared and which question is beyond a participant's competence. A conditional event definition is fixed for the next twelve months. Experts give separate probability ranges for the pathways, along with gross, net and liquidity consequences. Under the illustrative decision analysis, the board's appetite concerns the combined event's tail and service obligations, not a single color.

The initial qualitative label “High operational risk” is insufficient because it obscures correlation with claims servicing and legal obligations. The analytical output is a causal diagram, source register, independent uncertainty judgments, an explicitly conditional set of scenarios and a comparison of mitigation options. If a policy-wording review narrows coverage, probability and severity questions must be revised consistently; a reduction of expected insured loss does not automatically remove operational or conduct exposure. The outcome is a structured decision proposal with open assumptions, not a fabricated percentage.

This example demonstrates a legitimate use of qualitative assessment: to define the risk correctly, allocate investigative attention and formulate hypotheses for later quantitative modeling. It also shows why an unexplained composite score could misdirect capital and control resources.

Agreement and Governance

Agreement, expertise, and substantive dissent

The convergence of expert assessments may support procedural consistency, yet the meanings of agreement and validity are different. Experts can agree because they share sound evidence, because they all used the same mistaken input, or because organizational incentives suppressed disagreement. Agreement needs an explicit measurement design before it is interpreted.

For categorical judgments, simple percentage agreement is often misleading because it ignores the marginal frequencies of categories. Cohen's chance-corrected kappa coefficient (1960) formalized the issue. A common statistic for two nominal raters is Cohen's kappa, κ=(po−pe)/(1−pe)\kappa=(p_o-p_e)/(1-p_e), where pop_o is observed agreement and pep_e is the agreement expected under a specified chance model based on the raters' marginal classifications. As a thought experiment, suppose one rater calls all ten assessments High and another calls eight High and two Low. Raw agreement is 80%, but pe=0.8p_e=0.8 and κ=0\kappa=0. This does not prove either rater is incorrect; it shows that the two summary statistics address different properties and that a dominant category can strongly affect interpretation.

If severity ratings are ordered, a weighted chance-corrected statistic or an ordinal agreement measure may be more appropriate, but weighting choices must match how different errors matter to the decision. Multiple raters, missing values, prevalence shifts and hierarchical classification require another design. Krippendorff's reliability measurement analysis (2004) explains why apparently simple agreement coefficients need careful structural assumptions. In insurance applications, report confusion matrices, marginal category frequencies and disputed cases alongside any single statistic.

Disagreement is potentially information. One expert might judge a catastrophe event less probable but assign a heavier tail, while another might accept the probability and dispute the reinsurance recovery assumption. Averaging their ordinal labels would erase the location of the dispute. A useful risk committee keeps separate the empirical disagreement, uncertainty about policy interpretation, difference in protected objective and difference in acceptable risk appetite. Each requires a different resolution process.

Governance roles and conflicts

An insurer needs to distinguish the producer of a risk judgment from the officer or body authorized to rely on it. First-line business units identify and manage exposures in their operations and underwriting decisions; risk management and actuarial functions challenge and integrate assessments as their mandate requires; compliance and internal audit have separate roles under local governance requirements; executive management decides within delegated authority; and the board establishes and oversees appetite and the enterprise framework. The exact allocation is jurisdiction-, group- and company-specific.

The IAIS risk management standards emphasize the framework's integration with risk management and controls. APRA CPS 220 explicitly places ultimate responsibility for an appropriate framework with the relevant board. Such governance requirements give an insurer a reason to preserve the original expert record, documented challenges, residual disagreements and reasons for any management override.

Conflicts of interest can arise in both directions. Underwriting and sales functions may benefit from lower estimates of loss or concentration; a risk function may prefer categorically conservative ratings that are difficult to disprove; vendor specialists may normalize their own technology risks; external advisers may have repeat-engagement incentives. Institutional bias can be reduced through independent elicitation, conflict declaration, version-controlled assumptions, dissent channels and conditional use of independent external expertise. A role or credential is evidence of qualification for specified questions, not a guarantee of neutrality.

A decisive control is separation between what is judged and what action is authorized. The chair of a risk committee should not retrospectively relabel the experts' distribution to make an approved decision appear better supported. Instead, the record should state whether the decision rests on affordability, strategic risk acceptance, policyholder protection, risk limits or a changed technical finding. A justified decision can be made under uncertainty while remaining open about the uncertainty.

Documentation, escalation, and multi-entity groups

An assessment record adequate for later review should identify: the legal entity and scope; specific risk statement; decision horizon and materiality criteria; experts and qualifications; conflicts and source dependencies; dated evidence; elicitation method; category or distribution definitions; plausible alternative outcomes; dependence assumptions; proposed responses; executive challenge; decision and conditions; responsible owner; monitoring triggers; and the date for reconsideration. Sensitive details may be appropriately access-controlled, while lineage must remain auditable to authorized reviewers.

The international comparison cannot be reduced to formal equivalence. In the United States, the NAIC ORSA framework describes key risks, significance and prospective capital needs under state adoption and filing arrangements. European Solvency II ORSA duties connect an undertaking's particular risk profile, risk tolerances and strategy to its own assessment. Australian CPS 220 requirements place material-risk coverage in a framework linked to strategic objectives and business plans. Common themes concern identification, governance and decision relevance; local rules determine exact documents, thresholds, responsibilities and supervisory consequences.

For internationally active groups, qualitative assessments must preserve the distinction between consolidated exposures and legal-entity obligations. The same event may have distinct loss sharing, regulatory capital, reinsurance recoverability and liquidity consequences in different subsidiaries. A group-wide average rating can hide an entity-specific concentration that is critical to its policyholders. Internal consistency requires a common dictionary and aggregation logic, together with explicit local deviations. Supervisors' international ComFrame reference helps orient group-level analysis but does not erase host-country regulatory requirements.

Comparative supervisory applications

The strongest contrast is between governance structures and technical questions. IAIS principles seek integrated ERM, APRA applies a prudential standard across specified regulated institutions, the NAIC materials articulate ORSA under a system of state regulatory adoption, and the EU provides an ORSA regime with specific reporting duties. These sources converge on the need to recognize material risks and make management accountable. They do not establish that every risk class should receive the same numeric probability bands, or that a single expert panel design suits every group.

Emerging exposures reveal the need for contextual methods. EIOPA's non-affirmative cyber supervisory statement identifies a specific challenge in exposures that may reside in policies without explicit cyber coverage and recommends attention to identification and measurement. In climate-related ORSA, EIOPA's scenario and materiality guidance addresses forward-looking physical and transition risk. The former can turn on policy wording and accumulation mapping; the latter extends through alternative long-term pathways. Both call for qualitative framing, but they demand different specialist knowledge, horizons and validation strategies.

A coherent international assessment practice consequently has two levels: a common evidentiary discipline that defines questions, records provenance, invites challenge and documents decisions; and regime-specific implementation addressing each insurer's legal duties, business model and risk. This is an analytical synthesis of the surveyed sources, not a new universal regulatory requirement.

Retrospective Validation

Scoring predictions and assessing outcomes

An expert assessment becomes scientifically more informative when it makes statements that can later be confronted with observations. A risk owner who forecasts a defined event probability can be evaluated for calibration and predictive discrimination; an expert who describes a causal failure mode can be evaluated for the specificity and eventual support of the mechanism; a committee that assigns a category can be evaluated for classification consistency and decision usefulness. These are distinct validation targets.

For binary events with predicted probability pip_i and realized indicator yi∈{0,1}y_i\in\{0,1\}, the Brier score is commonly represented in a binary form as

BS=1n∑i=1n(pi−yi)2.BS=\frac{1}{n}\sum_{i=1}^{n}(p_i-y_i)^2.

Lower values are better under this loss convention. Brier's original probability forecast verification (1950) is the foundational publication; the binary formula here is a specialization appropriate to a single yes/no event. Gneiting and Raftery's proper scoring rules study (2007) sets such evaluation in the wider theory of scoring probabilistic forecasts, emphasizing rules designed to reward honest probability reporting in expectation.

The target event must be defined before outcome inspection. For an insurer forecasting material cyber claims, “a major event” is too vague for a meaningful realized outcome. A verifiable event might be at least one aggregate gross loss above a specified threshold, arising from a defined cause, within a specified insurance portfolio and observation window. Claims may be reported late and reserves may develop for years. Outcome coders must therefore record the event definition, evidence-completion date and censored or unresolved assessments. Reclassifying outcomes post hoc to match a prior forecast would make the score misleading.

A deliberately small numerical demonstration is instructive. Imagine four independent-looking but actually hypothetical binary predictions p=(0.2,0.6,0.8,0.1)p=(0.2,0.6,0.8,0.1) and outcomes y=(0,1,0,0)y=(0,1,0,0). The mean squared probability error is (0.04+0.16+0.64+0.01)/4=0.2125(0.04+0.16+0.64+0.01)/4=0.2125. A constant 0.25 probability predicts the same four outcomes with Brier score 0.18750.1875. This example shows how a seemingly differentiated set of forecasts may score worse than a constant baseline in one realized sample. Four observations are far too few to establish comparative expert performance, and choosing the baseline after the outcomes would be methodologically invalid as an operational benchmark. A real assessment would pre-specify the reference forecast and report uncertainty around observed performance.

Calibration tests whether outcomes occur at approximately the frequencies suggested by stated probabilities across comparable cases. Discrimination concerns whether higher forecasts help distinguish events from non-events. A scoring rule can summarize aspects of both; it does not alone explain why a forecast fails. In portfolios with evolving exposures, clustered catastrophes and changing contractual coverage, calibration must be considered conditionally. A forecast that applied to a previous policy wording set should not be scored as if it concerned the amended portfolio.

For continuous loss forecasts, quantile coverage, proper interval scores and other distributional metrics may be more appropriate. An expert may correctly identify the median while grossly understating the tail; an expected loss can be close while the implied distribution is inadequate for capital and liquidity decisions. The Gneiting–Raftery scoring framework makes clear that the evaluation metric and forecast object must be aligned. The relevant insurance question may be the exceedance probability for a retention, the duration of a payment backlog or the aggregate financial consequences of correlated incidents.

Testing qualitative ratings and controls

Qualitative risk registers can be validated in parts even when a risk category itself is not a probability forecast. One may test whether independent raters interpret category definitions consistently; whether material incidents were present in earlier inventories; whether high-rated scenarios generate investigatory actions on time; whether risk owners document and close conditions; and whether control effectiveness claims survive inspection. Such tests use different denominators and evidence.

Agreement testing should use blinded or separately elicited classifications where possible. A before-and-after training comparison may show higher consistency of category application, but this is evidence about interpretive reliability. To assess predictive association, the study must connect ratings to later defined outcomes while accounting for the intervention that the rating itself triggered. A high-rated risk receiving stronger controls may experience fewer losses precisely because management acted; naive hindsight can misclassify a useful alarm as an inaccurate forecast.

The risk matrix limitations identified by Cox motivate direct testing of cross-category ordering and within-category heterogeneity. One can compare the matrix's ordinal priority with observed or modeled losses under a documented objective, examine how many materially different exposures fall into the same cell, and test sensitivity to thresholds. Duijm's matrix recommendations support careful design and communication. A matrix that screens efficiently may pass an administrative-purpose test while failing as a mechanism for allocating capital to maximize expected financial benefit.

A policy-compliance check has a different purpose. It verifies that prescribed assessments, challenges and escalations occurred; it cannot establish the factual correctness of subjective claims. Likewise, a file signed by a senior actuary attests to role and review, while numerical reasonableness depends on the substantive analysis. These distinctions should be reflected in the insurer's internal assurance program.

Learning under nonstationarity and sparse losses

In insurance, low observed loss frequency does not imply low exposure probability. A new product may have no credible claims history, policy language may change judicially, a climate peril may shift in frequency or severity, and cyber accumulation can emerge through common external infrastructure. Such nonstationarity limits retrospective extrapolation. A good validation program dates each forecast and archives its information set, then distinguishes calibration failure from changes in external conditions and changes in the risk definition.

Three families of tests are complementary. Outcome-based tests use sufficiently matured events or losses. Process tests use elicitation adherence, source provenance, inter-rater interpretation and recorded challenges. Mechanism tests investigate whether causal pathways and control assumptions withstand incident review, engineering inspection or adversarial scenario testing. Process compliance and mechanism plausibility are not a substitute for observed accuracy, but they support responsible decisions when sufficiently frequent outcomes do not exist.

Risk research also involves uncertainty about the credibility of the evidence. The IPCC's treatment of calibrated uncertainty offers an instructive cross-disciplinary distinction between confidence, which considers evidence and agreement, and likelihood, which concerns probability of an outcome. This climate-science convention should not be copied as an insurer's regulatory rating scale; its methodological contribution is the disciplined separation of confidence in a finding from probabilistic assertions about events. An insurer can therefore report that the mechanism is well-supported while the event probability remains weakly determined.

Learning from near misses requires similar care. A near miss can reveal a pathway or control weakness without proving a particular annual probability. Incident counts need relevant denominators: policies, insured values, exposure-years, transaction volumes or system hours. Changing detection practices can alter observed incident frequency even with stable underlying risk. Retrospective validation should state whether its object is a portfolio exposure, event, report, claim, loss or board decision.

Validation design for the hypothetical case

Return to the shared claims-platform scenario. Before any intervention the insurer archives the pre-assessment dependency inventory, contractual interpretations, probability ranges and severity scenarios. The board approves an audit and recovery program under specified cost and service obligations. Later evidence includes revised vendor architecture, tested failover duration, policy-wording review, outage incidents, restoration times, claims payment backlogs and any correlated insured claims. These observations can confirm or disconfirm particular mechanism assumptions.

A post-intervention decline in outages would not by itself identify the causal effect of the program: changes in infrastructure, incident detection, vendor behavior and exposure volume may have occurred simultaneously. A credible design would track these changes and compare like-for-like exposures where available, supported by test evidence and clearly documented alternatives. It would distinguish forecasts made before controls from forecasts conditional on their implementation.

The risk owner should report three different findings if appropriate: “The dependency model was incomplete and has been corrected”; “Risk classifications became more reproducible across the panel”; and “The probability of a defined aggregate insured loss remains weakly estimable.” These statements are compatible. The institution may have improved its decision process without possessing a validated event probability.

This is an especially important practical conclusion: retrospective assessment should preserve the original information state and the decision's causal context. Otherwise the organization rewards forecasts that were lucky, punishes mitigation that succeeded and erases valuable uncertainty from its historical record.

Comparative Synthesis and Research Conclusions

Evidence synthesis and decision implications

The combined evidence supports a structured account of what expert judgment contributes to insurance risk governance. International prudential sources require institutions to identify material risks and organize risk management, while methodological scholarship describes how judgments can be elicited, represented, pooled and tested. The value of the method depends on the relationship between those two requirements. A purely procedural risk register lacks evidentiary depth; a carefully elicited probability without a decision mandate may fail to affect exposure.

The source synthesis yields five research conclusions.

First, expert judgment is indispensable at boundaries of observed evidence, including emerging risks, coverage ambiguities, long-horizon developments and dependencies that historical portfolios have not adequately sampled. Expert judgment must be recognized as one source of information with stated domain, uncertainty and incentives. Its quality cannot be inferred from its numerical appearance.

Second, qualitative assessments are legitimate organizational representations of possible loss mechanisms, control weaknesses, uncertain materiality and escalation needs. Their classification logic must be documented and tied to a purpose. When ordinal scales are used as quantitative aggregators or capital allocation formulas, they require assumptions and validation beyond the scale labels themselves.

Third, agreement is informative about consistent interpretation when measured with suitable methods and denominators. The epistemic meaning of disagreement is often greater than the value of a group average. Deliberate preservation of expert dissent is particularly useful where participants have distinct information about coverage, causes or tail consequences.

Fourth, regulation and governance locate responsibility for identifying and managing material risk. The IAIS supervisory principles, EU ORSA guidance, APRA risk management standard and NAIC ORSA overview share the broad concern with material exposures and oversight. Their legal status, entity scope and implementation details differ. A common scientific method can inform several regimes without transforming distinct regimes into a single legal prescription.

Fifth, validation should evaluate what the judgment actually asserted. Prospective probabilities can be scored against defined outcomes with appropriate temporal and sampling controls. Categories can be checked for consistency and decision utility; mechanisms can be investigated; governance controls can be audited. These tests support different conclusions and should be reported separately.

Operational case comparisons

A property catastrophe underwriter may begin with mature event catalogs and vendor hazard models, while still requiring expert interpretation of vulnerability, exposure data, policy terms and model limitations. Here expert judgment often concerns the credibility of model inputs, tail structure and local dependencies. Testing can draw on event analyses, engineering evidence and successive model vintages, but the independence of model providers and the stability of hazard assumptions matter.

A liability insurer examining a new class of judicial claims confronts different uncertainty. Past settlements may offer weak guidance if the legal theory changes. Claims and legal specialists can identify mechanisms and plausible development periods; probabilistic estimates must specify a legal regime and time horizon. The relevant validation record includes changes in judicial outcomes and claims reporting rather than solely immediate event counts. A single annual loss-frequency band is often a poor representation of long-tail exposure.

For cyber accumulation, exposure mapping, common service providers, contractual coverage and cascading dependencies take priority. EIOPA's non-affirmative cyber risk statement provides a supervisory example of identification challenges that require more than historical incidents. A panel should separate contract interpretation from cyber-event frequency and make shared-data dependencies explicit.

For climate-related insurance risk, the scenario horizon and physical-versus-transition distinctions are central. EIOPA's climate ORSA guidance is illustrative of forward-looking practice. The IPCC uncertainty framework offers methodological language for evidence confidence and likelihood. An insurer must still translate those insights into its actual business, assets, policy obligations and applicable rules.

Across these cases, the scientific process remains consistent in its discipline but differs in object, available evidence, actors and validation horizon. This comparative finding is an original analytical integration of the cited documents; it is not an empirical measurement of insurer performance.

Limitations, research agenda, and conclusions

The present research is documentary, methodological and analytical. It provides no field experiment, confidential insurer panel, original reserve triangle, supervisory examination dataset or causal estimate of the financial effect of expert elicitation. Some methodological sources originate outside insurance and are transferred only to the extent justified by analogous problems of uncertainty and decision-making. Normative material is jurisdiction-specific and may be amended; the date and legal status of the cited version govern the proposition supported.

Research capable of refining these conclusions would include preregistered comparisons between unstructured and structured insurer judgments on the same well-defined questions, blind prospective probability forecasting with sufficiently long loss-maturation windows, studies of how review boards respond to persistent disagreements, and empirical analysis of whether ordinal categories improve action timing after controlling for exposure severity and mitigation. Cross-jurisdictional comparisons should keep entity size, insurance line, applicable prudential regime and supervisory reporting duties separate. Research designs should report negative and null findings rather than only examples of apparent expert success.

A further priority is the measurement of shared information dependence. Experts often use the same vendor catastrophe model, data feed, legal memorandum or portfolio extract. Without documenting this common base, apparently diverse panel results can overstate evidence breadth. Likewise, the effect of interventions triggered by an assessment should be modeled when testing subsequent outcomes; otherwise an effective warning may be scored as unnecessary.

The principal scientific conclusion is precise. Expert risk judgment is most defensible when it is treated as a traceable, revisable evidentiary process linked to specified insurance decisions. The insurer should state the proposition, explain the basis of the estimate, distinguish categories from probabilities, expose dependence and unresolved uncertainty, document who may accept the risk, and compare later evidence with what was actually claimed. This conclusion integrates established methodological work and supervisory materials and does not establish a new theory.

Conclusion

Insurance enterprise risk assessment includes hazards whose significance becomes apparent before their statistical frequency or financial magnitude can be reliably measured. Structured expert judgment gives institutions a disciplined means of identifying mechanisms, specifying uncertainties and prioritizing investigation and action. Its evidentiary value is strengthened by independent elicitation, explicit event definitions, transparent aggregation, preserved disagreement, decision-specific severity conventions and controlled retrospective tests.

The methodological evidence establishes recurring limits of unaided subjective estimates, ordinal risk matrices and simple consensus measures. The supervisory evidence places responsibility for material-risk identification, oversight and reporting within insurer governance, subject to the applicable jurisdiction. Together these sources support a practical scientific standard of reasoning: record what is being judged, the evidence and expertise that support it, the scale on which it is expressed, the decision to which it applies and the future observations that could change it.

A qualitative assessment can be a scientifically valuable result when it makes the scope of uncertainty and causal exposure clearer. A probability estimate can be valuable when its reference class, conditions and calibration are specified. A board decision can be justified when the relation between accepted risk, strategic objectives and policyholder obligations is explicit. Institutional learning arises when all three records remain distinguishable and mutually traceable.

References