Insurance by BogdanovaThe Encyclopedia of Insurance and Risk Science

RESEARCH DOSSIER

Insurance Market Conduct Risk Scoring and Supervisory Prioritization

Abstract. Insurance market conduct supervision allocates scarce investigative and remedial capacity across practices that may compromise the interests of policyholders, beneficiaries, and other customers. Risk scoring translates heterogeneous evidence into comparable assessments, yet the validity of these assessments depends on the unit of analysis, the exposure denominator, the observation process, and the economic and legal meaning of the predicted outcome. This monograph examines the organization of market conduct risks, indicator construction, comparative benchmarking, probabilistic calibration, uncertainty, supervisory resource allocation, and independent evaluation. International standards, European product governance and value-for-money benchmarking, United States market regulation and Market Conduct Annual Statement reporting, and United Kingdom consumer outcomes monitoring establish complementary institutional perspectives. Seven reproducible synthetic examples demonstrate how harm severity, portfolio composition, weights, sparse observations, rare outcomes, resource constraints, and evaluation design change priorities. The central analytical result is that ranking evidence of possible harm, estimating the probability and burden of harm, and selecting the most valuable lawful intervention are distinct operations. A defensible system preserves these distinctions, combines quantitative signals with documented judgment, distinguishes observation from adjudication, and evaluates impact against independent evidence. The study establishes a methodological synthesis rather than an empirically fitted universal rating or a new regulatory doctrine.

Keywords: Insurance market conduct; risk-based supervision; policyholder harm; supervisory prioritization; market conduct indicators; risk scoring; exposure adjustment; calibration; inspection allocation; consumer protection; regulatory evaluation.

Introduction

The economic function of insurance depends on the correspondence between contractual protection and the protection actually experienced by customers. Products are frequently purchased before their performance can be observed, while claim settlement, cancellation, renewal, and dispute resolution may take place years after the initial transaction. An insurer therefore possesses information and operational control that the policyholder cannot readily replicate. Market conduct supervision addresses this asymmetry through examination of the design, distribution, administration, and performance of insurance arrangements within the applicable legal framework.

The supervisory authority must decide where to direct finite analytical, investigative, and remedial resources. Treating every firm, product, and transaction identically would ignore differences in exposure, vulnerability, potential severity, and available evidence. Prioritizing only the largest firms would leave smaller but severe concentrations of harm unexamined. Prioritizing only the most visible complaints would reward visibility rather than reveal the underlying distribution of harm. Risk-based supervision addresses this allocation problem while preserving the obligations attached to the legal mandate.

The international foundation is provided by the International Association of Insurance Supervisors (IAIS, 2024), whose Insurance Core Principles connect ongoing supervisory review under ICP 9 with conduct of business under ICP 19. Off-site monitoring and on-site examination form complementary means of understanding supervised entities and their activities. Fair treatment concerns the life cycle of insurance business and the respective responsibilities of insurers and intermediaries. The principles establish an international supervisory standard; each jurisdiction determines the enforceable legal powers and requirements through its own framework.

This study asks how observed conduct indicators can inform comparative assessments; when those assessments warrant probability interpretation; and how supervisors should convert the assessments into action without confusing statistical association, confirmed misconduct, and causal improvement. Its evidence base consists of official standards, regulatory reporting and methodological publications, and two peer-reviewed studies of predictive evaluation. The comparison is purposive, not a systematic review of every national regime. The mathematical examples are constructed, not estimates from confidential insurance records. The conclusions concern the logic and limits of risk measurement and decision-making, rather than actual scores of particular firms.

Insurance Conduct Risk Taxonomy and Supervisory Mandates

Conduct risk as a multidimensional object

Conduct risk concerns the potential for insurance business practices to generate adverse customer outcomes through unsuitable product characteristics, distribution behavior, administration, claims handling, or other interactions within the applicable conduct mandate. Actual customer detriment can take the form of financial loss, exclusion from needed protection, delayed access to benefits, impaired comprehension, or burdensome resolution processes. A risk assessment must specify which of these outcomes it is intended to identify.

The product-design dimension includes coverage, exclusions, target market, pricing structure, benefit complexity, and distribution arrangements. A contract can be administered accurately while still presenting a serious product-level suitability concern. The distribution dimension includes information, advice where applicable, incentives, sales practices, and treatment of customers with limited capacity to understand or compare offers. Servicing concerns changes to contracts, premiums, cancellation, renewal, accessibility, and communications. Claims concerns the assessment of entitlement, timeliness, fairness, and payment. Complaint handling and remediation extend across all dimensions because they reveal and sometimes compound the original problem.

Some harms are diffuse, repetitive, and individually small. Others are concentrated, severe, and comparatively infrequent. An unsuitable add-on sold to many customers can create substantial aggregate loss despite a low individual amount. An unjustified refusal of a large benefit may impose exceptional harm on a single household. Vulnerability, urgency, and the reversibility of injury can modify the significance of an otherwise identical monetary loss.

Risk therefore has several coordinates: probability, exposed population, severity, persistence, distribution across customer groups, and possibility of correction. A single aggregate score compresses these coordinates for an administrative purpose. The coordinates themselves should remain available for supervisory reasoning, since the justification for intervention may depend on one dimension that aggregation obscures.

Inherent risk, controls, and outcomes

The World Bank's (2022) risk-based supervision guidance distinguishes market-wide and individual provider assessment and discusses inherent risk, impact, controls, residual risk, and supervisory judgment. The distinction is useful for insurance because business model and inherent exposure differ from the quality of the mechanisms intended to manage conduct risk.

Inherent risk is the potential burden associated with activities, products, and customers before accounting for the effectiveness of relevant safeguards. Control evidence concerns governance, distribution oversight, product review, quality assurance, complaints response, and other preventive or corrective arrangements. Residual risk concerns the remaining exposure after the operation of controls has been evaluated. These elements are analytically related but are measured through different observations.

An attractive procedure manual establishes the existence of a documented design. It does not establish consistent execution. A high rate of staff training establishes participation according to the recorded definition. It does not establish customer comprehension or fair claims outcomes. Conversely, a poor outcome can occur despite well-designed controls because of an unexpected external event, data error, or legitimate contractual distinction. Supervisors must assess controls and outcomes together while differentiating causal explanations.

An ordinal rating of controls should not be subtracted mechanically from an ordinal rating of inherent risk and presented as a measured probability. Arithmetic on such ratings is an explicit convention for prioritization; its scale and anchors need to be defined. If numerical probabilities are intended, the relationship between controls, exposure, and subsequent observed harm must be estimated and validated against appropriate labels.

Supervisory mandate and decision boundary

The relevant classification is legal as well as statistical. Conduct duties, prudential obligations, and contractual disputes can intersect, but the authority's competence determines which action is available. A solvency concern may affect claims payment. An unfair sales practice may expose the insurer to remediation liabilities. These intersections justify information exchange and coordinated assessments without collapsing prudential and conduct objectives into one undifferentiated rating.

A conduct risk indicator supports a decision to obtain information, examine a practice, or intensify oversight. A finding of legal violation requires the applicable legal criteria, reliable facts, and the prescribed procedure. A predictive score cannot establish the elements of a violation by itself. Inspectors should retain the evidence that motivated selection separately from evidence obtained during investigation and from the legal analysis of any finding.

This boundary protects the meaning of the score. A firm placed in a high-priority band may have serious confirmed problems, a combination of alarming signals, a vulnerable customer population, or an information deficit. These are different reasons for additional attention. Suppressing the distinction increases the risk of unfair reputational inference and makes evaluation of the prioritization process difficult.

Units of Supervision and Harm Denominators

The unit precedes the metric

A supervisory population may consist of legal entities, insurance groups, products, distribution channels, claims administrators, geographical markets, or transactions. Each unit provides a different picture. Firm-level scoring supports allocation across entities. Product-level scoring identifies problematic offerings within an otherwise acceptable firm. Channel-level scoring can isolate practices shared by several insurers using the same distribution network. Customer-level and transaction-level data can reveal disparate experiences hidden by aggregates.

The choice of unit should follow the question and the legal power. When a sales practice spans multiple insurers, the channel may be the appropriate analytical unit even though legal responsibility must be determined separately for each participant. When a particular product option creates poor value, an entity-wide average is insufficient. Group-level consolidation is useful for coordinated assessment, but local differences in law, contract definitions, and distribution arrangements must remain visible.

Denominators establish the population that had the opportunity to experience the recorded event. Complaints per policy-year, complaints per claim, complaints per new sale, and complaints per customer answer different questions. A claim acceptance rate uses an eligible claims population under the reporting definition. A cancellation rate may use policies in force or newly written business depending on the event and period. Mixing these measures makes comparisons unreliable.

An operational data dictionary should establish unique identifiers, period boundaries, inclusion and exclusion criteria, treatment of reopenings, duplicates, cancellations, zero denominators, missing fields, currency and coverage differences, and the relationship between submitted and corrected records. The authority should preserve a record of every definitional change. Longitudinal movement after a reporting revision cannot be presumed to reflect a change in actual conduct.

Frequency and severity

Consider two synthetic books of 10,000 policy-years each. Book A experiences 100 confirmed improper charges of USD 30; book B experiences five confirmed improper benefit refusals of USD 20,000. The event frequencies are 10 and 0.5 per thousand policy-years. Total recorded monetary harm is USD 3,000 and USD 100,000, respectively. Event frequency ranks A first; monetary harm ranks B first.

This example does not establish a universal valuation of misconduct. The economic burden of delay, stress, exclusion, or damage to vulnerable customers may be poorly represented by nominal compensation amounts. Monetary estimates also depend on the counterfactual entitlement and the completeness of identification. The example demonstrates why frequency and severity should be retained as separate fields and why an aggregate harm measure requires an explicit definition.

A count of incidents and a count of affected customers need not coincide. Repeated improper charges to the same person may create several events but one affected customer. A single systematic product defect may affect thousands of customers. Both the event process and the customer burden matter; they should not be conflated by a denominator chosen for administrative convenience.

Complaints, claims, and observation

A complaint is an observed expression of dissatisfaction under the relevant definition, not a representative draw from all harmful events. Knowledge of rights, ease of reporting, language, time, intermediary assistance, and perceived prospects of redress influence whether customers complain. A low complaint rate may reflect satisfactory conduct, limited exposure, restricted accessibility, or low complaint propensity. A high rate may reflect poor conduct or effective identification and reporting.

The Financial Ombudsman Service's (2026) annual complaints data illustrate the importance of distinguishing incoming cases from the outcomes of resolved cases and recognizing changes in the composition and mechanism of referrals. An uphold rate concerns complaints resolved within the relevant classification, rather than the population of all policyholders or all incoming cases during the same reporting interval. Changes in referral arrangements can influence the observed caseload.

Claims metrics share this observation problem. A low claim frequency can be compatible with low incidence of insured events, restrictive coverage, limited claim awareness, or customers abandoning attempted claims. A high acceptance rate can coexist with adverse outcomes if claim initiation is difficult or settlements are inadequate. It is necessary to examine the full sequence from exposure and event to attempted notification, registered claim, decision, payment, reopening, complaint, and remediation. Each stage has its own denominator.

The FCA's (2021, updated 2026) general insurance value measures report defined claim frequency, claim acceptance, average claim payouts, and claims complaints measures under a specific reporting framework. These fields are complementary. The data definitions and scope restrictions are part of the evidence, not incidental technical details. A supervisor using them should preserve that meaning rather than construct a composite from superficially similar measures collected under other regimes.

Case-mix adjustment

A crude ratio can invert comparisons when the composition of business differs. Suppose firms A and B each have 1,000 synthetic claims. A has two adverse outcomes among 100 simple claims and 45 among 900 complex claims. B has 27 among 900 simple claims and six among 100 complex claims. A's adverse rate is lower in both strata: 2% versus 3% for simple claims and 5% versus 6% for complex claims. Yet its aggregate adverse rate is 4.7%, compared with 3.3% for B.

Direct standardization to a common 50:50 mixture gives A a rate of 3.5% and B 4.5%. The crude rate describes the actual book under the assumed data. The standardized rate describes performance under a reference composition. Both are legitimate and answer different questions. An authority interested in current total harm may retain the crude result; an authority investigating differential handling quality may emphasize the standardized comparison.

Adjustment variables should correspond to legitimate differences in exposure or observation. Adjusting for a factor produced by the questioned practice, such as whether a customer successfully registered a claim after encountering barriers, can eliminate part of the very outcome of interest. Adjusting away vulnerability can similarly obscure disparate harm. Supervisors should document why each factor enters the standardization and publish or retain unadjusted group results where lawful and meaningful.

The reference population must remain stable or its revisions clearly documented. A score based on a changing peer distribution can move even if the entity's own conduct remains constant. Absolute thresholds, time trends, and relative benchmarks can usefully coexist.

Scoring Architecture and Indicator Weighting

The layered assessment

A defensible scoring architecture begins with validated observations and preserves the path from those observations to the final priority. The first layer contains raw counts, denominators, definitions, dates, and provenance. The second contains normalized indicators and estimates of measurement reliability. The third contains dimension-level judgments or predictions. The fourth combines these assessments for a clearly specified supervisory purpose. The final decision layer considers legal urgency, investigative cost, expected value of action, and documented discretion.

Normalization may take the form of rates, standardization to a reference portfolio, percentile rankings, deviations from policy thresholds, or model residuals. A percentile is a relative position, not a probability. A standardized deviation from a peer mean measures departure under chosen assumptions; it does not establish customer detriment. A threshold based on a legal obligation has a different meaning from a statistical outlier. Preserving the transformation is essential to interpreting the number.

Directionality also matters. Higher complaint frequency can indicate potential concern, whereas higher legitimate claim acceptance can indicate favorable experience. The direction depends on the metric definition and the underlying process. A score should not transform every variable using an assumed monotonic relationship without checking whether that relationship makes sense across the relevant range.

Weighted composites and sensitivity

A conventional weighted composite for aligned indicators is S = sum(w_j x_j), with nonnegative weights summing to one and all x_j measured on a specified common scale. It is an administrative index unless an empirical mapping establishes an outcome interpretation. Weights reflect the relevance assigned to dimensions, along with scale and correlation choices.

Take three synthetic entities with scores between 0 and 100 for harm, exposure, and control weakness.

Entity Harm Exposure Control weakness Equal weights Harm-led weights
A 80 20 50 50 62
B 40 90 50 60 52
C 55 55 55 55 55

The equal-weight index orders B, C, A. With weights 0.6, 0.2, and 0.2, the order becomes A, C, B. Neither ordering is a property of the firms independent of the chosen policy. The first gives equal standing to the three dimensions; the second emphasizes the harm measure.

Weight selection should involve the legal mandate, the purpose of the ranking, measurement reliability, and sensitivity to plausible alternatives. A correlation analysis can identify redundancy. Claims complaints, total complaints, and complaint-related remediation may all partly reflect the same underlying process. Counting them independently at full weights can magnify one dimension without an explicit policy decision. Reliability weighting can reduce noisy variation, but it must not consign poorly observed groups to permanent neglect.

Where differences in rankings are unstable across defensible weights, the system may support broad tiers rather than precise ordering. Retaining a near-boundary category enables targeted evidence gathering without presenting minor numerical distinctions as material facts.

Noncompensatory rules and product benchmarks

Weighted aggregation allows a favorable score on one dimension to offset an unfavorable score on another. Such compensation may be inappropriate when urgent harm or a legally significant signal warrants attention regardless of other indicators. An explicit override or mandatory referral rule can preserve this distinction. It must have a defined trigger, authority, evidentiary threshold, review process, and reason record.

Product benchmarking provides another kind of signal. The European Insurance and Occupational Pensions Authority's (2026) revised value-for-money methodology uses comparable product groupings and cost and value indicators to identify unit-linked and hybrid insurance products warranting further supervisory analysis. The revised approach distinguishes an insurance wrapper and specific investment option in the analytical unit. Distributional reference values provide screening information; the assessment retains a caution area and further analysis rather than equating a benchmark departure with automatic misconduct.

Product peers can differ in guarantees, services, surrender conditions, risks, and investment characteristics. Comparison improves when relevant features are considered, but cluster membership itself remains a modeling choice. A benchmark derived from the observed market distribution also cannot prove that the market's prevailing practices are acceptable. Relative assessment should therefore be complemented by independent examination of contractual value, customer outcomes, and regulatory obligations.

Probabilities and burden estimates

A probabilistic model requires a precisely defined outcome, horizon, population, and prediction date. For example, it might estimate the probability that an independently investigated claim within a given period exhibits a specified material handling deficiency. A firm-level binary label, such as at least one confirmed deficiency in a year, has a different meaning and depends heavily on the size of the firm and intensity of investigation.

If a model estimates an expected event rate per eligible exposure and the authority has a severity estimate conditional on events, an expected burden can be represented by the sum across exposures of the predicted probability multiplied by conditional harm. This approximation requires compatible time windows and outcome definitions. Expected burden is not a substitute for the distribution of severe cases or for nonmonetary criteria.

Probability scores, monetary burden, ordinal controls, and legal overrides can coexist on the same dashboard. Their separation allows supervisors to reason about a case without pretending that all quantities are interchangeable.

Bias, Calibration and Model Uncertainty

Sparse observations and partial pooling

Small denominators make extreme observed rates common even when underlying differences are limited. A single adverse event among two examined files produces a crude rate of 50%. Twenty-five adverse events among 100 examined files produce 25%. The precision of these statistics is different.

For a synthetic illustration, let a binomial event probability have a common Beta(2,18) prior, with mean 10% and concentration 20. For firm A with one event in two observations, the posterior distribution is Beta(3,19), with mean 3/22, approximately 13.64%. For firm B with 25 events in 100 observations, the posterior distribution is Beta(27,93), with mean 27/120 = 22.5%. The crude ordering reverses under partial pooling.

The result follows from the illustrative common prior and the observed sample sizes, not from knowledge about actual insurers. A real implementation would need to justify exchangeability, estimate or validate hyperparameters, test sensitivity to the reference distribution, and distinguish a probability sample from an inspection-selected sample. If the observations were chosen because they already appeared suspicious, binomial sampling from the entire book is not established by the likelihood notation.

Partial pooling can stabilize estimates but can also suppress a genuine emerging problem in a small population. An exceptional severe documented case can warrant attention even when a pooled rate appears low. A model should report uncertainty intervals and counts, not just a posterior mean. Where underlying practices are heterogeneous, pooling across incompatible product or customer segments may create false precision.

Missingness and selective visibility

The observation of conduct harm depends on both customer behavior and supervisory behavior. A regulator conducting intensive examinations discovers problems not found in entities receiving limited examination. An insurer with strong internal detection may report more incidents than one with deficient recording. Treating unobserved outcomes as negative labels converts surveillance intensity into apparent conduct quality.

Missing data may be random conditional on observed characteristics, may depend on unobserved outcomes, or may result from systematic data refusal. The appropriate response differs. Statistical imputation can help quantify uncertainty under justified assumptions. Material reporting deficiencies may separately warrant supervisory inquiry. A fabricated zero is not an acceptable default for a denominator, event count, or unresolved outcome.

Observed complaint and enforcement information can be affected by business size, customer mix, reporting systems, legal changes, previous supervisory attention, and accessibility of redress. An authority should assess how each source enters the observation process. Independent samples, external dispute data, audit trails, and reconciliations can supplement self-reporting, while legal safeguards govern their collection and use.

Calibration and discrimination

Discrimination measures whether a model tends to assign higher scores to cases with the defined outcome than to cases without it. Calibration measures the correspondence between predicted probabilities and observed event frequencies in an appropriate evaluation population. These are different properties. A monotonic transformation can preserve ranking while changing the numerical probabilities.

Van Calster et al. (2019), in their analysis of predictive model calibration, distinguish calibration-in-the-large, calibration slope, and more flexible evaluation of the relationship between observed and predicted outcomes. Their work concerns general prediction methodology in a medical research context. Its mathematical distinctions apply to conduct prediction, subject to the different insurance observation process and legal outcome definitions.

For binary observations y_i and predicted probabilities p_i, the Brier score is the mean of (p_i - y_i)^2. This proper scoring rule evaluates probability predictions, but it alone does not identify particular calibration failures or establish the usefulness of prioritization. A calibration plot, interval estimates, and comparison with baseline performance help reveal whether predicted probabilities are meaningful. The evaluation population and horizon must match the intended use.

Rare adverse outcomes and workload

Suppose an independently labeled synthetic population contains 10,000 cases and 100 adverse outcomes. A classifier detects 80 of the adverse cases and flags 495 of the 9,900 other cases. Its sensitivity is 80%; its specificity is 95%; its precision is 80/575, or approximately 13.91%. The operational workload is 575 flagged cases, and 20 adverse cases remain unflagged.

Saito and Rehmsmeier (2015) explain the value of precision-recall analysis when positive outcomes are rare. Precision makes the relationship between flags and confirmed outcomes visible, whereas apparently favorable discrimination may coexist with a considerable investigation burden. The relevant baseline precision depends on the prevalence of the outcome. Precision calculated on an enriched inspection sample cannot automatically be transported to a population with a different prevalence.

An authority choosing a decision threshold must assess the costs of missed cases, false flags, review burden, and severity. Investigation is a purposeful information-generating activity; some investigations that do not confirm a suspected problem may nevertheless improve knowledge. False-positive terminology should therefore refer to a specific defined prediction target, rather than imply that every unconfirmed inquiry was unjustified.

Uncertainty, drift, and fairness

Uncertainty comes from sample size, measurement error, outcome definition, model specification, changing business practices, and the selective observation of outcomes. Data drift can follow product redesign, changes in reporting, new distribution channels, regulatory reforms, or external shocks. A fitted model that once provided useful triage may lose relevance without a visible change in software.

Monitoring should examine feature distributions, missingness, outcome rates, calibration, ranking performance, and subgroup behavior over time. Changes should trigger investigation before automatic retraining. A shift in complaint submissions following a reporting reform may require a new interpretation rather than merely a new coefficient.

Subgroup assessment should distinguish conduct fairness from the statistical properties of a predictive system. Vulnerable customers may face both greater harm and greater difficulty reporting it. A model trained on formal complaints can underidentify their experiences. Equalizing aggregate error rates cannot by itself determine the correct legal or ethical treatment of groups. The authority must understand the outcome, access mechanism, severity, and applicable obligations, and protect sensitive data in the process.

Supervisory Prioritization and Inspection Allocation

Risk ranking and action selection

An entity's assessed conduct risk is one input to supervisory action. The value of an intervention also depends on the available instrument, feasibility, timeliness, causal effectiveness, and alternative uses of regulatory resources. A high-risk problem may be difficult to address through an immediate examination but amenable to a coordinated information request or product review. Another problem may be readily correctable through targeted engagement despite a lower aggregate risk rating.

Potential intervention benefits should be defined incrementally relative to the appropriate counterfactual. Detecting USD 1 million of historical harm does not establish that a particular examination prevented USD 1 million of future harm. Remediation of past losses, prevention of future losses, information gains, and deterrence are distinct consequences. An evaluation should identify and avoid double counting overlapping effects.

Where probabilities and intervention impacts are credibly estimated, expected avoidable burden may inform decisions. Where such estimates are unavailable, documented multi-criteria judgments and scenarios can be more defensible than invented precision. Legal necessities and urgent severe harms may constrain the feasible decision space before numerical optimization is attempted.

Capacity and discrete allocation

Consider an illustrative 100-hour examination budget. Inspection A requires 70 hours and has an expected avoidable harm benefit of 140 units. B and C each require 50 hours and have expected benefits of 95 units. A offers the greatest individual benefit and a benefit-to-hours ratio of 2.0, compared with 1.9 for B and C. Selecting A first leaves 30 hours unusable for either remaining inspection under the indivisibility assumption. Selecting B and C yields 190 units within the budget.

For indivisible inspections and additive expected benefits, the conventional binary optimization problem is to maximize the sum of selected benefits subject to the sum of selected hours not exceeding capacity. The illustration establishes that highest individual benefit or highest benefit-to-cost ordering need not maximize portfolio benefit. Actual examinations may be divisible, interdependent, or subject to minimum staffing and coordination requirements. The model must reflect those constraints.

If A represents imminent severe harm that legally requires immediate action, the problem changes. Such a requirement belongs in the constraint set or an explicit overriding decision. An optimization model supports choices within the authority's lawful discretion; it does not displace the legal mandate.

Selection portfolios and information value

A balanced supervisory program may contain mandatory responses, targeted high-risk examination, thematic work, independent exploratory sampling, and follow-up to verify remediation. The categories address different information needs. Targeted work can increase the rate of discoveries among examined cases. Exploratory or probability-based sampling can improve understanding of the broader population and expose blind spots. Thematic exercises can examine common practices spanning firms.

If all capacity is directed using the current model, cases outside its preferred region may rarely produce labels. The model then receives feedback mainly from its own earlier choices. An authority can mitigate this selective-label problem by reserving a documented portion of eligible discretionary capacity for independent sampling and by retaining inclusion probabilities where feasible. The appropriate allocation depends on the mandate, resources, severity, and the value of reducing uncertainty. The study does not prescribe a universal exploration percentage.

Information itself has value when it changes future decisions. A relatively inexpensive file review can resolve uncertainty before an intensive examination. Conversely, repeated information requests can postpone necessary action when the basic facts are already clear. Staged inquiry should specify what evidence would confirm, weaken, or resolve the concern and what next action corresponds to each result.

Governance and escalation

A decision record should identify the triggering evidence, applicable rules, score version, material uncertainty, proposed instrument, expected resource needs, urgency, and justification for any override. Supervisory staff should be able to distinguish automated calculations from manual judgments and subsequent findings. Review of overrides can identify useful expert information or patterns of inconsistency.

The score should be treated as a living assessment with a defined update cadence. Material events can generate out-of-cycle review. Confirmed remediation should reduce priority only after evidence shows that the relevant mechanism has been corrected, not simply because the firm has declared the project complete. Repeated unresolved signals should trigger reconsideration of the existing intervention strategy.

Procedural protections include access controls, retention rules, correction mechanisms for erroneous data, segregation of sensitive information, and review of appropriate disclosures. A confidential intelligence score may need different disclosure treatment from a published market statistic or a formal enforcement finding. The authority should clearly govern these categories under local law.

International Comparative Supervisory Frameworks

International standards and consumer protection supervision

IAIS (2024) ICP 9 supplies the supervisory review context; ICP 19 addresses conduct of business. Together they establish that risk assessment should connect ongoing information with examination and supervisory response, while conduct expectations extend across relationships and product stages. The international standards do not establish a universal numerical market conduct score or identical enforcement process across jurisdictions.

The World Bank (2022) guidance develops risk-based financial consumer protection supervision at both market and provider levels. Its broader financial-services perspective highlights the importance of consumer harm, risk assessment, and supervisory intensity. Insurance adaptation requires exposure definitions, product distinctions, and legal mandates particular to insurance. Prudential measures designed for financial soundness cannot simply be reused as proxies for fair customer treatment.

European Union: product governance and benchmarking

Directive (EU) 2016/97 (European Parliament and Council, 2016), in the consolidated insurance distribution text dated January 9, 2024, addresses insurance distribution within its scope. Article 17 establishes conduct requirements associated with customer interests and information; Article 25 concerns product oversight and governance. Application requires attention to scope, national implementation, exceptions, and subsequent legal developments. The cited consolidation is explicitly dated and is not offered as a comprehensive statement of every current national rule.

EIOPA's 2026 revised benchmarking methodology provides a more specific product-oriented example. The methodology uses product clusters and indicators including entry costs, total costs, reduction in yield, surrender value, and investment internal rate of return as applicable to the described investment component. Benchmark calculations include distributional reference points, including upper cost and lower value reference positions and caution areas reflecting dispersion. Products outside reference values can require deeper examination. The benchmark is a screening and analytical mechanism, not a general statutory price cap or an automatic determination that an individual product is unlawful.

This framework illustrates the significance of matching the comparison unit to the question. Wrapper and investment option characteristics affect observed costs and returns. An insurer-wide average can conceal a problematic combination, while an indiscriminate cross-product comparison can attribute legitimate design differences to poor value. The supervisory conclusion requires examination of both the quantitative signal and the product's relevant characteristics.

United States: market regulation and MCAS

The National Association of Insurance Commissioners' (n.d.) market conduct regulation overview describes state insurance market regulation, including examination, complaint information, and coordination. State authorities exercise powers under their respective legal frameworks. NAIC resources facilitate consistency and coordination but do not make every state law identical.

The NAIC's (2026) MCAS reporting resources provide data-year-specific instructions and scorecard resources. Line-specific definitions are fundamental to comparisons; a metric meaningful for one line of insurance may lack an equivalent denominator in another. The existence of a 2026 reporting page does not establish that completed 2026 data are available, and this study makes no inference about actual 2026 company results.

The MCAS data dashboard (NAIC, n.d.) explains a ratio aggregation based on summed numerators and denominators. If entity j has n_j events and d_j exposure, the pooled ratio is sum(n_j)/sum(d_j), which is the exposure-weighted average of individual n_j/d_j ratios where denominators are positive. A simple average of company ratios assigns equal weight to companies regardless of exposure. These summaries answer different questions, so the authority should preserve the aggregation definition when comparing results.

The U.S. example demonstrates the importance of standardized reporting and comparative intelligence within a decentralized system. Data comparability supports selection rather than abolishing the need to examine local law, context, and evidentiary reliability.

United Kingdom: outcomes and multiple evidence streams

The Financial Conduct Authority's Consumer Duty rules (FCA, version consulted 2026), particularly PRIN 2A.9, establish outcome monitoring expectations within their specified scope. Relevant firms should identify deficiencies and differences in outcomes between groups as required by the rules, investigate causes, and take appropriate action. These duties support a supervisory focus on outcomes alongside processes, subject to the rules' application.

In its insurance outcomes monitoring review (FCA, 2024, updated 2025), the authority examined 20 larger insurance firms and reported limitations in approaches that emphasized process activity without sufficiently demonstrating consumer outcomes. The sample is not a representative prevalence estimate for the entire market, and the review's scope should not be generalized to unexamined requirements. It nevertheless illustrates a recurrent measurement problem: evidence that an activity occurred is not necessarily evidence that customers received the intended outcome.

The general insurance value measures and ombudsman complaint publications add distinct streams of information. Value measures describe specified aspects of product and claims experience; dispute outcomes provide evidence conditioned on referral and resolution. Neither stream alone represents all harm. A supervisory system can triangulate them with product documentation, firm monitoring, contract analysis, independent files, and customer research.

Comparative synthesis

Framework Primary analytical emphasis Contribution to prioritization Principal limit
IAIS ICPs Supervisory review and fair treatment Mandate and lifecycle context Jurisdictional implementation
World Bank guidance Market and provider consumer risk Impact and risk assessment organization Not a universal insurer rating
EU and EIOPA Distribution and product-level value Comparable product screening Cluster and scope dependence
U.S. market regulation and MCAS Line-specific conduct reporting Comparative intelligence and coordination State law and data definitions
UK FCA and ombudsman Consumer outcomes, claims and disputes Triangulation and remediation inquiry Selected observations and varying scope

The frameworks are complementary rather than interchangeable. A common analytical vocabulary can support international comparison of risk dimensions and limitations, but thresholds derived from one market should not be transferred mechanically to another. Legal scope, product structures, customer reporting channels, market composition, and available supervisory powers determine the meaning and practical use of each indicator.

Validation, Transparency and Outcome Evaluation

Independent evaluation design

Validation begins by fixing the prediction date, available information, target population, outcome definition, and horizon. Data recorded after the prediction date must not leak into the inputs. Repeated records from one entity, product, claim, or customer can produce optimistic evaluations if closely related observations appear in both development and test sets. Separation by time and, when appropriate, entity or group can improve the relevance of testing.

A score designed for examination selection should be evaluated at the actual feasible inspection capacity. Global ranking metrics can conceal poor performance at the particular top segment that determines action. Useful metrics include precision at the available capacity, recall against independently established eligible positives, capture of severe harms, information gains, and resource consumption. Probability estimates also require calibration evaluation.

Consider eight synthetic eligible units A through H, independently labeled after the test horizon. Adverse outcomes occur in A, C, D, and F, with severe outcomes in A and D. A baseline selects B, C, and E. Its precision at three is 1/3, recall is 1/4, and severe outcome capture is zero. A candidate selects A, C, and D. Its precision is 3/3, recall is 3/4, and it captures both severe outcomes. These calculations demonstrate evaluation at fixed capacity and the importance of outcome severity. They establish no real-world model performance and no prevented harm.

In practical evaluations, the true labels of all eligible units are seldom known. Using only investigated cases estimates performance within a selected sample. Probability-based selection, independent reviews, or appropriately justified weighting may improve population inference. When case i has nonzero known inclusion probability pi_i, a design-weighted total of an observed quantity can be estimated by summing y_i/pi_i across selected cases under the relevant sampling design. Zero inclusion probabilities prevent recovery of the unobserved region by that method alone. High weights can make estimates unstable and require appropriate uncertainty analysis.

Detection versus supervisory impact

Detection addresses whether the authority identifies actual problems. Supervisory impact concerns the change in outcomes induced by its action. A system that detects many events may perform well at triage while having limited preventive effectiveness if the selected instruments do not correct the cause. A system that prevents a large number of future events may observe fewer confirmed violations because prevention changed the event process.

Before-and-after comparisons are vulnerable to regression to the mean, secular trends, changes in reporting intensity, concurrent interventions, and shifts in the insured population. A causal evaluation requires a credible comparison or design. Depending on the legal and operational setting, phased implementation, suitable comparison groups, interrupted time-series analysis, or randomized selection among eligible low-urgency alternatives may support stronger inference. Mandatory interventions and urgent harm must remain outside any design that would improperly withhold action.

Measures of completed examinations, records reviewed, and remediation programs initiated describe outputs. Verified repayment, corrected contractual performance, improved accessibility, reduced recurrence, and changes in adverse outcomes describe consequences. Monetary restitution measures require a defined eligible population and evidence of actual delivery. The evaluation should assess whether affected customers who did not complain were identified and appropriately remediated where applicable.

Transparency and accountability

Transparency has several audiences. Analysts need definitions, models, uncertainty, data transformations, and correction histories. Decision-makers need reasons for priority, the alternatives considered, and material limitations. Supervised entities may need information about reporting obligations and the basis of formal findings under applicable procedures. Public users need aggregate explanations that avoid presenting unconfirmed intelligence as established misconduct.

Publishing every operational threshold can facilitate gaming, while opacity can undermine accountability and comparability. A defensible approach identifies what is public, what is confidential for a documented legal or operational reason, and what independent review can access. Changes to indicator definitions, weights, overrides, and model versions should be governed and logged.

Scoring should also be robust to strategic responses. If an indicator rewards rapid complaint closure, a firm might close cases prematurely. If it rewards low complaints, a firm might discourage complaints. Independent measures of substantive resolution, reopening, accessibility, sampled outcomes, and customer restitution can identify these patterns. The authority should investigate anomalous changes before interpreting them as improvement.

Scientific limitations and research agenda

The consulted sources establish institutional methods, reporting structures, and established statistical evaluation concepts. They do not provide a common confidential dataset, fitted coefficients for cross-jurisdictional insurance conduct prediction, universal monetary severity weights, or causal intervention effects. Those empirical questions remain open. The illustrative examples establish arithmetic and methodological consequences under explicit assumptions.

A future empirical study would define a jurisdiction and a conduct outcome, establish reporting and sampling definitions, obtain lawful independent labels, compare baseline rules and candidate models, assess calibration and severity capture, and evaluate the relation between selection and remedial outcomes. It should publish the extent of missingness, selection assumptions, subgroup limitations, and uncertainty rather than claim general accuracy from a selected examination sample.

Cross-jurisdictional validation would first require harmonization of outcome definitions and scope. Where harmonization is impossible, the appropriate result is a comparison of local model performance and limitations, rather than a pooled score whose apparent precision conceals semantic differences.

Conclusion

Market conduct risk scoring is an instrument for organizing supervisory evidence and directing attention toward potential policyholder harm. Its scientific validity depends on explicit definitions of the population, outcome, exposure, and observation process. Its administrative validity depends on a lawful decision framework, adequate information, documented judgment, and the relationship between the selected intervention and the objectives of supervision.

The study establishes three distinct levels of analysis. Indicators describe evidence and comparative position. Probabilistic and burden estimates require defined outcomes, assumptions, and validation. Action selection considers expected benefits, resource constraints, uncertainty, urgency, and legal requirements. Treating these levels as interchangeable creates misleading precision and obscures the reasons for supervisory action.

International standards and operational frameworks provide diverse and complementary sources of evidence. IAIS supplies the international supervisory and conduct context; World Bank guidance organizes consumer protection risk assessment; EIOPA develops product benchmarking; MCAS supports defined comparative reporting; FCA outcomes monitoring and dispute data illuminate customer experiences through different observation mechanisms. Their useful integration requires attention to institutional scope and data semantics.

The synthetic calculations demonstrate that portfolio composition can reverse rankings, sparse samples can change estimated rates, indicator weights embody consequential choices, rare harms can create substantial flagging workloads, and discrete capacity constraints can defeat simple priority ordering. Independent validation and impact evaluation are necessary to distinguish a successful screening procedure from successful consumer protection.

A scientifically defensible supervisory program is therefore evaluated not by the sophistication of its numerical score alone, but by the quality of its definitions and evidence, the transparency of its reasoning, its capacity to identify severe and hidden harms, and its demonstrated ability to support proportionate and effective intervention.

Methodological Appendix

Reproduction of synthetic examples

All numbers below are hypothetical. No real insurer, product, claim, or supervisory authority is represented.

Example Calculation Result
Monetary burden 100 × 30; 5 × 20,000 3,000; 100,000
Event rate per thousand 1,000 × 100/10,000; 1,000 × 5/10,000 10; 0.5
Crude case mix (2+45)/1,000; (27+6)/1,000 4.7%; 3.3%
Common reference mix (2%+5%)/2; (3%+6%)/2 3.5%; 4.5%
Equal weights (80+20+50)/3; (40+90+50)/3; (55+55+55)/3 50; 60; 55
Harm-led weights 0.6h + 0.2e + 0.2c 62; 52; 55
Beta posterior means (2+1)/(20+2); (2+25)/(20+100) 13.64%; 22.5%
Rare outcome precision 80/(80+495) 13.91%
Capacity portfolio B + C: 50+50 hours; 95+95 benefit 100 hours; 190 units
Precision at three 1/3; 3/3 33.33%; 100%
Recall at three 1/4; 3/4 25%; 75%

Minimum assessment record

A reusable assessment record should preserve the entity and analytical unit; date and applicable horizon; raw numerator and denominator; definition version and source provenance; missingness and correction history; comparison group and reference period; indicator direction and transformation; prediction or rating semantics; uncertainty and subgroup limitations; score and model version; intervention decision, rationale, and override; legal scope and review status; independent findings; verified remediation and follow-up outcomes.

This record describes the information necessary to reproduce an analysis and review a decision. It is not a proposed universal data collection mandate. Data minimization, confidentiality, legal authority, and system capacity determine which fields may be collected and retained in practice.

References

  • International Association of Insurance Supervisors (IAIS). (2024). Insurance Core Principles and Common Framework for the Supervision of Internationally Active Insurance Groups. Revisions adopted December 5, 2024. ICPs 9 and 19. Source.

  • World Bank. (2022). An introduction to developing a risk-based approach to financial consumer protection supervision. Guidance note, December 2022, especially pp. 18–21. Washington, DC: World Bank. Source.

  • European Parliament and Council of the European Union. (2016). Directive (EU) 2016/97 of 20 January 2016 on insurance distribution (recast). Official Journal of the European Union, L 26, 2–59. Consolidated text, January 9, 2024, arts. 1, 17, 25. Source.

  • European Insurance and Occupational Pensions Authority (EIOPA). (2026, July 6). Revised methodology on value for money benchmarks. EIOPA-BoS-26-304, especially sections 2–3. Source.

  • National Association of Insurance Commissioners (NAIC). (2026). Market Conduct Annual Statement: 2026 data year. Reporting and scorecard resources. Source.

  • National Association of Insurance Commissioners (NAIC). (n.d.). MCAS data dashboard. Source.

  • National Association of Insurance Commissioners (NAIC). (n.d.). Market conduct regulation. Source.

  • Financial Conduct Authority (FCA). (2026, version consulted). FCA Handbook, PRIN 2A: The Consumer Duty, especially PRIN 2A.9. Source.

  • Financial Conduct Authority (FCA). (2024, June 26; updated 2025, December 3). Insurance multi-firm review of outcomes monitoring under the Consumer Duty. Source.

  • Financial Conduct Authority (FCA). (2021, November 10; version updated 2026, July 16). General insurance value measures. Source.

  • Financial Ombudsman Service. (2026). Annual complaints data and insight 2025/26. Source.

  • Van Calster, B., McLernon, D. J., van Smeden, M., Wynants, L., & Steyerberg, E. W., on behalf of the STRATOS Topic Group. (2019). Calibration: The Achilles heel of predictive analytics. BMC Medicine, 17, Article 230. DOI: 10.1186/s12916-019-1466-7. Source.

  • Saito, T., & Rehmsmeier, M. (2015). The precision-recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced datasets. PLOS ONE, 10(3), e0118432. DOI: 10.1371/journal.pone.0118432. Source.