RESEARCH DOSSIER
Insurance Conduct Supervisory Data Collection and Indicator Calibration
Abstract. Insurance conduct supervision depends on the ability to connect the treatment of customers to measurable events throughout the contract lifecycle. This monograph examines how supervisory authorities collect, define, reconcile, analyze, and calibrate insurer conduct indicators for product design, distribution, pricing, servicing, claims, and redress. Comparative documentary analysis draws on international supervisory principles, the United States Market Conduct Annual Statement, the United Kingdom Financial Conduct Authority's value-measures reporting and consumer-outcomes requirements, European retail risk indicators, and Australian internal dispute resolution reporting. The study distinguishes event populations, exposure denominators, retrospective reporting periods, and longitudinal cohorts; examines how incomplete reporting, outcome delays, portfolio composition, and inconsistent complaint coding distort inference; and evaluates statistical screening, direct standardization, partial pooling, and sensitivity analysis through reproducible synthetic examples. The central analytical conclusion is that a conduct indicator acquires supervisory meaning through the joint specification of its observation process, comparison population, uncertainty, legal relevance, and follow-up decision. A benchmark is an instrument for selecting and testing evidence. Substantive conclusions about unfair treatment require examination of contracts, customer circumstances, operational processes, and the applicable legal standard. The work develops an integrated comparative synthesis of established measurement methods and supervisory practices, with explicit boundaries on causal inference and empirical coverage.
Keywords: insurance conduct supervision; market conduct; supervisory reporting; consumer outcomes; indicator calibration; claims; complaints; data quality; risk-based supervision; case-mix adjustment; international insurance regulation
Introduction
The scientific object of conduct measurement
Insurance is a contingent financial promise. Its performance becomes visible at distinct moments: the selection of risks and customers, the description of coverage, the determination of premium, the administration of the contract, the response to an insured event, and the resolution of a disagreement. The supervisory record of a promise therefore spans years and multiple institutions. Premiums may be received before losses arise; sales may be made through an intermediary; an outsourced administrator may resolve the claim; an ombudsman may adjudicate a complaint long after the original contract expires. A snapshot of any one institution's transaction ledger is an incomplete observation of this process.
Conduct supervision concerns how insurance customers are treated and how the commitments created by the insurance relationship are fulfilled. The IAIS Insurance Core Principles (online edition, ICP 19) establish fair customer treatment across the contract lifecycle and the risk-based supervisory framework in ICP 9 supports differentiated supervisory activity. These principles supply the international institutional context. The enforceable rights, duties, remedies, collection powers, and procedural safeguards applicable to any particular insurer derive from the relevant jurisdictional instruments.
Supervisory conduct data are recorded observations acquired for regulatory monitoring, inquiry, and evidence-based decision-making. An indicator is a specified transformation of such observations. Its interpretation depends on the data-generating process: customers experience an event, some communicate about it, an institution records and classifies it, a reporting process aggregates the records, and a supervisory system transforms the return. Each transition can change the information available to the next stage.
This perspective separates three scientific questions. First, what is the observable population and which aspects of customer treatment are represented? Second, what causes the resulting measurement to differ between institutions, groups, or time periods? Third, what supervisory decision can the measurement appropriately support? A description of paid claims can establish the distribution of recorded payments. It cannot independently establish that unpaid contractual benefits did not exist. Measuring market conduct therefore requires the reconstruction of opportunity, process, and realized outcome.
Research method and limits
The inquiry proceeds through comparative reading of primary supervisory standards, reporting specifications, and published findings, followed by an analytical examination of measurement validity. The United States, United Kingdom, European Union, and Australia are selected as complementary institutional cases: centralized line-specific filings, product-level value reporting and broader consumer-outcomes monitoring, supranational indicators built partly from prudential returns, and structured internal complaint reporting. The comparison is functional rather than a claim that these jurisdictions exhaust international practice.
The research cutoff is November 19, 2026. Historical reporting manuals are explicitly dated. The 2025 FCA data publication refers to observations in 2025 and publication in 2026. The IAIS survey concerns responses collected in 2020–2021; it provides evidence of practices at that time, not a current census. Published summaries and selected source passages support the empirical statements. The study does not claim access to confidential insurer submissions or independent reconstruction of the downloadable firm-level datasets. Numerical examples are constructed for demonstration and are identified as synthetic. The analytic methods are established statistical techniques applied to the supervisory context; no new theory, legal doctrine, or empirical performance ranking is proposed.
The guiding analytical distinction is between descriptive validity, predictive validity, and adjudicative sufficiency. Descriptive validity concerns whether a statistic measures its stated operational object. Predictive validity concerns whether it forecasts a separately defined adverse outcome in a specified population. Adjudicative sufficiency concerns whether the collected facts establish an applicable violation or support a particular lawful intervention. Evidence can satisfy the first condition while failing the second and third. The study returns to that distinction in each phase of the supervisory process.
Regulatory Conduct Data Architecture
From statutory competence to a measurable population
A data collection begins with its lawful purpose. Supervisory authorities have different competencies over insurers, intermediaries, delegated administrators, products, and consumer disputes. The reporting perimeter must express which entities and activities are subject to the return, the legal basis for mandatory submission, submission timing, permitted uses, confidentiality obligations, correction rights, and restrictions on disclosure. Data coverage cannot be inferred from the geographic market alone: an insurer may write cross-border business, a branch may administer a portfolio originated elsewhere, and a group subsidiary may operate under a separate authority.
The IAIS (2022) survey of supervisory indicators reports responses from 51 authorities. Of these, 46 reported explicit direct powers to collect insurer data and 41 a formal framework; the survey identified 201 differently calculated indicators, reflecting considerable definitional variety. Among the 46 authorities answering the indicator-count question, 37 collected at most 20. Those counts support two distinct findings: institutional access to data and institutional agreement about measures are separate achievements; the broad population of reported indicators does not mean that each authority routinely collects all of them.
The evidence supports organizing collections by use rather than by a desire to accumulate every available field. Routine returns serve stable comparisons over time. Targeted collections examine emerging risks, atypical events, or gaps in an existing return. Case files support investigation of specific practices. Consumer surveys and mystery shopping can reveal unrecorded experiences. An authority needs a mechanism for treating each source on its own evidentiary terms and for combining corroborating sources without counting the same event repeatedly.
A return may contain census data for its eligible reporting entities and remain incomplete for the underlying consumer population. A minimum premium threshold excludes smaller portfolios; reporting exemptions exclude some companies; uninsured people and unsuccessful applicants may have no policy record. The denominator for a statement about reporting insurers is different from the denominator for a statement about access to insurance. Metadata should make that difference explicit. Coverage assessment precedes dataset aggregation.
Observation units and relational structures
The foundational design problem is the selection of the unit of observation. A policy, a customer, a covered life, an insured vehicle, a distribution transaction, a claim, a benefit request, and a complaint represent different economic or administrative units. In a multi-beneficiary contract one policy can encompass several lives. A single incident can give rise to multiple claims under different coverages. Multiple complaints can arise from one claim; one complaint may concern several policies. A dispute file may consolidate complaints concerning related actions.
A workable relational specification separates the legal entity and group hierarchy from the product hierarchy. A stable policy identifier identifies the contract, while coverage identifiers distinguish insured risks and benefit obligations. A claim identifier records an individual adjudication object, and an episode identifier distinguishes reopening or supplementary payment. Complaint and dispute records remain independent entities with optional links to the underlying claim, policy, sales interaction, or servicing episode. Disconnected records cannot simply be dropped from aggregates because failure of linkage can itself concentrate among consumers with more difficult journeys.
An appropriate common specification can be summarized as follows. It is an analytical example of conventional relational design rather than an existing mandatory international return.
| Dataset | Observation grain | Essential contextual fields | Intended analytical use |
|---|---|---|---|
| Reporting perimeter | Entity, jurisdiction, period | License, activity, product scope, exemptions, schema version | Coverage and reconciliation |
| Product and coverage | Product version and coverage segment | Target population, contract term, exclusions, deductible, distribution plan | Product comparability |
| Exposure | Coverage-period record | Start and end dates, cancelation, units at risk, insured lives or assets | Denominator construction |
| Sale and renewal | Transaction or eligible offer | Channel, price components, advice status, offer outcome | Distribution and pricing |
| Claims | Claim and episode | Report date, decision dates, disposition, payment, reason | Claims process and benefits |
| Servicing | Contact or service request | Request type, channel, completion, elapsed time | Access to support |
| Complaints and disputes | Complaint or external case | Receipt, issue, disposition, remedy, referral and link quality | Redress and corroboration |
| Revisions | Return version and changed field | Prior value, corrected value, reason, approval and time | Reproducibility |
A reporting authority may need only certain aggregated cells rather than identifiable individual events. In that case, it specifies granular aggregation dimensions, permitted overlaps, unique count rules, the treatment of missing values, and the reconciliation constraints to be applied before submission. Event-level records are justified when the analysis needs longitudinal linkage, outcome maturity, audit samples, or a properly authorized investigation. Data acquisition should be proportionate to the question and the risks of the collection.
Source systems and institutional incentives
A conduct return may inherit information from underwriting, policy administration, claims management, call-center records, accounting, intermediary reports, complaints teams, or external resolution institutions. These systems use operational concepts optimized for different purposes. A claims system might distinguish reserve closure from customer notification; an accounting system might record a payment reversal in a different period; a service system may treat a quick complaint resolution as an ordinary contact. A common supervisory schema requires controlled translations and documented ownership of those translations.
The NAIC MCAS program (2024-data-year snapshot) illustrates the institutional benefit of agreed definitions by insurance line and centralized analysis. A common collection can reduce the cost of producing separate incompatible filings, while permitting state-level comparisons and retaining line-specific distinctions. The architecture does not eliminate differences in legal jurisdiction, product eligibility, or underlying exposure; it makes those differences more visible and measurable.
The operative information flow also includes a feedback loop. Submission rules create incentives for firms to improve data quality and for supervisors to revise definitions. A validated anomaly can reveal a new type of conduct harm, an inadequate code list, or a denominator defect. The return owner must distinguish these cases. Correcting the definition may change the historical series, whereas correcting customer treatment changes the process that generates future observations. Both changes require a record.
Reporting Perimeters and Definitions
Denominator alignment
A ratio is interpretable when the denominator represents the opportunities that could generate the numerator. A number of complaints per policy-year describes a rate over contract exposure. A share of rejected adjudications among decided claims describes the conditional disposition of a claims process. A proportion of external appeals upheld among resolved appeals describes an external case outcome conditional on escalation and resolution. These objects differ even if each is colloquially labeled a complaint or rejection rate.
Let E denote exposure accumulated over the period, R the number of relevant recorded events, and k the reporting scale. An event-rate statistic is r = kR/E. E can be policy-years, member-months, or another additive exposure unit. In contrast, a binomial outcome proportion p = X/N requires N distinct eligible outcome trials and X trials exhibiting the specified outcome. A claim complaint count can exceed the number of claimants when repeated complaints are allowed; it is then an event rate, not a probability. Using a binomial model for such a count would impose an incorrect support and variance structure.
The denominator may be a stock, flow, or integrated exposure. End-period in-force policies form a stock; new sales form a flow; the sum of fractions of a year insured approximates integrated exposure. If a portfolio expands rapidly, a year-end stock can substantially exceed the average population exposed to the opportunity for complaints. Consider a synthetic portfolio growing at a linear rate from 10,000 to 20,000 policies during a year, with 300 recorded complaints. Under continuous linear growth, exposure is 15,000 policy-years, corresponding to 20 complaints per 1,000 policy-years; using the end-year stock gives 15 per 1,000 policies. These are different quantities, and an unchanged event count produces a 25% lower numerical rate under the latter denominator.
Annual contracts, monthly renewable plans, travel policies of days' duration, and long-duration life insurance require different interpretation of policy counts. Ten thousand one-week contracts and ten thousand year-long contracts do not imply equivalent time at risk. For a sales-practice indicator, transactions may nevertheless be the relevant denominator because each transaction creates an opportunity for inappropriate selling. The denominator follows the mechanism of harm, not the convenience of a readily available policy count.
The FCA's SUP 16.27 reporting definitions specify an average of month-end policies for claims frequency. In the NAIC 2024 scorecard formulas, some life and long-term-care complaint indicators use policies in force, while other measures use coverage-specific claim or determination populations. The exact formula is essential information. A shared name is insufficient for equivalence, and the historical edition should never be silently substituted for the current applicable instructions.
Event identity and the observation clock
The time variable attached to an insurance event should reflect the question asked. For claims, relevant timestamps can include date of loss, initial notice, registration, receipt of requested evidence, eligibility determination, acceptance or rejection, first payment, final payment, customer communication, reopening, and external dispute resolution. A statutory clock may begin at a legally defined milestone that differs from the customer's experienced wait. Both clocks can be legitimate measures of different obligations.
An annual period return counts events satisfying defined timing rules during that year. A cohort return follows events initiated during a defined window, even when their outcomes occur later. A period closure ratio uses claims closed during the period, including those received previously. A cohort denial proportion follows reported claims until resolution and attaches outcomes to the original registration cohort. A third object, the outstanding inventory, counts unresolved claims at a specified time. Comparisons must respect the object.
The current NAIC private passenger auto instructions linked for the 2026 data year illustrate the importance of event identity. They specify separate counting and aging for a claim reopened for a supplemental payment. A common cross-system claim ledger should preserve both the original claim and the administrative episode so that the return can reconstruct the source definition without losing the history experienced by customers. The reopened episode may be a legitimate new reporting count without representing a new accident or a new policyholder.
Consider an insurer with 100 registered claims in January. By June, 70 have been paid at least partly, 10 rejected, and 20 remain open. The observed paid share of the original cohort is 70%, the rejection share of decided claims is 10/80 = 12.5%, and the paid share among decided claims is 87.5%. If every unresolved claim is ultimately either paid or rejected, the possible final rejection share lies between 10% and 30%. A 12.5% rejected share of decided claims is a valid conditional description but can be biased for the final cohort outcome when slow cases differ systematically. The outstanding population and its maturity must accompany the result.
Product and population taxonomy
A common taxonomy should retain contract features that materially alter consumer experience: insurance line, coverage, benefit trigger, contractual exclusions, deductibles, term, target population, sale channel, policy issue date, geography, and distribution arrangement. Product labels alone rarely identify homogeneous exposures. Motor coverage includes collision, liability, theft, and ancillary benefits, with differing claim paths. Health insurance can have networks, prior authorization, covered lives, and multiple bill lines. Long-term care can require repeated benefit determinations over long durations.
The 2026 MCAS resources list separate reporting instructions and revisions for particular insurance lines. Their architecture reflects the practical necessity of preserving product-specific measurement. A harmonization program can retain common primitives (registered event, decision, complaint, payment, exposure) and line-specific parameters (eligibility triggers, benefit units, categories and timing definitions). Cross-product aggregation is defensible only when the intended analytical comparison can tolerate the differences that remain.
Data labels should distinguish a recorded zero, a genuinely inapplicable field, an unknown value, a suppressed cell, a pending case, an unavailable system field, and a missing return. Zero complaints for a reporting entity is meaningful only when the complaint collection covers the relevant activities. A missing denominator makes a ratio undefined even when the numerator is known. A suppressed count reveals some range of information and does not mean zero. These states form an essential part of the measurement specification.
Claims, Pricing, Distribution and Redress Data
Claims indicators and customer outcomes
The claims stage is a primary manifestation of the value of insurance, but claims information can describe contractual entitlement, efficiency, and customer experience only when the corresponding observations are available. A valid claims dataset should distinguish rejection, closure below deductible, customer withdrawal, fraud determination, duplicate claim, invalid coverage, partial payment, and full payment. The categories often have legally distinct consequences. Closure without monetary payment may follow a contractual exclusion, a service delivered in kind, or a customer withdrawing after receiving an explanation. These should not be conflated with a wrongful refusal to honor an insured claim.
The FCA's SUP 16.27 definitions distinguish registered claims from specified excluded categories and period rejections from the subset of claims newly registered: the claims acceptance formula is the number registered minus the number rejected, divided by registered claims. A paid-and-closed claim is defined separately. An exact reporting implementation therefore needs the original numerator rules and the timing basis; a statistic labeled acceptance need not equal the proportion of a matched inception cohort that has been paid.
This distinction matters empirically. The FCA (2026) 2025 value measures publication reports home acceptance rates of 62–71%, travel of 83–86%, and motor of 99%, while warning of inconsistent home-insurance reporting. The observed contrast warrants investigation of product scope, event capture and classifications. It does not establish a comparable difference in wrongful-denial incidence. A scientifically defensible response is to obtain the actual reporting conventions, reconcile them to source records, and then sample the relevant claim decisions.
Payment timing likewise requires qualification. Report the eligible claims and the proportion still open at selected maturities; include quantiles rather than relying on the mean alone. For claims requiring ongoing periodic benefits, initial eligibility, interruption, reinstatement, payment cadence, and terminal discharge are separate milestones. Time from customer notice to the initial substantive decision captures different impacts from time from complete evidence to decision. A supervisor can compare compliance with a local statutory clock and separately assess practical barriers experienced by claimants.
An operational reconciliation expresses the outstanding claims inventory at period end as opening inventory plus new episodes and reopenings counted as additions, minus closures and other authorized removals, with explicit transfer or adjustment terms. All components must use the same observation grain. If a reopening is already included in new episodes, adding it again double-counts. If claims are recorded at coverage level and the denominator at policy level, the conversion must be disclosed. Every composite indicator should be reproducible from primitive counts.
Pricing, value, and distribution
Pricing data concern contractual cost to the consumer, the allocation of risk and expense, and the configuration of the distribution chain. Data needed for conduct assessment may include quoted and accepted premiums, renewal offers, coverage and deductible changes, fees, commissions, cancellation refunds, and optional add-ons. A comparison of renewal prices requires consistent coverage and risk profiles, as well as information about changes in insured characteristics, inflation, taxes and surcharges. Average premiums for two firms with differently composed portfolios measure different portfolios.
A premium-to-benefit measure can be useful when it expresses the corresponding time basis and benefit population. Claims payments divided by current written premium compare cash or accounting flows of different vintages; an incurred-loss measure divided by earned premium aligns a different set of economic concepts. Both measures have legitimate uses and can diverge during premium growth, portfolio run-off, changes in claim severity, or catastrophe periods. Neither determines the value of risk transfer to an individual consumer in isolation.
The FCA (2026) reported GAP insurance figures offer a concrete illustration: the stand-alone claims-cost-to-premium figure fell from 104% in 2024 to 53% in 2025 after prior intervention and substantial changes in premium flow. A percentage over 100 can arise when claims from earlier sales are paid against diminished current written premium. That observation is consistent with runoff dynamics; it is not alone evidence of superior consumer value or a defective data record. The applicable coverage, claim costs, premiums, reporting basis, and circumstances determine interpretation.
Product oversight needs evidence about the target market, actual distribution, and subsequent outcomes. In the EU, Article 25 of the IDD links product approval, target-market assessment, distribution arrangements, information exchange, and ongoing review, with the large-risk exception specified in the article. The conduct measurement implication is to preserve the product version and distribution channel associated with the original transaction. A change to exclusions after the sale cannot retrospectively identify what the customer originally purchased.
Examples of distribution indicators include the share of sales outside a documented target population, early cancellation per issue cohort, replacement by channel, cancellation following renewal price changes, repeated attempted contacts before cancellation, and complaints relating to disclosures or advice. Each has a different event denominator. An outside-target classification requires a lawful and operationally workable target label; legitimate exceptional sales accompanied by documented customer needs may differ from systematically misdirected sales. Early lapse may signal unsuitable distribution, changes in financial circumstances, or better competing offers. It becomes a stronger indicator when combined with transaction documentation and subsequent outcomes.
Distribution data should retain separate seller, product manufacturer, and administrator identifiers. Economic responsibility and regulatory accountability may differ. An authority can investigate whether a poor outcome is concentrated in an intermediary's sales or across all channels of one product. Comparative analysis of outsourcing requires complete submissions from service providers or reconciled records delivered through the regulated firm. Outsourced processing should never create a blind spot solely because the policy administration ledger resides elsewhere.
Service access, complaints, and redress
A complaint is a recorded expression of dissatisfaction falling within the applicable definition. Recorded complaints are jointly generated by underlying problems, the willingness and ability to complain, the channels available, institutional recognition and classification, and submission rules. High rates may reflect serious problems, accessible redress, or both. Low rates can result from good service or barriers to complaint registration. The measurement of complaint incidence therefore has a surveillance dimension and an access dimension.
ASIC's Report 801 on internal disputes (2024) identifies 1,561,824 general insurance complaints in the 2023–24 financial year and cautions that variations in complaint handling culture can affect reporting totals. Approximately 76% were resolved on the day received. These observations support measuring the recording process and resolution quality in addition to volume and speed. Same-day resolution may be beneficial, but it is also compatible with superficial closure if the underlying issue remains unaddressed. Case reviews and recurrence indicators help distinguish these possibilities.
A complaint fact record should retain original receipt date, channel, products involved, primary and secondary issues, acknowledgment, disposition date, root cause, remedy type and amount, referral, reopenings, and linked earlier contacts when lawfully available. Multiple issue codes may be necessary to describe the complaint. The number of issue selections and the count of distinct complaints then diverge. Published dashboards must state whether their percentages sum over complaints or issue mentions. Duplicate imports between the firm's complaint database and an external dispute organization should be reconciled without suppressing legitimately distinct records of internal and external stages.
The UK Financial Ombudsman Service's (2026) 2025/26 complaints and insight distinguishes new cases from uphold outcomes among resolved cases. Its car or motorcycle insurance category reports 13,420 new cases and a 35% uphold rate. Multiplying these quantities to infer the exact number of upheld new cases is invalid unless the resolved population is known to be the same cohort. The external dispute population is selected through consumer awareness, eligibility, procedural progression and referral. It is not a random sample of all contracts or all rejected claims.
A useful supervisory portfolio therefore connects complaint rates per exposure, escalation among eligible internal complaints, time to effective resolution, reopened cases, actual remedy completion, recurrence of the underlying issue, and independently sampled complaint-register completeness. Redress amount per case should distinguish monetary compensation from unpaid benefits, premium refunds, interest, expenses, and nonmonetary remediation where feasible. An apparent increase in remediation expenditure may indicate more detected harm or more effective correction of existing harm. Read together, the event counts, payment records, and root-cause information are more informative than a unidirectional target.
Coverage, vulnerability, and the unobserved customer
A purely policy-based dataset systematically misses people who could not obtain insurance, failed to understand the terms, abandoned an application, or decided not to pursue a claim. An authority assessing fair treatment needs relevant information on product availability, unsuccessful applications, accessible communication, claim abandonment and service barriers within the legal scope of its mandate. Surveys and appropriately designed samples of communications can add observations to the policy register, provided their sampling mechanisms and uncertainty are recorded.
The FCA's (2024) insurance outcomes monitoring review examined governance information requested from 20 larger organizations and identified inadequate differentiation among customer groups in some approaches. This is qualitative evidence from selected organizations and should not be generalized to an exact sector prevalence. It nevertheless demonstrates why aggregate averages may miss uneven outcomes across customer circumstances and service channels.
Vulnerability indicators require great care. Data should be collected only on a legitimate basis, reflect the measurement purpose, and distinguish unknown characteristics from confirmed absence. Consumers may disclose circumstances selectively, and access to support can affect the probability of being identified as vulnerable. The observed difference between groups can reflect both classification and treatment. A justified investigation examines whether the institution's service processes accommodate relevant needs; statistical adjustment should not automatically remove a disadvantage that constitutes the very conduct question under study.
Data Quality and Reporting Governance
Validity, completeness, and reconciliation
Data quality is multidimensional. Syntactic validity concerns whether a return conforms to types, ranges, mandatory fields and enumerations. Semantic validity concerns whether the stored values correspond to the prescribed definitions. Completeness concerns whether all eligible records are represented. Consistency concerns whether different fields, periods and systems tell a coherent story. Timeliness concerns whether observations are available when supervisory decisions are made. Provenance concerns whether a result can be traced to its recorded source and transformations.
Many errors survive schema validation. A count of 400 complaints is a valid integer and may reconcile to the reporting database while excluding all complaints resolved at first contact. A rate of 10% is arithmetically valid while derived from incompatible event populations. A null code for vulnerability may be silently converted to absence. Semantic audits require walkthroughs of business processes, examination of mappings, and independent sampling of underlying events.
The return specification should separate hard errors from plausibility warnings. A negative count where only nonnegative cardinalities are permitted, an invalid product identifier, or a duplicate primary key is a hard error. A substantial deviation from last year, a very low complaint incidence, or a high withdrawal rate is a reason for inquiry. Some legitimate values become extreme during market shocks or changes in reporting scope. A warning can therefore be accepted with documented explanation, whereas a hard error requires correction or authorization under a defined exception mechanism.
The reporting system needs independent controls over aggregate construction. Record completeness can be assessed by comparing submissions with production-system extracts, accounting control totals, policy exposures, claim ledgers, customer interaction samples, and other authorized sources. A discrepancy is not automatically a reporting breach: two systems may legitimately have different time bases or definitions. Reconciliation should specify transformations that explain expected differences and isolate the unexplained residual.
A control report should show both numerator and denominator movements. Suppose complaints increase from 100 to 120 while exposure rises from 10,000 to 15,000 policy-years. Counts rise 20%, but the rate falls from 10 to 8 per 1,000 policy-years. A dashboard showing counts alone implies deterioration; a dashboard showing rates alone can obscure the additional operational workload. The appropriate interpretation depends on the question. A market-conduct investigation also needs to examine whether the reported fraction of relevant interactions changed.
Missingness and measurement error
Missing values and missing institutions create different problems. A present row with an unknown claim disposition can be treated as an unresolved classification; an entire absent return may remove the exposure and associated outcomes from the observed population. Data suppression means that a value was recorded but cannot be disclosed at the relevant granularity. Absence of an eligible record means an observation opportunity never entered the system. These situations require different assumptions for estimation.
Rubin's (1976) analysis of missing data explains conditions under which a missingness mechanism may be ignored in likelihood or Bayesian inference. Here, the relevant conclusion is that ignorability depends on assumptions about the observation process. It cannot be established from a low proportion of missing cells alone. Selection into complaints or interviews may depend on the underlying adverse experience even after adjusting for observed portfolio variables.
For a binary claim outcome, suppose 1,000 eligible claim records are known, 900 outcomes are observed, and 90 of those observed outcomes are adverse. The complete-case adverse proportion is 10%. If exactly 100 outcomes are unknown and no information constrains their dispositions, the overall adverse proportion is bounded by 90/1,000 = 9% and 190/1,000 = 19%. An assumed adverse proportion of 20% among missing outcomes implies 110/1,000 = 11%. These are arithmetic identification bounds for this specified population, not confidence intervals. A statistical model can narrow uncertainty under additional assumptions; it cannot make the missing outcomes observed.
Misclassification also produces systematic distortion. If T is the unobserved number of actual complaints and only a fraction q is recognized and entered, the expected recorded count in a simplified no-false-positive scheme is qT. If two institutions have the same actual complaint count but different q, comparing recorded complaints confounds underlying dissatisfaction and recording sensitivity. Where audit samples allow estimation of recording sensitivity, report its uncertainty; where such samples are unavailable, test scenarios instead of silently imposing q = 1. Introducing false positives requires a further parameter and suitable validation data.
Complaint coding changes should be versioned. If an earlier scheme treated renewal dissatisfaction as a general service issue and a new scheme has a separate premium category, category-specific changes partly represent recoding. Bridge tables or parallel classifications on a matched sample can quantify the break. Historical data should retain its original codes and the mapped category separately. An unbridgeable break should appear in the time series rather than be hidden through a spurious continuous trend.
Reporting governance and revision control
Supervisory data have a lineage from generation through regulatory use. A named owner should be responsible for each definition and reporting transformation, with a distinct reviewer for high-impact changes. Changes to population, formula, transformations, or source system should record effective date, reason, intended effect, regression tests, and transition policy. Once a supervisory decision has been taken on the basis of a particular return vintage, that vintage should remain recoverable even if a later correction is made.
The FCA (2026) 2025 value measures data notes distinguish a fixed publication snapshot from later material corrections and caution about revisions. This is an example of why a published series needs explicit revision status. An authority evaluating a previously issued alert should be able to reproduce its data as originally available, identify corrected records, and assess whether the decision remains supportable.
A return-validation process has several layers. Before submission, a firm validates record-level integrity, population coverage and aggregate reconciliation. The collection portal verifies schema conformance and required fields. Supervisory analysts examine plausibility and external consistency, and refer material anomalies for clarification. An independently selected case sample tests whether operational source records fit the declared definitions. Corrections are logged as revisions, not unexplained overwrites. Management attestation can assign responsibility but does not replace technical or substantive validation.
Data stewardship must also encompass reporting burden. High-frequency granular requests impose changes to systems, staff time, and validation effort; the benefits should relate to the speed and specificity with which the information can support a justified decision. A recurring field with substantial cost and no demonstrated analytical use is a candidate for removal or simplified aggregation. A rare but highly consequential harm may justify a targeted event-reporting requirement despite low volume. Proportionality is a practical design problem in addition to an institutional principle.
Confidentiality and lawful linkage
Detailed claim and complaint records may contain medical, financial, geographic, or other sensitive information. The authority should determine the legal basis and purpose of collection, minimize fields and access, segregate identifying information, encrypt transfers and storage, log queries, and apply retention and deletion rules. The IAIS ICP 2 confidentiality provisions treat the protection of confidential supervisory information as an institutional responsibility. Specific privacy duties and exceptions are established in the applicable jurisdiction and require separate legal assessment.
A pseudonymous join key can reduce the need to share direct identifiers across datasets. The mapping should be controlled separately, with role-based authorization and explicit rules for permitted joins. Pseudonymization does not guarantee anonymity: rare combinations of location, claim amount, and time can permit reidentification. The decision to publish a granular table should consider single-cell disclosure, combinations of tables, differencing between releases, and small intersectional populations. Suppression or aggregation protects confidentiality but changes the resolution of the available statistics; an authorized internal analysis may retain greater detail under safeguards.
Linkage error should be measured as a scientific issue. Unmatched cases may cluster among customers who changed contact details, used intermediaries, or filed disputes under variant names. A false match can assign an adverse experience to the wrong contract or person. Validation samples should estimate match precision and recall in relevant strata. A privacy-preserving linkage procedure that omits match-quality evidence can reduce personal data exposure while introducing unresolved analytical error. The appropriate design balances both purposes.
Indicator Estimation, Benchmarking and Calibration
Statistical estimands and peer populations
Calibration is the process of setting or revising the relationship between an indicator and a supervisory interpretation or action threshold. It begins with a clear estimand. For example, the conditional share of denied claims among closed episodes is a different estimand from the final rejection probability of newly reported claims. A count model for complaints requires exposure and an assumption about event generation. A severity measure requires coherent currency and price-date conventions, treatment of zero values, disclosure of tail observations, and a definition of paid versus incurred costs.
Peer groups should be selected using features that affect comparability and are relevant to the conduct question. Product line, coverage type, distribution method, exposure age, event severity, policy duration and jurisdiction can be legitimate stratifiers where data support them. A group defined too broadly creates confounding; one defined too narrowly produces sparse samples and unstable benchmarks. A benchmark designed to test distribution differences should preserve variation in distribution rather than mechanically condition it away. A benchmark testing claims operations conditional on comparable coverage and event type may reasonably adjust for portfolio composition.
Three weighting schemes answer different questions. The pooled ratio sum(X)/sum(N) gives the rate over all eligible observations and weights large institutions by volume. The unweighted average of firm rates describes a typical firm under equal weighting. The median firm rate is less affected by extremes but does not represent the experience of a randomly selected consumer. The benchmark and weights need to be identified explicitly. A dominant firm can shift a pooled benchmark toward its own performance; leave-one-firm-out or independently fixed reference populations can reveal that influence.
Consider a synthetic three-firm market. The first has 10 adverse outcomes among 100 observations; the second has 250 among 1,000; the third has 15 among 100. The pooled rate is 275/1,200 = 22.92%. The equal-firm mean is (10% + 25% + 15%)/3 = 16.67%, and the firm median is 15%. A policy that measures customer-level prevalence may use the pooled rate; an examination of the distribution of institutions may use the median or show all three. The difference between the benchmarks is justified by the different populations each describes.
Case mix and direct standardization
Consider a binary adverse outcome for low-complexity and high-complexity claims. The following illustrative counts describe two insurers; every claim is assumed decided on the same definition by the observation cutoff.
| Insurer | Claim complexity | Adverse outcomes | Eligible claims | Adverse proportion |
|---|---|---|---|---|
| A | Low | 9 | 100 | 9% |
| A | High | 144 | 900 | 16% |
| B | Low | 90 | 900 | 10% |
| B | High | 18 | 100 | 18% |
A has a lower adverse proportion within both strata. Its crude rate is 153/1,000 = 15.3%, exceeding B's 108/1,000 = 10.8% because its portfolio contains many more high-complexity claims. Using a common 50/50 distribution, A's standardized rate is 12.5% and B's 14.0%. This is an instance of aggregation reversal under unequal case mix. It shows that a crude difference can be reversed even when every stratum favors the other institution.
The example does not establish that complexity adjustment is always appropriate. If the classification of complexity is assigned after an insurer chooses to delay a claim, the classification may be influenced by the process under investigation. If a variable captures exclusion from services or a disadvantage associated with inaccessible procedures, adjustment can conceal conduct harm. Predefined variables should be temporally prior where feasible and justified against the substantive question. The unadjusted result should remain available beside the adjusted result, with a decomposition of differences.
For direct standardization, let p_ig denote the observed adverse proportion of institution i in stratum g and w_g a common reference weight, with nonnegative weights summing to one. The standardized rate is P_i = sum_g w_g p_ig. If a required stratum has no observations in an institution, direct standardization is not empirically identified for that stratum. Pooling, prediction, coarser strata or a common-support restriction may be used explicitly. A model prediction in an unobserved stratum depends on extrapolation assumptions and should be flagged.
Indirect standardization instead applies reference probabilities to the institution's actual population, obtaining E_i = sum_g n_ig p_ref,g. The observed-to-expected ratio O_i/E_i expresses the number of adverse outcomes relative to expected outcomes under that reference and is undefined where the expected count is zero. Its comparison across different institutional populations requires care. In particular, observed-to-expected ratios with differing weights and residual confounding do not necessarily have the same interpretation as directly standardized rates.
Sampling variation and rare adverse events
A firm may submit all its claims for a year, so the reported rate is a census description of those records. Statistical uncertainty arises when the objective is to infer a persistent underlying propensity, forecast future outcomes, assess a stochastic process, or generalize from a sample used to validate records. A confidence interval does not by itself measure misclassification, missingness, legal ambiguity, or unobserved selection.
For an independent binary process, a Wilson interval provides an alternative to the usual Wald approximation. Brown, Cai and DasGupta (2001), in Interval Estimation for a Binomial Proportion, show why normal-approximation coverage can behave poorly and recommend alternatives for small samples. The Wilson interval with observed proportion p, sample size n, and standard-normal critical value z has center c = (p + z²/(2n))/(1 + z²/n) and half-width h = z sqrt(p(1-p)/n + z²/(4n²))/(1 + z²/n). This expression is applicable to the specified binomial model, not directly to repeated-event complaint counts.
With 2 adverse outcomes in 20 trials and z = 1.96, the estimated rate is 10% and the Wilson interval is approximately 2.8–30.1%. With 200 in 2,000 trials, the estimated rate is also 10%, and the corresponding interval is approximately 8.8–11.4%. The two equal point estimates carry very different precision. A dashboard that treats them as identical-quality measurements conceals this distinction.
A one-sided approximate binomial screening statistic can compare X against the expected count np0 for a fixed benchmark p0. When observations are independent and the normal approximation is adequate, the standard deviation is sqrt(np0(1-p0)). A funnel display shows how the width of the screening region decreases with sample size. For extreme rates or small exposure, exact binomial tail calculations or an appropriately checked alternative are preferable. When many firms share catastrophe events, providers, common channels, or correlated claims, independence is inappropriate. A cluster-aware variance estimate, hierarchical model, or explicitly estimated overdispersion can widen the uncertainty interval and change the alert population.
Multiple testing also matters. If 100 independent valid tests are conducted at a 5% significance level under true null hypotheses, the expected number of false positives is five. Dependence changes the distribution of that count, not its expectation when each marginal null test has size 5%. False-discovery procedures, stricter escalation criteria, or confirmation on independent data can manage multiplicity; supervisory triage may also tolerate initial false positives where investigation cost is modest and the potential harm is substantial. The action threshold and the statistical false-positive rate are different objects.
Partial pooling and temporal benchmarking
A hierarchical model can stabilize rates in small populations while retaining institution-level variation. For independent binomial observations with a beta prior of parameters a and b, the posterior mean is (x+a)/(n+a+b). This conventional conjugate calculation illustrates the effect of partial pooling; hyperparameters should be estimated or justified separately, preferably using training periods and relevant peer groups rather than future observations.
With a synthetic Beta(2,18) prior, representing mean 10% and strength 20, a firm recording 2 adverse events among 5 eligible claims has an observed 40% rate but a posterior mean of 4/25 = 16%. An insurer recording 80 among 200 has the same crude rate and a posterior mean of 82/220 = 37.27%. The stronger evidence of the larger population reduces the influence of the common prior. A model can hide a serious issue in a small firm if the common prior is inappropriate. Show the raw counts, estimate, uncertainty, prior sensitivity, and substantive exception flags together.
For multi-jurisdictional or multi-product data, a logistic mixed model can express the conditional probability of an adverse event as a function of measured covariates and institution-level effects. Cluster and time effects can be included where justified. Such a model assumes some structure of exchangeability; the population from which random effects are drawn must be credible. A Poisson or negative binomial count formulation with a log-exposure offset may be suitable for complaint events, subject to exposure accuracy and model diagnostics. None of these formulations is a universal threshold or a new theory of conduct.
Temporal benchmarks should distinguish seasonal cycles, short-run shifts, structural breaks, and long-run trends. A seasonal catastrophe-related spike may be expected, and still create a genuine urgent service need. A system migration can create an artificial break in event coding. A regulatory change can alter reporting expectations, eligibility, or underlying behavior. Store versioned benchmarks and preserve the vintage used for earlier decisions. Backtesting alerts on revised data without acknowledging what was known at the original decision date introduces hindsight bias.
Thresholds, economic impact and statistical significance
A screening threshold can be an absolute performance benchmark, a deviation from peers, a change relative to an institution's own history, a legal trigger, or a prediction-based priority criterion. Each has a different justification. A statutory maximum processing period, where established, is a compliance condition; it should not be relaxed solely because peers also perform poorly. An empirical percentage threshold might flag relative outliers; it may shift as market conditions and reporting distributions change. A sudden change from an insurer's own baseline can reveal deterioration even when the firm remains near the market average.
Consumer impact can be estimated separately from statistical evidence. An excess adverse-event count of X - E is a descriptive comparison to a specified reference, not a causal estimate of preventable harm. If an investigation establishes that a defined subset of events is attributable to a prohibited practice and determines appropriate remedies, harm calculations can incorporate benefit shortfalls, time, interest, nonmonetary effects and uncertainty. Before that assessment, the quantity is a prioritization input. A high-consequence event may deserve immediate examination at low incidence, while a statistically significant but economically minor deviation may be handled routinely.
Benchmark revision requires an auditable decision. Define the population, period, exclusions, data vintage, risk adjustment, model parameters, threshold, expected alert volume, and review date. Evaluate the proposed threshold against suitably adjudicated outcomes where available; simulate sensitivity to drift, missingness and portfolio growth. Maintain the previous threshold for historical interpretability and avoid using revised criteria to imply that historical decisions were made under a rule that did not yet exist.
Cross-Jurisdictional Standardization
Common constructs and local institutional meaning
International comparisons are scientifically valuable when they establish which populations and processes are genuinely comparable and disclose the differences that remain. The IAIS principles describe supervisory objectives at an international level. Jurisdictions operationalize them through differing legal powers, institutions, filing specifications, and redress mechanisms. This diversity creates informational complementarity rather than a readily pooled global table of equivalent conduct rates.
The institutional examples examined here can be compared by their principal reporting function.
| Institution or instrument | Main observation context | Strength for comparative study | Major comparability limit |
|---|---|---|---|
| IAIS ICPs and 2022 survey | International supervisory principles and reported authority practices | Common objectives and documented variation in indicator use | Survey period and differing mandates |
| NAIC MCAS | United States insurer returns by line and participating jurisdiction | Defined variables, state and national scorecard architecture | Line-specific rules, participation changes and event conventions |
| FCA SUP 16.27 and Consumer Duty | Relevant UK retail product reporting and firm outcome monitoring | Product-value measures, consumer-outcome lens | Reporting thresholds, event definitions, product scope and separate firm-specific duty monitoring |
| EIOPA retail risk indicators | European market monitoring, partly using prudential information | Comparable market-level signals and broader evidence combination | Prudential line aggregates do not isolate every consumer product |
| ASIC internal dispute resolution | Australian financial-firm complaint records | Structured complaint issues, handling and remedies | Recording culture, sector and product coding, consumer selection |
| Financial Ombudsman Service | UK external complaints | Independent dispute signal and adjudication outcomes | Referral selection, different arrival and resolution cohorts |
The NAIC 2025 MCAS filing resources document New York's first participation for that data year. A national trend spanning the change requires a consistent-jurisdiction panel or an explicit decomposition of changes in reporting coverage and changes in activity. A new jurisdiction entering a return can shift the measured market distribution even if every previous participant's conduct is unchanged.
EIOPA's (2021) retail risk indicator methodology treats prudential reporting as a source of screening information while identifying limits of product granularity and the need for complementary context. The implication is that a prudential loss or expense ratio can flag a possible conduct issue, but the observation population must be translated before conclusions about retail customer outcomes are drawn. Aggregates over commercial and retail contracts can respond to changing portfolio mix rather than altered retail treatment.
EIOPA's (2025) consumer trends heatmap also identifies survey input as part of its evidence base. Surveys can reveal reported experiences and perspectives outside administrative returns; their coverage depends on sampling design, response mechanisms, definitions and the population represented. Administrative observations and survey answers are different measurements; convergence can strengthen a supervisory hypothesis, while divergence can identify underreporting, selection or an inadequately specified construct.
Harmonization through explicit mappings
A useful harmonization dictionary contains a concept identifier, a local field name and version, the source return, legal and product scope, observation grain, inclusion and exclusion rules, reference clock, unit, aggregation function, treatment of missingness, transformation, validation rule, and comparability class. A comparison can be exact, equivalent after a documented transformation, approximate with stated limitations, or unsupported because essential information is unavailable. These classes describe evidence for a mapping, not a numerical scale of institutional quality.
For example, a local complaint-per-policy measure can be transformed into complaints per 1,000 policies by multiplying the rate if the underlying denominator population and time basis are identical. Transforming an end-year stock into policy-years requires exposure history or a justified approximation. Mapping closed-without-payment directly to denied claims requires category information that separates nonpayment reasons. Mapping ombudsman uphold rates to insurer rejection performance requires matched eligibility, linked cases, and the process of referral, none of which can be obtained from general public aggregate rates alone.
Monetary comparisons require consistent treatment of currencies, inflation, date conventions, taxes and fees, and coverage. A currency conversion can transform nominal amounts on a specified date or using a specified averaging scheme, but it does not make benefits equivalent in purchasing power or contract scope. Sometimes a ratio can remove currency units, while retaining differences in time basis and benefit definitions. Comparability requires examination of the causal and accounting processes, not just standardized numerical units.
Legal status, burden, and cross-border information
Legal bases for routine filings, compulsory inquiries, processing of personal information, sharing with other authorities, and public disclosure vary. A global framework can define common reporting concepts yet permit national returns to retain locally necessary variables, thresholds, or legal exclusions. Implementers need to record which authority can collect each field, which entity must submit it, when the obligation applies, whether joint or group-wide reporting is permitted, and the confidentiality restrictions governing dissemination.
Cross-border insurance creates particular challenges. A claim may be administered in one jurisdiction under a policy sold in another, with an intermediary elsewhere and reinsurance that does not itself establish the insured's consumer relationship. Jurisdiction identifiers should distinguish place of risk, domicile, residence where relevant, licensing authority, sale geography, and the competence of the supervising authority. An aggregate group return and separate legal-entity returns may overlap; ownership and legal identity do not automatically make the records additive.
A harmonized core accompanied by local extensions facilitates comparison without flattening legally meaningful distinctions. Governance should provide procedures for proposing new fields, revising definitions, conducting impact assessment, consulting stakeholders, testing implementation and assigning versions. The central dictionary can preserve historical mappings and mark discontinuities. Jurisdictional differences that cannot be reconciled are scientific findings to be retained, rather than missing work to be concealed by imputation.
Statistical Validation and Supervisory Use
From a screening discrepancy to an evidential record
Indicators prioritize finite supervisory resources. The initial stage can combine magnitude, frequency, trend, data-quality issues, historical context, number of potentially affected customers, and severity of outcomes. An anomalous result can produce a request for explanation, a targeted data call, a review of definitions, examination of relevant customer files, or a supervisory meeting. The decision must reflect the authority's legal powers and procedural duties.
In the UK the FCA Consumer Duty monitoring rule addresses identification of risks to consumer outcomes. A supervisor reviewing an insurer's monitoring should examine whether the organization's governance information supports recognition and correction of poor outcomes, including where the aggregate market indicator looks ordinary. The reporting regime and the obligation to monitor outcomes are related but distinct: submission of a completed return does not establish the quality of the firm's internal oversight.
An analytical dossier for an alert should contain the exact indication, numerator, denominator, data vintage, peer selection, time window, uncertainty, alternative explanations, and evidence required to distinguish those explanations. Suppose an insurer's adjusted claim denial rate has increased. Evidence requests might target changes in contract terms, claim types, decision instructions, claims-system migration, outstanding inventory, reporting rules and sampled correspondence. A lawful finding about unfair treatment would require the applicable standard and the facts of the relevant practices, not merely the existence of an outlier.
An intervention may itself change the indicator-generating process. A firm asked to improve complaint recognition may record more complaints during implementation. A firm reducing repeat contacts may show lower complaint counts even if the policy-level incidence of dissatisfaction is unchanged. A portfolio suspension can reduce premium while previously incurred claims remain payable. Supervisory monitoring must distinguish the effects of improved measurement from improvements in conduct.
Validation design and decision calibration
Construct validation asks whether the indicator corresponds to the aspect of conduct under study. Content validity requires coverage of relevant lifecycle events, and criterion validation compares the indicator with an independently established measure such as audited file outcomes or verified remedial findings. Predictive validation examines how well the indicator forecasts a subsequent defined outcome. Each validation design needs explicit definitions and credible adjudication.
If the objective is to rank firms for investigation, an independently assessed target might be a material conduct issue established in a standardized examination. Supervised models trained on previous investigations face selection: examined entities are usually selected because they looked risky. Unexamined entities are not confirmed negatives. A validation program can include a probability sample outside the alert population, document examination intensity, and account for selection where feasible. A model fitted only to historic enforcement actions may reproduce institutional priorities, resources and enforcement practices instead of underlying harm.
An example illustrates decision calibration. In a synthetic validation sample of 1,000 independently assessed cases, suppose 100 have a verified issue. At threshold A, the screening system alerts on 160 cases, of which 80 have the issue. Sensitivity is 80%; specificity is (900-80)/900 = 91.11%; positive predictive value is 50%. At threshold B, 80 cases are alerted, of which 60 are true positives. Sensitivity falls to 60%, specificity rises to 97.78%, and positive predictive value rises to 75%. The stricter threshold halves investigation volume and doubles missed cases from 20 to 40. These trade-offs have practical consequences even though both thresholds can be described through familiar summary statistics.
For a predicted probability model, calibration in the statistical sense concerns agreement between predicted risk and observed event frequency in a suitable validation population. Discrimination concerns sorting high-risk cases ahead of low-risk cases. A model can discriminate well and produce systematically exaggerated probabilities; the reverse is also possible. Reliability plots, calibration intercept and slope, proper scoring rules, uncertainty, and temporal out-of-sample testing provide different information. A statistical calibration procedure should not be confused with selection of a policy threshold based on intervention cost and legal duties.
A threshold-selection exercise must define the target, evidence quality, harms of false positives, harms of false negatives, expected investigation capacity, special urgent categories, and requirements of applicable law. For a simplified expected-cost decision with a calibrated probability p, constant false-positive cost C_FP and false-negative cost C_FN, and zero cost assigned to correct classifications, flagging minimizes the modeled cost when p exceeds C_FP/(C_FP+C_FN). In real supervision the costs are not constant; serious risks can justify immediate attention and lawful duties may determine action independently of this simplified calculation. The expression is useful for exposing assumptions and not for replacing the authority's decision.
Model stability, independent challenge and gaming
Validation should be conducted against data not used to select the threshold or fit the model, with explicit temporal separation. If the system is intended to forecast a future verified issue, every feature must be available at the forecast time. Using a later known resolution, complaint outcome or future revision leaks information and overstates performance. Holdout periods should represent plausible future operating conditions; distinct product types and small populations need separate diagnostic evaluation when feasible.
Drift monitoring examines changes in inputs, reporting coverage, missingness, outcome prevalence, relationships between predictors and outcomes, and model performance. Increased alert rates may indicate a true market deterioration, greater recording completeness, or a model that no longer fits the population. A change in benchmark may similarly reflect market composition rather than conduct. Compare the current result with a stable baseline and the most recent version, and record which one was used for actual action. Trigger reviews on relevant changes, not simply elapsed time.
Independence of validation refers to freedom from the pressures to make the indicator appear successful. A data owner can validate source reconciliation, while a separate reviewer challenges definitions, assumptions, peer construction, thresholds, and unintended effects. Audit sampling should extend beyond favorable or easy-to-review cases. Documentation needs to describe rejected specifications and changes motivated by evidence, not merely the final result.
Measurement affects incentives. If a published metric penalizes countable complaints, firms may unintentionally or deliberately route dissatisfaction into categories that escape the complaint register; if performance is evaluated solely on closure speed, complex claims may be prematurely closed; if an acceptance metric excludes certain categories, coding may become consequential for the published rate. The architecture should retain primitive event data, closure histories, revision records, alternative measures, and independent sampling. It is important to distinguish intentional manipulation from ordinary operational and definitional differences before asserting misconduct.
Disclosure and follow-up
Public statistics can promote transparency and encourage improvement, but their usefulness depends on the presentation of coverage, uncertainty, exclusions and time basis. A scorecard band may reduce misleading significance attributed to small numeric differences. An aggregate rate may protect confidential information and describe market experience, while a restricted supervisory dashboard retains more detail for investigation. Public and internal purposes should each have a justified level of granularity.
A published statistical warning should make clear whether it concerns a reporting irregularity, a relative outlier, a verified customer outcome or an established breach. The information provided to firms and consumers should facilitate appropriate interpretation and avoid an implied ranking of contract quality across incomparable products. For a public correction, retain the edition and describe the material revision without disclosing restricted underlying records.
After a supervisory response, monitoring should assess the correction itself: whether the affected population was identified, appropriate redress was provided, the defective practice was changed, and similar harm recurs. A reduction in alerts is not sufficient if the reporting mechanism changed simultaneously. Compare with a stable control population where meaningful, examine before-and-after trends, and inspect customer-level evidence. Nonrandom interventions, anticipation effects, exposure changes and contemporaneous market events constrain causal inference. A controlled design, when feasible, can improve confidence; otherwise results should be described as changes observed following the intervention rather than definitive causation.
Conclusion
Insurance conduct supervisory data collection is the institutional and statistical infrastructure through which customer experience becomes available for regulatory examination. Its scientific foundation lies in the precise specification of observable events, eligible populations, time bases, reporting coverage, and lawful decision purposes. The comparable unit is a defined measurement of a defined population, rather than a shared indicator label.
The international evidence shows complementary ways of constructing this infrastructure. The IAIS provides cross-lifecycle objectives and records methodological diversity; United States market-conduct returns institutionalize line-specific reporting; the FCA connects retail value measures and firm-level outcomes monitoring; EIOPA demonstrates the analytical use and limits of prudential data for conduct surveillance; Australian internal dispute-resolution reporting and UK external dispute information illuminate the observation and selection processes behind complaint statistics. Their data are suitable for functional comparison and for specified harmonized constructs. Naive pooling of their published rates is scientifically unjustified.
The analytical demonstrations establish several consequences. Exposure-weighted and equal-institution measures answer different questions. Cohort and reporting-period outcomes can diverge when claims mature slowly. Case mix alone can reverse an aggregate comparison. A small observed proportion may have large uncertainty; partial pooling stabilizes estimates but can obscure exceptional cases when its assumptions are inappropriate. Missingness and misclassification alter interpretation independently of model-based sampling variation. Thresholds require calibration to investigation resources, consumer impact and legal relevance, together with independent validation of the underlying indicator.
The original contribution of this work is the integrated comparison of supervisory measurement choices across lifecycle events, statistical estimands, data governance and regulatory decision-making. Each analytical method is used within its established scope. The study establishes no new proprietary statistical model or legal doctrine; it provides a source-critical synthesis and transparent examples that allow the reader to examine the consequences of alternative measurement and decision choices. It concludes that an effective reporting architecture must preserve the evidence needed to move from a signal to a defensible assessment of the processes producing customer outcomes.
Future empirical research should test definitions against confidential or legitimately accessible matched records, measure complaint capture and linkage quality, evaluate the performance of calibrated alerts against independently adjudicated outcomes, and assess changes in customer treatment following specific supervisory measures. The results of such studies would extend the comparative and methodological evidence developed here. The present monograph's documented scope supports conclusions about measurement architecture and analytical validity, and does not assign empirical conduct rankings or infer the incidence of unfair treatment in an unobserved population.
Method and Reproducibility Appendix
Study design and source interpretation
The documentary corpus was selected for direct relevance to the scientific object. The source hierarchy distinguishes international standards, jurisdictional law or rules, regulatory reporting instructions, official published analyses, and statistical research. Reporting-instruction versions are distinguished from data years. Empirical figures are taken from institutional publications and retained with their stated scope. No underlying confidential files were available. The statistical examples are author-constructed deterministic illustrations, not estimates of real insurers.
Selection was purposive rather than a systematic review with bibliographic-database screening. Sources were included where they directly supported a definitional, institutional, empirical, or methodological claim. The evidence ledger identifies actual retrieved passages, including PDF text retrieval where a visual page request failed, and the abstract-only scope of Rubin (1976). The manuscript's methodological deductions should be read as conditional mathematical arguments applied to conduct reporting.
Reproducible numerical examples
All quantities in the following table are synthetic. The unit of exposure, timing and assumptions are stated in the relevant chapters. Percentages are rounded to a reasonable display precision.
| Example | Input | Reproducible calculation | Result |
|---|---|---|---|
| Growing exposure | 300 events; initial 10,000 and final 20,000 policies; linear growth | 1,000 × 300/15,000 versus 1,000 × 300/20,000 | 20 versus 15 |
| Cohort status | 70 paid, 10 rejected, 20 open | 10/80; (10+20)/100 | 12.5% decided rejection; upper 30% eventual rejection |
| Complaint reconciliation | 100/10,000 versus 120/15,000 | 1,000 × event count/exposure | 10 versus 8 |
| Missing outcomes | 90 adverse among 900 observed, 100 unknown | 90/1,000 to 190/1,000 | 9–19% identification bounds |
| Peer weighting | 10/100, 250/1,000, 15/100 | 275/1,200 versus mean of three rates | 22.92% versus 16.67% |
| Case-mix reversal | A: 9/100, 144/900; B: 90/900, 18/100 | Crude counts; 0.5 × each stratum rate | 15.3% vs 10.8%; standardized 12.5% vs 14.0% |
| Wilson interval | 2/20 and 200/2,000; z=1.96 | Center plus or minus half-width given in text | 2.8–30.1%; 8.8–11.4% |
| Beta-binomial update | Prior (2,18); 2/5 and 80/200 | (x+2)/(n+20) | 16%; 37.27% |
| Screening A | 80 true positive, 80 false positive, 20 false negative, 820 true negative | TP/(TP+FN), TN/(TN+FP), TP/(TP+FP) | 80%; 91.11%; 50% |
| Screening B | 60 true positive, 20 false positive, 40 false negative, 880 true negative | Same calculations as above | 60%; 97.78%; 75% |
The Wilson interval calculation takes the observed rate and a standard-normal critical value as inputs. For a count, zero exposure produces an undefined rate; a binary probability interval is calculated only for a positive integer number of eligible trials and a valid adverse-event count within that number. For a direct standardized rate, nonnegative common stratum weights must sum to one and be associated with observed or explicitly modeled stratum probabilities.
An operational data release supporting independent reproduction would also supply the signed data dictionary, versions of mapping tables and query code, eligibility and exclusion rules, data extraction date, immutable original and revised aggregates, the model specification and training window where applicable, the benchmark definition, and a disclosure assessment. Personal records and protected supervisory data need not be made public to document these methodological choices.
References
Australian Securities and Investments Commission (ASIC). (2024, December). Insights from internal dispute resolution data reporting: July 2023 to June 2024. Report 801. Source
Brown, L. D., Cai, T. T., & DasGupta, A. (2001). Interval estimation for a binomial proportion. Statistical Science, 16(2), 101–133. doi:10.1214/ss/1009213286. Source
European Insurance and Occupational Pensions Authority (EIOPA). (2021, December 9). Retail risk indicators: Methodology update. Technical note; PDF hosted by EIOPA. Source
European Insurance and Occupational Pensions Authority (EIOPA). (2025, December 17). Consumer trends in insurance and pensions 2025: Heatmap with key findings. Source
European Parliament and Council of the European Union. (2016). Directive (EU) 2016/97 on insurance distribution. OJ L 26, February 2, 2016, 19–59. Consolidated text, January 9, 2024. Source
Financial Conduct Authority (FCA). (2021, as amended). FCA Handbook, SUP 16.27: General insurance value measures reporting. Definitions in SUP 16.27.6 and reporting provisions. Accessed November 19, 2026. Source
Financial Conduct Authority (FCA). (2023, as amended). FCA Handbook, PRIN 2A.9: Monitoring of consumer outcomes. Accessed November 19, 2026. Source
Financial Conduct Authority (FCA). (2024, June 26; updated December 3, 2025). Insurance multi-firm review of outcomes monitoring under the Consumer Duty. Source
Financial Conduct Authority (FCA). (2026, July 21; updated September 11). General insurance value measures data 2025. Source
Financial Ombudsman Service (FOS). (2026). Annual complaints data and insight 2025/26. United Kingdom. Source
International Association of Insurance Supervisors (IAIS). (2022, June). Supervisors’ use of key indicators to assess insurer conduct. Public report, 22 pp. Source
International Association of Insurance Supervisors (IAIS). (n.d.). Insurance Core Principles and Common Framework for the Supervision of Internationally Active Insurance Groups: ICPs 2, 9 and 19. Online tool, accessed November 19, 2026. Source
National Association of Insurance Commissioners (NAIC). (2024). 2024 Market Conduct Annual Statement Ratios. Version 2024.0.1, updated August 6, 2024; 30 pp. Source
National Association of Insurance Commissioners (NAIC). (2025 data year). MCAS 2025 — Market Conduct Annual Statement. Reporting resources and 2026 filing calendar. Source
National Association of Insurance Commissioners (NAIC). (2026 data year). MCAS 2026 — Market Conduct Annual Statement. Reporting resources, accessed November 19, 2026. Source
National Association of Insurance Commissioners (NAIC). (2026 data-year resource). Private Passenger Auto Data Call & Definitions. Resource linked as 2026.2 on MCAS 2026 page. Source
National Association of Insurance Commissioners (NAIC). (n.d.). Market Conduct Annual Statement. Insurance Topics; 2024 filing snapshot as of September 2025. Source
Rubin, D. B. (1976). Inference and missing data. Biometrika, 63(3), 581–592. doi:10.1093/biomet/63.3.581. Source