RESEARCH DOSSIER
Statistical Cleaning and Reconciliation of Insurance Claims Data
Abstract. Insurance claims data are generated by transactions, administrative decisions, contractual interpretations, payment systems, and reporting routines whose units and time conventions differ from those required by statistical inference. A claim record may represent a loss occurrence, a notice, a coverage component, a reserve estimate, a payment, a recovery, a litigation state, or an updated version of an earlier record. Statistical cleaning is consequently an exercise in reconstructing the insurance meaning of observations, rather than a cosmetic operation on rows. This monograph examines the foundations, formal identification problem, methods, controls, and inferential consequences of reconciling claim amounts, duplicate identifiers, reopened cases, accounting dates, payment streams, case reserves, recoveries, and exposure denominators. It develops a traceable distinction between physical source records, adjudicated economic claims, longitudinal claim states, and analysis-specific statistical units. The discussion integrates probabilistic record linkage, temporal joins, event-sourced reconstruction, accounting conservation identities, missing-data mechanisms, actuarial reserving, predictive evaluation, and regulatory data-quality requirements. Three transparent synthetic calculations demonstrate how a duplicated transaction changes gross and net paid loss, how administrative duplication distorts frequency and severity in opposite directions, and how a misclassified development payment propagates through a chain-ladder estimate. The original contribution is an integrated, decision-oriented interpretation of established methods: every proposed transformation is assessed against an explicit estimand, an immutable audit trail, and independently checkable monetary and population totals. No novel named theory, empirical industry prevalence estimate, or proprietary-system validation is claimed.
Keywords: Insurance claims data; statistical data cleaning; reconciliation; record linkage; duplicate detection; claim reopening; payments; recoveries; case reserves; claim development; bitemporal data; data provenance; exposure matching; missing data; actuarial reserving; data quality; sensitivity analysis; statistical estimands.
Introduction
A statistically useful insurance claim is the result of an identification and valuation process. It does not appear in the insurer's information systems as an invariant natural unit. A first notice of loss may lead to one file, several coverages, multiple injured parties, dozens of payment transactions, revised estimates of outstanding liability, rejected invoices, recovery actions, and later reopening. Insurance systems therefore accumulate facts about claims through time, while actuarial models typically require one observation per specified unit, valued at a defined information cutoff. The transformation between these representations is a scientific part of the analysis.
The distinction has practical consequences for every major insurance statistic. Duplicate notices can inflate claim frequency; duplicated payment transactions can inflate total paid losses; unlinked recoveries can distort gross-to-net comparisons; a reopened file can be counted as an additional new claim; a payment posted after the valuation cutoff can leak future information into a reserving triangle; and an incorrect policy join can assign a loss to exposure that never covered the event. Even when a data extract satisfies elementary checks for nulls and data types, its analytic content may be invalid.
The actuarial profession's ASOP No. 23 on Data Quality (Actuarial Standards Board, 2016) calls for data selection, review, evaluation of material limitations, and disclosures proportionate to the actuarial assignment. It explicitly distinguishes a professional data review from a formal audit. The European Solvency II data-quality requirements (European Commission, 2015, Article 19) specify completeness, accuracy, appropriateness, and documented processes for data used in technical provisions. Both sources support a central methodological proposition: the meaning and adequacy of data depend on their intended use.
The research question is therefore precise: under what documented transformations do heterogeneous operational records become a valid sample for a specified insurance estimand? The answer requires a traceable chain from raw event to analytic claim; an explicit choice among occurrence, claim, coverage, claimant, invoice, transaction, and policy-period units; temporal consistency; reconciliation with independent totals; and a sensitivity analysis of unresolved decisions.
The evidence base combines official professional standards, the publicly accessible regulatory rulebook, publisher-confirmed statistical and actuarial research, and documented data-management methods. The record-linkage framework of Fellegi and Sunter (1969) supplies a formal approach to uncertain entity resolution. The data-cleaning classification of Rahm and Do (2000) explains why integration and schema transformation must accompany cleaning. Research on stochastic claims reserving, including Mack's distribution-free derivation (1993) and the analysis of England and Verrall (2002), establishes the statistical relevance of the claims development matrix on which cleaned transactions eventually act.
The method throughout is documentary analysis and formal reasoning, supplemented by invented numbers whose arithmetic can be reproduced without access to confidential insurer records. The examples are demonstrations of mechanism, never estimates of actual market behavior. National legal definitions of an insurance claim, permitted benefit, recovery or insured event differ; their contractual and statutory particulars must be recorded rather than inferred from a universal data schema.
Raw Claims Tables and Transactions
The claim as an object with several representations
An insurance event is a factual occurrence or series of occurrences that may engage contractual coverage. A claim is an administrative and legal demand or file associated with a reported event. A coverage component is a portion evaluated under an insurance provision, deductible, limit, or responsible party. A payment transaction records an authorized disbursement, adjustment, or reversal. A case reserve is an estimated outstanding amount associated with known reported circumstances. The same information system may use the word claim for more than one of these objects. Reconciliation begins by writing their distinct definitions and keys into the data contract.
A usable conceptual decomposition is:
The arrows describe possible administrative relations, not compulsory one-to-one mappings. A catastrophe occurrence generates many claims; one reported claim may cover several damaged property items; one motor accident may involve several claimants; one claimant may receive installments; one file may contain multiple coverage assessments. Deduplication at the event level therefore has a different purpose from deduplication at the payment-message level. The correct grain depends on the inferential target.
For an insurer's claims data environment, separate logical entities are useful even when a source vendor stores them in a single denormalized table:
| Logical object | Preferred key | Required temporal context | Typical measures | Principal counting risk |
|---|---|---|---|---|
| Loss event | event ID plus source provenance | occurrence date/time | event attributes, location | multiple notices of one occurrence |
| Claim file | persistent claim ID, entity namespace | notice, opening and closure dates | reported status, coverage decision | duplicate IDs after migration |
| Coverage component | claim ID plus coverage ID | effective dates and decision version | limit, deductible, share | summing component and parent totals |
| Payment or reversal | stable transaction ID plus version | economic, posting, value dates | signed amount and currency | re-imports, retries and voids |
| Recovery | recovery transaction ID | receipt and recognition dates | salvage, subrogation, ceded flows | missing or sign-reversed receipts |
| Case-reserve movement | claim/component ID plus sequence | effective and recorded times | outstanding level or change | adding snapshots as transactions |
| Policy exposure | policy/coverage-period key | inception, termination and amendment | insured units, earned exposure | unjustified many-to-many joins |
| Analytical observation | analysis-defined key | valuation and information cutoff | target and covariates | changing denominators or leakage |
A robust claim source retains both the operational claim ID and its source namespace. Identifier 12345 in two acquisitions need not denote the same claim. Claims transferred to a new administrator may acquire new identifiers; a crosswalk must distinguish migration aliases from genuinely independent files. Conversely, several payment references may legitimately belong to one claim. A unique constraint on claim_id in a transaction table would erase real payment history.
Three independent axes of identity, amount and time
Identity determines which economic entity a record describes. Amount determines what the record means financially. Time determines which state of knowledge the record expresses. The axes interact, but they cannot substitute for one another. Exact agreement of amount and date is weak evidence of identity when many benefit schedules pay standard amounts. Agreement of claim ID and transaction ID is stronger evidence when key governance is sound. A payment's bank value date, ledger posting date and accident date can each be correct while occupying different model periods.
A useful event record has the tuple
where is source, is a source-defined key, is record version, is semantic event type, is its effective or economic time, is record/ingestion time, is currency, is signed amount when relevant, and identifies the ingestion batch and transformation lineage. Not all operational sources possess every element. Missing elements are recorded as limitations rather than silently fabricated.
The event ledger is distinct from a current-state table. If a reserve snapshot is $9,000 at one date and $7,000 at a later date, those amounts represent successive valuations of one liability component; their sum is not a $16,000 expense. If a payment of $2,000 appears twice as an exact transport retry with the same authoritative transaction key, the second representation does not constitute a second cash outflow. If a transaction has an explicit posted reversal of −$2,000 and a corrected reissue of $1,800, all three events may be necessary to reconstruct the financial history, even though their net economic impact differs from a naive total of positive entries.
These distinctions reproduce the central integration problem described in Rahm and Do's data-cleaning survey (2000): single-source errors, multi-source conflicts, and schema transformations form one analytical problem. The ISO 8000-61 process reference model (International Organization for Standardization, 2016) is relevant as a general framework for organized data-quality management, although its public abstract does not prescribe an insurance claims data model or a universal reconciliation algorithm.
Source hierarchy and authoritative meaning
A production claims sample may be assembled from claims administration, payment disbursement, bank confirmation, general ledger, policy administration, reinsurance, third-party administrators, litigation management, and external loss-adjustment systems. They possess different authority for different questions. A bank settlement record may be strongest evidence that cash moved; a claim approval record establishes the purpose of the payment; a legal settlement document defines the obligation; the actuarial valuation extract supplies a particular time-stamped estimate; and the ledger determines how the insurer recorded the transaction in its accounting framework. No single source can authoritatively resolve every conflict.
The reconciliation design should specify, for each variable: operational meaning, source of authority, permissible aliases, precision and units, sign convention, valid ranges, effective-date logic, version precedence, correction process, aggregation rule, and documentary owner. This contract is particularly important when consolidating different insurers, lines, intermediaries, branches, currencies, and eras of administration software. A generic CSV column labelled paid cannot reveal whether the measure excludes allocated adjustment expenses, includes recoveries, uses gross or net reinsurance, or reports cumulative rather than incremental movement.
Conservation identities and the boundaries of equality
For a defined set of claim transactions through a stated valuation cutoff, gross paid loss may be reconstructed as
where disbursements are positive and valid financial reversals are negative. If actual recoveries are expressed as positive receipts in a separate table, then an explicitly defined net-of-recoveries paid measure is
This definition is not synonymous with net of ceded reinsurance: reinsurance recoverables and collections require separate legal and accounting classification. Deductibles, coinsurance, third-party recoveries and subrogation should not be collapsed into one unlabeled adjustment. Likewise, case incurred for an ordinary undiscounted gross loss basis may be represented as
where is the latest valid outstanding case-reserve snapshot at . It is neither the ultimate economic loss nor automatically the IFRS 17 liability for incurred claims. The latter operates under a wider accounting measurement framework, as the IFRS Foundation's IFRS 17 key-term explanations make clear.
For each equivalence claim there must be a basis: same portfolio, legal entity, time cutoff, claims type, currency, gross/net convention, loss adjustment expense treatment, recoveries, and recognition rules. A claim subledger equal to a bank statement in cash terms may not equal claims expense for that month because accrual, timing, and categorization differ. A failed equality is therefore a hypothesis requiring diagnosis, not immediate evidence that one source is erroneous.
Why descriptive completeness is insufficient
Data completeness must be evaluated against a relevant population. A dataset containing every row in one administration system can omit all business serviced externally or claims awaiting first notice. A transaction sample can be perfectly reconciled to a payment subledger yet miss case reserves needed for incurred loss analysis. A complete reported-claim file can omit incurred-but-not-reported obligations by construction. Under Solvency II Article 19, data used for technical provisions are assessed for the sufficiency of historical information and relevant homogeneous risk groups, as well as consistency and intended-use appropriateness. This implies that entity coverage, not merely null-field percentage, belongs in the quality assessment.
The first result of a defensible cleaning process is a documented population definition: covered contracts, insurers, geographical boundaries, lines, accidents, reporting periods, claim types, and monetary basis. The second is an immutable raw snapshot from which the current analytic representation can always be regenerated. Only after these are established does it make sense to optimize row-level cleaning rules.
Duplicated and Reopened Claims
Three different forms of duplication
Duplicate claim data arise at least in three ways. A duplicate transport record is a second delivery of the same underlying message, often after a retry or batch reprocessing. A duplicate administrative file represents the same economic claim under a new or parallel identifier, for example after migration or a repeated notification. A duplicate analytic observation occurs when a correct many-to-one source relationship is inadvertently expanded into multiple sample rows by a join or aggregation. Each case needs a different remedy.
Transport deduplication should use an authoritative message or transaction key plus source namespace and event version. Administrative-file reconciliation requires evidence about claimant, event, coverage, loss location, occurrence date, and migration history. Analytic duplication requires auditing the cardinality of joins, especially policy-to-claim, claim-to-payment, and claim-to-cover relationships. Applying a generic drop_duplicates() call to a fully denormalized table confuses these levels.
The classical Fellegi–Sunter linkage theory (Fellegi and Sunter, 1969) frames uncertain matching through agreement patterns between candidate record pairs. In simplified form, the weight for pair with comparison vector is
where denotes a true match and a nonmatch. Positive and negative evidence accumulate across attributes when the relevant probabilistic model is justified. The original method permits match, nonmatch, and review decisions. Modern implementations frequently adjust comparison distributions for dependencies, missingness, blocking, and class imbalance; a numerical threshold should be validated on representative adjudicated pairs rather than transferred unchanged from another portfolio.
The U.S. Census Bureau's Winkler record-linkage overview (2006) surveys developments and applications of statistical matching. For claims data, plausible blocking dimensions include insurer/legal entity, line, year and approximate loss date, with explicit exceptions for cross-border events or delayed reports. Blocking can dramatically reduce candidate comparisons, but false exclusion of a true pair from every block is an irrecoverable false nonmatch unless additional search stages are used.
Decisions must be attached to objects
Suppose notices A and B share an accident date and vehicle registration but concern different injured parties. At event level they may be associated with one occurrence; at claim-component level they are separate obligations; at transaction level their payments are distinct. Merging them as one claim because they share an accident would reduce claimant frequency and understate per-claim case complexity. Conversely, when a duplicate notice is adjudicated to be the same demand, counting it twice raises observed reported-claim frequency. Accordingly, linkage output should include match_level (event, claim, claimant, coverage, payment), decision (confirmed, probable, rejected, review), effective_basis, evidence, reviewer, and the surviving canonical identifier.
Matching is also a graph problem. If A matches B and B matches C, automatic transitive clustering can be dangerous: A and C may differ materially when pairwise rules allow weak links. Clusters should obey domain constraints such as incompatible contract periods or distinct injured persons. A stable canonical claim ID may be chosen without deleting source IDs. The crosswalk remains the record of how the administrative population was interpreted.
False matches and false nonmatches carry asymmetric costs. A false match can erase legitimate claim counts and combine unrelated severities; a false nonmatch can increase counts, duplicate monetary measures, or incorrectly split temporal histories. A risk-sensitive matching policy uses separate review thresholds for high-severity, litigation, cross-border, long-tail and multi-coverage cases. Sampling the manually adjudicated disagreements supplies estimates of precision and recall, but these are population- and sampling-frame-dependent.
Reopening is a state transition
A reopened claim is an earlier claim file that returns from a closed to an active state. Whether reopening creates a new reportable claim number depends on the source and the required reporting definition; analytically it ordinarily contributes a status transition and renewed payment/reserve exposure to the persistent claim's development history. A closure followed by reopening must not automatically be counted as two independent loss occurrences. The importance of this distinction is underscored by Lindholm and Zakrisson's claim-openness model (published online 2021), which explicitly models claim closing and reopening within reserving dynamics.
Let be a descriptive administrative state. A sequence might be reported → open → closed → reopened → closed; the model observation at cutoff is the last valid state after applying ordered events, but the event sequence itself is preserved. The count of distinct reports during a calendar year may differ from the count of open claims at year end, claims ever reopened, and claims receiving payment during the year. Each numerator answers a different question.
Reopening is often entangled with reserve and payment conventions. A claim closed with a zero reserve can subsequently reopen and incur an additional expense. If the cleaning rule filters any row with closed = true, the historical loss may disappear or later transactions become orphans. A state reconstruction instead maintains the claim in the cohort and attributes each new cash flow to its correct original occurrence year and payment calendar year, except where explicit contractual or statistical definitions direct otherwise.
Duplicate candidates are not invalid observations
Equal payments frequently arise from standard medical tariffs, recurring disability benefits, fixed indemnities, annuity installments, and split settlements. Therefore, amount/date equality is a useful anomaly signal but a weak deletion criterion. Conversely, the same bank transaction may appear with slightly different labels or timestamps in successive extracts. A reliable deletion or consolidation rule identifies the semantic invariant that the source system intends to preserve and checks whether canceling and corrected entries are linked by explicit reversal identifiers.
The decision record should be append-only. Each merge or suppression references the source records affected, pre- and post-cleaning keys, evidence and reason code. The raw payments remain in the archive; the analytic view suppresses repeated transport representations or expresses valid reversals in a conserved signed ledger. A reconciliation report can then separately count source rows, excluded duplicates, canonical claims, accepted payment events and exceptional unmatched records, making the interpretation accessible to auditors and analysts.
Estimating identification uncertainty
If matching is uncertain, a definitive single dataset can create false statistical precision. One practical sensitivity analysis constructs two internally coherent datasets from ambiguous cases: one conservatively retains possible independent claims; another merges only candidates that pass a lower threshold under documented review rules. Differences in frequency, total paid and target-model outputs are then reported as identification sensitivity. For a deeper probabilistic analysis, linkage uncertainty may be propagated across multiple plausible partitions, but model assumptions and computational expense must be stated.
The important conclusion is epistemic. Duplicate detection is a statistical inference about the identity of underlying obligations. The data engineer's chosen canonical key shapes the analyst's observed population. Confidence in downstream models cannot exceed confidence in the entity resolution on which those models depend.
Date, Payment, and Status Reconciliation
Accident, report, booking, payment, and knowledge time
At least five clocks can accompany a claim. Occurrence time assigns the risk to an accident or event cohort. Report time locates first notice and informs reporting delay. Booking or posting time records when an amount enters an accounting system. Effective or value time describes when a financial transaction is recognized according to a specified convention. Ingestion or knowledge time marks when a data processor first possessed a record. Additional dates include coverage start and end, reserve evaluation, closure, reopening, claim adjudication and settlement approval. A model that treats these dates as interchangeable changes the statistical object under study.
For reporting triangles, define accident period , valuation period , and development age under an explicit period calendar. Claim payments enter a cell based on their assigned accident cohort and eligible payment date. A reporting triangle instead records when claims become known, and an incurred triangle combines paid development and outstanding estimates. A dataset that uses last_modified_at to determine accident year will systematically reassign late-edited long-tail claims to recent cohorts.
A sound period calendar records calendar system, timezone, effective midnight rule, daylight-saving treatment, source precision, fiscal-period mapping, and whether a transaction is booked or economic-dated. For example, a payment instructed at 23:50 local time and posted at 00:10 the next day may fall in different months. Neither choice is inherently an error; the analyst must state the convention and reproduce the resulting balance bridge. Currency conversion similarly requires a conversion basis and date; gross amounts in mixed currencies cannot be compared or summed before unit normalization.
Bitemporal reconstruction
A later correction may say that the earlier economic fact was different from what the analyst knew at the time. A bitemporal perspective distinguishes the time when a fact applied (valid_time) from the time when the system recorded or learned it (transaction_time). This distinction is useful even where the source database does not implement a formal bitemporal schema. A retrospective estimate may use corrected knowledge, whereas a genuinely prospective backtest must use the information available at the historical decision date.
Represent the dataset as , where limits economic or valuation time and limits record availability. A historical prediction at decision date should use a measurable feature set built from information genuinely accessible by . If the feature table is refreshed today with a claim's final settlement code and joined to a prediction made two years ago, the resulting model is contaminated even if the correct accident date is retained. The predictive-modeling treatment by Kuhn and Johnson (2013) emphasizes disciplined preprocessing and data splitting; the insurance-specific extension is to enforce temporal availability within those splits.
The reconstruction rule should resolve superseded records by source sequence and knowledge cutoff, never by taking the lexicographically latest text field or the largest monetary amount. A corrected reserve snapshot may have an earlier valid date but a later entry date. An analyst selecting the latest recorded version for a past backtest would be using unavailable future information. The same record may appropriately produce two views: an as-reported historical view and a restated best-current view. These are separate datasets with separate purposes and labels.
Incremental financial movements and stock snapshots
Payments and recoveries are flows; case reserves are stocks at a valuation point. Combining them requires preserving this dimensional distinction. For a known claim component, let denote cumulative valid gross payments through and the selected valid outstanding case balance. Then is an elementary case-incurred measure. If the reserve source stores incremental adjustments , the final reserve can be reconstructed from an initial balance and its authorized deltas. If it stores absolute snapshots, summing snapshots is a category error.
An amount labelled incurred_loss may mean gross case incurred, net case incurred, loss plus expense, or a financial-reporting estimate. Cross-system reconciliation needs explicit, named transformations, such as:
The ordering is descriptive, and each adjustment must be separately defined. A deductible paid by the insured may reduce the insurer's cash payment without reducing the underlying event's physical loss. Similarly, a reinsurance recovery from a reinsurer is not the same as salvage recovered from damaged property. ASOP No. 43 on unpaid claim estimates (Actuarial Standards Board, 2007, revised 2011) explicitly asks the actuary to identify gross or net treatment, covered claims, types of adjustment expense, and the specific recoverables considered.
A fully worked transaction reconciliation
Consider the following wholly synthetic history for one reported property claim. All amounts are in USD; the original occurrence, coverage and claim identifiers are stable. The reserve on the cutoff date is assumed to be the latest valid gross outstanding case balance.
| Transaction date | Authoritative event | Signed gross loss cash flow | Collected recovery | Interpretation |
|---|---|---|---|---|
| February 3 | initial claim payment | +4,000 | 0 | valid disbursement |
| March 10 | erroneous supplemental payment | +1,500 | 0 | initially posted cash movement |
| March 11 | explicit reversal of March 10 | −1,500 | 0 | linked financial reversal |
| March 12 | corrected supplemental payment | +1,200 | 0 | valid reissue |
| March 12 | retry of same payment-message ID | 0 after deduplication | 0 | second representation only |
| April 2 | subrogation receipt | 0 | +300 | recognized recovery, separate flow |
| April 30 | latest case reserve snapshot | 0 | 0 | gross outstanding 2,500 |
The accepted gross cumulative paid amount is . Collected recovery is , giving a defined net-of-collected-recoveries paid amount of . Gross case incurred is ; net of only the already collected recovery, with the gross case reserve otherwise unchanged, the illustrative figure is . These amounts use distinct bases and should not be relabeled interchangeably.
If the ingestion retry is incorrectly treated as another valid +1,200 payment, gross paid rises to $6,400, net-of-recovery paid rises to $6,100, and gross case incurred rises to $8,900 with the same reserve. If the correction process instead drops the +1,500 payment but retains its −1,500 reversal, gross paid becomes $3,700. Both mistakes arise from treating related financial events as independent rows and demonstrate why a simple positive-payment filter is not a valid accounting transform.
A formal invariance test for a transport retry is
provided the adjudicated event mapping is correct and all source signs are transformed consistently. The equality is an audit target rather than a guarantee supplied by software. Bank-cash, general-ledger and claims-subledger totals may require explicit outstanding-in-transit, period-boundary or reclassification bridge items before such equality is expected.
Reopened status, claim reserves, and the valuation boundary
A reopened claim can receive a new reserve snapshot after an earlier closing reserve of zero. A state machine should preserve both the closure event and later reopening, reconstruct the reserve relevant at each cutoff, and retain the original claim cohort. A late reserve estimate must not be substituted into an earlier financial close if the purpose is to replay what was known then. Conversely, a retrospective restatement may deliberately incorporate corrections that were unavailable at the original cutoff; the distinction belongs in the dataset name and metadata.
Negative amounts require examination, not blanket removal. Valid negatives can arise from reversals, refunds, corrections, salvage netting, or reclassification. Invalid negatives can arise from reversed sign conventions, OCR or migration errors, or mixed expense and recovery fields. The analyst checks whether the source expects signed net flows, nonnegative gross flows plus separately represented recoveries, or absolute balances. A claim's ultimate amount may also be negative in a narrowly defined net settlement dataset; an automated clipping rule would suppress its economic meaning.
Currency and accounting reconciliation
For multinational portfolios, summing currency-denominated claims without conversion creates a dimensionally invalid result. A reproducible reporting currency transform needs the original currency code, numeric precision, transaction and/or valuation date, approved exchange-rate source, rounding convention, and whether the line is monetary, inflation-adjusted, or discounted. A currency correction changes both amount and its audit explanation. It cannot be quietly represented as a duplicate adjustment.
The IFRS Foundation's IFRS 17 accounting framework (2017, effective from 2023) measures groups of insurance contracts using fulfillment cash flows and other components as applicable. Therefore, a cleaned claim-level paid-plus-case field is an actuarial operational measure, while reported insurance liabilities can require discounting, risk adjustment, grouping, and additional cash-flow projections. The obligation to reconcile operational data with financial accounts does not imply numerical identity between these conceptually different measurements.
Data without precise dates
When date precision is limited, missing dates should be represented by ranges or an explicit imputation field with uncertainty, not fabricated noon timestamps. If only a month is known, the true event time falls within that month; assignment to accident-year totals may be stable while assignment to daily prediction features is uncertain. A conservative sensitivity test can place the ambiguous event at both extremes of its admissible range and measure changes in the relevant triangle or time-to-report distribution. Source-provenance notes must distinguish a missing source date, an invalid parser, an uncertain business rule, and a date intentionally withheld for privacy.
Claims and Exposure Joining
The population denominator is a scientific decision
Observed claim frequency is a ratio between a defined claim count and a relevant exposure measure. Its denominator may be insured vehicles, earned vehicle-years, policy-years, insured payroll, property values, benefit member-months or another line-specific quantity. A payment transaction is not an exposure unit, and a claim file is not a coverage period. Before joining, the analyst defines the covered population and target interval as carefully as the claimant identity.
For claim count , earned exposure , and gross aggregate loss on compatible bases:
where is a descriptive frequency rate, is mean loss per included claim, and is historical pure premium per exposure unit. These identities hold algebraically for the stated population when , but their statistical interpretation depends on claims maturity, censoring, coverage, truncation, inflation, and inclusion rules. The open actuarial Loss Data Analytics text (Actuarial Community, 2022) provides a broader educational treatment of loss modeling and applied statistical analysis.
Coverage joins and their failure modes
A claim must be linked to the appropriate contract, risk item, coverage and effective policy version. The matching rule requires both identifiers and time validity. For a loss occurrence at , the relevant coverage version may be the contract in effect at , while the premium exposure window uses earned coverage throughout a different interval. Endorsements retroactively changing insured interests, open-market policies, claims-made triggers, extended reporting periods, and delayed notifications make a single claim_date BETWEEN inception AND expiry test insufficient across all products.
A many-to-many relationship can be legitimate: one claim involves two coverages; one occurrence affects several insured locations; a liability claimant is associated with several defendants. However, naive joins multiply payment rows. If one claim payment of $5,000 is joined to three matching coverage records and then summed as $15,000, the economic total has been changed by relational duplication, not by an actual new transaction. Before monetary aggregation, a claim-to-coverage allocation rule must establish the appropriate split or select a canonical coverage. When allocation weights represent mutually exclusive shares of the same claim amount, require and document exceptions; when coverage records represent distinct obligations, the data model must not impose that identity artificially.
Every merge should report its cardinality: left-unmatched, right-unmatched, one-to-one, one-to-many, many-to-one, many-to-many, and duplicated keys after joining. Claim records without matched exposure are not automatically invalid; there may be canceled-policy tails, extended reporting periods, missing migration keys, or recoveries from historical cohorts. An omission decision must distinguish an orphan record, an uncovered loss, and a legitimate claim lacking a contemporary policy key.
A transparent frequency–severity demonstration
Consider a synthetic portfolio with 40 earned policy-years, five distinct valid claims, and $27,000 in total gross ultimate losses. The accepted descriptive metrics are five divided by 40 = 0.125 claims per policy-year, $27,000 divided by five = $5,400 per claim, and $27,000 divided by 40 = $675 per policy-year.
Now suppose an administrative duplicate of one already included $4,000 claim is erroneously treated as a sixth independent claim and its amount included again. The raw extract reports six claims and $31,000. Its apparent frequency is , an excess of 0.025, or 20% relative to the correct frequency. Its apparent severity is , approximately 4.32% below the correct $5,400; nevertheless its pure premium becomes , approximately 14.81% above the correct $675. An administrative error simultaneously increases observed frequency and total loss while decreasing average severity. The direction of a model's bias therefore depends on its specific outcome.
| Measure | Canonical five claims | Duplicate-counted six | Difference |
|---|---|---|---|
| Earned policy-years | 40 | 40 | 0 |
| Distinct claims | 5 | 6 | +1 |
| Gross ultimate losses | $27,000 | $31,000 | +$4,000 |
| Claims per policy-year | 0.125 | 0.150 | +20.00% relative |
| Average loss per claim | $5,400.00 | $5,166.67 | −4.32% relative |
| Loss per policy-year | $675.00 | $775.00 | +14.81% relative |
These figures are invented and independently reproducible. They illustrate propagation of a deduplication error through identities rather than a claim that any real insurer exhibits the same magnitudes.
Reporting delay, closed-claim sampling, and selective observation
At a valuation date, claims not yet reported cannot be directly observed in a reported-claim table. Claims reported but not settled may have censored severities. Restricting severity estimation to closed files selects claims on a post-occurrence process whose duration can depend on loss size, litigation, product and severity. The resulting sample can be biased toward faster-settling claims. Statistical cleaning cannot make such a selection representative merely by correcting syntax or merging identifiers.
Let indicate report by , indicate closure by , and represent ultimate loss. The average need not equal over all insured loss events. Their difference depends on reporting and closure mechanisms and on the chosen risk population. Rubin's missing-data analysis (1976) distinguishes conditions under which ignoring a missingness mechanism is valid. The Little–Rubin treatment (2019) develops likelihood-based and other approaches for missing data. These statistical frameworks illuminate why imputing a mean severity for every open claim can misrepresent outcome dependence and uncertainty.
Particular care is required when a recorded zero has several meanings: genuine zero payment, claim closed without payment, unpaid open claim, not-yet-populated field, coverage denial, offset by recovery, or a data export default. A zero is not automatically a missing value, and a null is not automatically zero. A principled cleaning process preserves the source's missingness and legal-status semantics before deciding how to model them.
Insurance-product distinctions
In property insurance, a single catastrophe may generate multiple claims while one insured building can have several coverage components. In automobile insurance, an accident can create bodily injury and property damage claims with different reporting and litigation pathways. In health insurance, one episode of care can yield multiple provider claims, reversals, adjustments and coordination-of-benefits transactions; claim-line versus episode-level units must be distinguished. In disability or long-term care insurance, periodic benefit payments create legitimate repeated amounts and claims often remain open over long durations. Agricultural insurance can involve a season, farm, crop and peril index whose exposure unit differs from the policyholder. These distinctions are governed by the product and its administrative data generation, rather than a single global duplicate threshold.
Consequently, claim frequency across lines cannot be pooled without standardizing the event definition, covered population, exposure, and reporting horizon. Pure premium should not be compared directly across currencies, changing coverage limits, inflation regimes or uncensored development without appropriate transformations and assumptions. The U.S. ASOP No. 53 guidance on future costs (Actuarial Standards Board, 2017) identifies historical claim adjustments, business mix, coverage, operational practice, accounting and reinsurance changes as relevant to prospective property/casualty estimation. These are precisely the changes that an apparently uniform historical claims dataset may conceal.
Privacy and permitted statistical joins
Claims data can contain identifiers, diagnoses, financial details, geolocation, disputes and sensitive household information. Reconciliation should use the minimum identifiers necessary for the defined legal and analytical purpose and protect crosswalks through access controls, retention rules and review procedures. Under the EU GDPR's data-processing principles (European Union, 2016, Article 5), purpose limitation, data minimization and accuracy constrain processing of personal data when the Regulation applies. Its legal scope and permitted bases must be assessed for the actual controller and jurisdiction; a global claims warehouse does not acquire an unrestricted legal basis merely by describing its use as statistical.
Privacy measures can alter linkage error. Tokenization that changes across source systems may block legitimate matches; excessive disclosure of direct identifiers raises unnecessary risks. A privacy-aware linkage design documents the permitted comparison fields, salted or privacy-preserving keys where lawful and technically justified, access controls for adjudication, and performance checks against appropriately protected validated match cases. Statistical validity and privacy compliance are related design requirements that need joint verification.
Audit of Cleaning Decisions
Reconciliation as a reproducible scientific argument
A cleaned dataset is a claim about how source records correspond to the target population. Its construction should therefore be reproducible and open to challenge. Every substantive transformation needs the raw input, business justification, rule version, affected records, exception path, count and amount before and after, approver where required, and the date on which the rule was applied. A sound audit trail allows the same historical snapshot to be reproduced, even when new corrections are received later.
The U.S. ASOP No. 23 data-review standard directs the practitioner to consider the intended purpose, internal consistency, data sufficiency and significant limitations; it does not substitute a full formal audit of source data. A scientific claims cleaning study should likewise distinguish tests performed by the author from validations performed by claims operations, accounting, internal audit and independent assurance functions. The existence of a data dictionary alone is not proof that its assertions are true in the underlying systems.
A useful auditable pipeline has the following sequence: register raw immutable extracts; profile source keys and monetary conventions; apply deterministic ingestion deduplication; resolve administrative identity with uncertainty labels; normalize event types and signed flows; reconstruct temporal state for each valuation; join to coverage and exposure with cardinality tests; reconcile balances and population counts; build target-specific sample; evaluate errors and sensitivities; then freeze the release candidate with checksums and data lineage. The steps can be repeated whenever additional data arrive, and the process must be stable under the same input and parameter versions.
Audit measures and exception rates
Quality metrics should describe failure categories and their denominators. Let denote all source rows in a defined intake batch, the number failing uniqueness constraints, the number with unresolved economic ownership, and the number with missing required fields. Then , , and are descriptive diagnostics, but the percentages are not interchangeable and overlapping categories need not sum to 100%. An insurer may have a very small exception rate by row count and a large exception rate by financial exposure if the unresolved records contain high-value liability claims.
For monetary reconciliation, define source total , transformed total , and documented bridge items on a matched basis. The residual is
For exact integer-cent, same-currency signed cash records under the same cutoff, is an appropriate acceptance criterion when all required transactions are included. For converted currencies, rounded published reports or actuarial estimates, an explicit tolerance may be reasonable. A tolerance must be justified by rounding and business significance, rather than chosen to force acceptance. The reconciliation report should show both raw residual and residual relative to a meaningful magnitude; a zero relative residual can be undefined when the base is zero, so the raw measure remains essential.
Separate at least four types of diagnostics: structural (schema, keys, relationships), temporal (ordering, cutoff and effective dates), financial (conservation and ledger bridges), and statistical (distributional shifts, population selection and model influence). A table may pass structural and financial checks while failing statistical relevance because the risk mix changed. Conversely, a legitimate catastrophe may produce an extreme loss distribution that is statistically unusual but financially supported and therefore must be retained.
Outliers, heavy tails, and erroneous amounts
Insurance severity distributions are often skewed, and extreme observations can be economically central. A maximum-value check that truncates every payment above a fixed percentile transforms the tail of the insured loss distribution. A high loss can be a genuine catastrophe, a currency conversion error, a limit error, a duplicate or a book entry. The appropriate response is to investigate source provenance, coverage and payment authorization; label verified extremes; correct demonstrated errors; and conduct analyses under stated winsorization, censoring or robust-estimation assumptions only when the estimand warrants it.
A useful diagnostic vector for a questionable monetary observation may include amount-to-limit ratio, currency and exchange-rate factor, payment versus reserve movement, duplicated payment identifiers, paid-to-date compared with case outstanding, reporting lag, loss type and known catastrophe identifier. Thresholds should be line-specific and time-aware. A high-value disability lifetime benefit and a high-value automobile glass claim have different baseline plausibility. Financial anomalies and statistical outliers are hypotheses for investigation, not automatic deletion orders.
Failure taxonomy and decision log
A transparent exception register can use compact, semantically defined classes:
| Exception | Detection evidence | Scientific risk | Permitted response |
|---|---|---|---|
| Replayed source message | identical authoritative message ID and version | duplicate transaction count and paid amount | suppress transport copy in analytic view; preserve raw |
| Migration alias | verified identifier crosswalk | duplicate claim frequency | merge canonical claim identity with lineage |
| Ambiguous matched pair | moderate probabilistic linkage score | false match or missed match | adjudicate or retain sensitivity alternatives |
| Contradictory reserve versions | overlapping effective intervals or sequence conflicts | invalid case-incurred snapshot | resolve authoritative version; document rule |
| Unlinked payment | no valid claim or component association | omitted or misallocated paid loss | quarantine for review; bridge financial totals |
| Out-of-period posting | event, booking or knowledge date mismatch | false cohort or data leakage | dual-date attribution under named convention |
| Ambiguous recovery sign | unclear source direction | gross/net reversal | obtain ledger meaning before transformation |
| Exposure join fan-out | multiple matched coverage rows | multiplied loss or biased denominator | establish allocation/ownership; rerun amount checks |
| Source date uncertainty | precision or timezone missing | invalid reporting lag | retain interval and measure sensitivity |
| Extreme credible claim | valid documentation, unusual severity | tail estimation distortion | retain, tag, model explicitly |
The list is an analytic classification for this monograph, not a newly named external standard. Materiality depends on use: one unmatched claim of $30 million can matter more for reserves than thousands of small stale notifications, while a cluster of duplicate low-cost claims can matter more for frequency models than for total losses. Portfolio-level error rates should be accompanied by weighted loss metrics and a breakdown by claims maturity, product, geography and source system.
Independent reconciliation layers
Reconciliation should be checked against more than one source where possible. A claims-system transaction extract can be compared to the claims subledger; the subledger to the general ledger; payment authorization to bank settlement; reserve movement to case-reserve snapshots; claim counts to reported-claim registers; and policy-linked claims to the exposure table. Agreement between two downstream systems that share the same upstream mistake does not provide independent confirmation. Effective control requires identifying which comparison sources have independent origin or processing.
Under the Solvency II documentation rule (European Commission, 2015, Article 265), the technical-provisions process documents data collection and analysis of quality, source and usage directories, assumptions, methods and validation. EIOPA's guidelines on valuation of technical provisions (2022 revision) address, among other topics, data deficiencies and adjustments. Their application depends on the EU supervisory context and is not an automatic legal mandate for every jurisdiction in this international encyclopedia.
Governance of transformations and version control
A cleaning rule should have an owner, effective date, tested examples, permitted input schema, assumptions, explicit output contract, failure mode and version. The output should record the exact hash of the transformation code and input snapshot where tooling supports it. Analysts should be able to regenerate prior valuation views rather than overwrite them. Human adjudication of material identity conflicts should be recorded as a typed decision applied by the pipeline on subsequent runs; this is preferable to an untraceable manual spreadsheet deletion.
The process should also distinguish error correction from assumption-driven statistical adjustment. Replacing USD mistakenly recorded as JPY following bank evidence is a correction of source meaning. Inflating historical claim amounts to a future cost level is a model adjustment. Imputing an outstanding amount from similar claims is an inferential procedure. Applying a cession percentage to gross claims is a contractual/accounting transform. All may be legitimate within a project, yet each changes what the resulting number represents. One generic cleaned_amount field conceals these epistemic differences.
Regulated and professional communication should report material changes and limitations in language appropriate to its audience. The ASOP No. 41 communication requirements (Actuarial Standards Board, 2010) provide a relevant U.S. professional framework for documenting actuarial results and disclosures. The author of a scientific dataset must likewise state what was checked, what could not be independently resolved, and which decisions may affect readers' interpretation of estimates.
A reproducibility checklist expressed as data assertions
The following assertions provide a compact computational audit without purporting to establish universal thresholds. A pipeline may require: raw file hashes unchanged; all source systems enumerated; stable unique event IDs under a defined namespace; no unintended many-to-many financial fan-out; each valid reversal linked to its underlying movement or documented as unresolved; currency and sign known for every amount entering a sum; valuation state selected by effective and knowledge cutoff; monetary bridge residual within its justified tolerance; analysis-specific sample count reproduced; externally reviewed exceptions retained; and outcome sensitivity assessed when uncertain records are material.
Such controls distinguish scientific reproducibility from an attractive summary dashboard. A final approved dataset is one whose records, transformations and limits can be traced and whose statistical implications have been examined. It remains a versioned representation of underlying events; subsequent claims development may legitimately change its future values.
Downstream Modeling Sensitivity
Cleaning changes the estimand's empirical support
The importance of cleaning becomes measurable when models are fitted to claims data. A frequency model observes a count process; a severity model observes amounts conditional on claim inclusion; a pure premium model observes total loss per exposure; a reserving model observes development patterns by cohort and time; and an operational model may predict settlement duration or reopening. Each target uses a different projection of the underlying claims history. A cleaning rule can improve one projection while leaving another unchanged or worsening it.
Suppose a duplicate payment is detected and removed from a claim's transaction ledger. The number of canonical claims may be unchanged, while the paid severity and incurred triangle decrease. Suppose instead that one duplicate claim file is merged with another and its legitimate separate payment events are kept. The claim count decreases while total paid loss remains unchanged; hence severity per claim increases and pure premium remains unchanged. These two corrections must not be conflated. A third case is a duplicated file with copied transaction records: merging both claim identity and redundant payments changes counts and totals. This is why a general phrase such as deduplicate claims is insufficient to describe a statistical pipeline.
For any statistic computed on dataset , a useful documented influence measure is
The difference is descriptive and conditional on the chosen cleaning decisions. It is not proof that either dataset equals an unobservable ground truth. Analysts should calculate the impact by error class, cohort and line, and report alternative plausible corrections for ambiguous cases. When a transformation modifies the observation mechanism or conditional sample, sensitivity must be assessed at both statistic and fitted-model levels.
Reserve triangle construction
For accident year and development age , let denote cumulative paid loss as of that development evaluation, formed from accepted transactions assigned to the corresponding accident cohort. The cumulative chain-ladder development factor can be estimated for mature cohorts by
where comprises accident years observed at both ages, and the development ages are consecutive within the chosen triangle. Estimated ultimate loss for a cohort at latest observed age is
with an explicit terminal age and any selected tail factor. This estimator depends on a coherent cumulative triangle, accident-period assignment, and model assumptions about development. Mack (1993) derives standard errors under distribution-free chain-ladder assumptions; England and Verrall (2002) discuss stochastic models and predictive uncertainty. Neither source implies that the numerical triangle will be correct without a separately validated data-generation and reconciliation process.
A worked triangle with one erroneous transaction
Consider three synthetic accident years and annual paid development, with monetary units in thousands of dollars. The clean cumulative triangle is:
| Accident year | 12 months | 24 months | 36 months |
|---|---|---|---|
| A1 | 12 | 18 | 21 |
| A2 | 15 | 22 | — |
| A3 | 20 | — | — |
The age-12-to-24 factor is ; the age-24-to-36 factor is . For A2, projected ultimate is . For A3 it is . The latest clean observed sum is ; projected ultimate across A1–A3 is . Consequently the simple aggregate unpaid development estimate implied by these illustrative data is about thousand dollars, with no tail factor beyond 36 months.
Now suppose a payment of 2 (thousand) in A2 at the 24-month development evaluation is represented twice in the ingest and is accepted as a distinct economic disbursement. A2's 24-month cumulative value becomes 24 while A1 is unchanged. The apparently measured 12-to-24 factor becomes . The mature 24-to-36 factor stays at . The projected ultimate for A2 becomes , and for A3 becomes . The sum of the latest observed values rises from 63 to 65; aggregate projected ultimate rises to . The projected unpaid amount becomes approximately , an increase of about thousand, despite the original duplicate of only 2 thousand.
| Derived measure (thousands USD) | Correct reconciliation | Duplicate +2 at A2/24 | Difference |
|---|---|---|---|
| Development factor 12–24 | 1.481481 | 1.555556 | +0.074074 |
| Development factor 24–36 | 1.166667 | 1.166667 | 0 |
| A2 projected ultimate | 25.666667 | 28.000000 | +2.333333 |
| A3 projected ultimate | 34.567901 | 36.296296 | +1.728395 |
| Total projected ultimate | 81.234568 | 85.296296 | +4.061728 |
| Latest cumulative observed | 63.000000 | 65.000000 | +2.000000 |
| Projected unpaid development | 18.234568 | 20.296296 | +2.061728 |
The result matters because the error enters not only the directly affected cohort but also a development factor applied to a younger cohort. Its influence depends on volume, maturity, other observations and the selected reserving method; another triangle could yield a different direction or scale. All calculations assume fixed selection of cohorts, no tail, no trend, and standard chain-ladder averaging. This arithmetic demonstrates mechanism but does not estimate a commercial reserving error distribution.
Statistical uncertainty versus data-reconstruction uncertainty
Model variability and data uncertainty have distinct origins. A stochastic reserving method may quantify process variation and parameter estimation conditional on the supplied triangle. It does not automatically incorporate uncertainty about which transactions were valid, which claims are duplicates, or how historical system migrations were resolved. Data-reconstruction uncertainty can be assessed by calculating results for a defined set of plausible verified or unresolved cleaning decisions. If those choices dominate the range of outcomes, a narrow conditional confidence interval from the statistical model is misleading as a statement about end-to-end uncertainty.
A comprehensive report can separate: process uncertainty in future loss emergence; parameter uncertainty of the fitted model; model-form uncertainty from selecting development and frequency structures; and reconstruction uncertainty associated with identity, classification, cutoff and exposure. These are practical uncertainty categories for analysis, not a newly proposed universal statistical doctrine. Combining them rigorously requires specifying dependencies; simply adding percentage margins may overstate or understate aggregate variance.
Leakage and retrospective revision
Operational claims histories are revised. If a model is backtested on retrospective tables after all claims were settled, fields such as final paid loss, closure date, litigation outcome or last known reserve may reveal information not available at the prediction date. A model then appears more accurate than an implementable real-time system. The appropriate test reconstructs historical features as of each evaluation date and evaluates the outcome at the specified future horizon. The Kuhn–Johnson preprocessing and data-splitting framework (2013) supplies relevant general predictive practice, but a claims model additionally requires adjudication of backdated transactions and later corrections.
Random train–test splitting at the transaction level can place multiple rows from one claim in both training and test sets, producing dependence and potential leakage. Split policy, claim or event groups according to the estimand and intended deployment unit; add temporal separation where future deployment is the goal. If the ultimate target is learned from a later valuation of the same claim, do not present it as information observed when the model was supposed to be applied. Cleaned sample versions must carry an explicit as_of and a known_by constraint.
Missingness, payment delays, and censoring
For paid claims at development age , the analyst observes partial cash histories. A severity model restricted to claims with final settlements may exclude complex or long-tail losses. Modeling the eventual ultimate outcome requires methods appropriate to censoring, reporting and settlement delays; naive zero replacement for unknown futures yields structural downward bias in many such settings. Rubin's missingness conditions (1976) and Little and Rubin's comprehensive statistical treatment (2019) make clear that missingness assumptions matter to valid inference. The claim-specific observation mechanism and contractual obligations must be added to these general statistical conditions.
For example, an open file with no payment at month six may represent a high-severity liability dispute, a legitimate no-loss claim, an administrative delay, or a reporting placeholder. Setting all four to zero as observed final severity would destroy their differences. The correct cleaning output distinguishes observed paid-to-date, current case reserve, status, reporting age, adjudication stage, and whether ultimate severity is censored. Downstream models can then use a framework that accounts for the relevant observability rather than hiding it in an engineered number.
Diagnostics beyond the final prediction score
The primary model report should quantify how cleaning decisions alter target distributions and predictive performance. For each correction class, compare sample size, claim-event count, paid and incurred totals, average and upper-tail severity, reporting delay, settlement duration, status mix, recoveries, exposure denominator, temporal cohort composition, and downstream estimates. For predictive models, track out-of-time calibration and error across relevant segments, with uncertainty reflecting clustering or dependence when appropriate. For reserving, compare triangles, factor selections, holdout forecast errors and sensitivity to reopening and large claims. The outcome metric must be relevant to the intended decision; no generic accuracy score can replace a correctly defined target.
A clean-looking curve may be suspicious when unusual losses were systematically removed. A stable overall average may disguise increased underrepresentation in a small line or region. Data-quality review therefore benefits from stratified comparisons and an explicit exception ledger. The cross-disciplinary study of data cascades in high-stakes AI (Sambasivan et al., 2021) concerns broader machine-learning systems, not insurance-specific empirical incidence; its documented account of downstream effects from data practices illustrates why quality decisions deserve first-class attention.
Materiality and model use
Whether a residual data defect is tolerable depends on the use. A reserving analysis concerned with gross unpaid losses may be highly sensitive to missing large outstanding claims and comparatively insensitive to a duplicated zero-payment notice. A frequency tariff may be sensitive to repeated low-severity claim files even if the total financial impact is small. A fraud-screening model may be sensitive to falsely merged identities and unreviewed adverse labels. A consumer-outcome study may be sensitive to subgroup-specific incompleteness masked by aggregate averages. Decision-oriented quality assurance should evaluate the consequences of defects for the particular target rather than declare all datasets uniformly clean or unclean.
ASOP No. 43 advises the actuary to consider the nature of claims, characteristics of data, methods and assumptions in estimating unpaid property/casualty claims. Solvency II Article 121 specifies accurate, complete and appropriate data for an insurer's internal model in its applicable supervisory context. These sources support the general scientific need to evaluate data in relation to the calculations they are intended to support, while their professional and legal scopes remain distinct.
Original synthesis and research agenda
The analysis yields four connected conclusions. First, source records and economic claims occupy different grains; errors in the mapping propagate mechanically into statistical observations. Second, time and financial bases must be reconstructed explicitly; an apparently consistent set of amounts may encode the wrong date or accounting meaning. Third, cleaning uncertainty is inferential uncertainty; a single final data table is inadequate when material identity and classification cases remain unresolved. Fourth, release quality requires evidence of conservation, valid sample selection and downstream sensitivity, not merely a passed schema validator.
These are analytical conclusions drawn from established record-linkage, data-quality, accounting and actuarial literature, supported by the synthetic demonstrations in this monograph. They are not named new theories. A future empirical program can test them using legally accessible, de-identified transaction-level data with source-of-truth adjudication and an agreed historical snapshot protocol. It should measure how different linkage decisions alter claim counts and tails, compare as-known and restated backtests, estimate reviewer disagreement, and examine reserve and price sensitivity across different insurance products. The minimum evidence for empirical novelty would include a preregistered or transparently specified target, representative data, independent adjudication, reproducible code, uncertainty intervals and an explicit statement of where findings do and do not generalize.
Conclusion
Statistical cleaning and reconciliation of insurance claims data is the controlled reconstruction of an insurer's economic and administrative claims history for a specified quantitative purpose. Its fundamental object is the relationship among events, notices, files, coverages, claimants, payments, reversals, recoveries, case reserves, legal status, valuation dates and insured exposure. Each relationship defines an analysis-specific observation and an associated error risk.
A valid methodological sequence begins by defining the population, amount basis, reporting horizon, target and allowed information set. It preserves immutable raw data, distinguishes source record identities from canonical claims, identifies real transactions separately from retries and reserve snapshots, reconstructs states through time, enforces coverage and exposure join constraints, reconciles signed financial movements with documented bridges, and reports both unresolved exceptions and their downstream influence. It retains economically credible extreme losses and distinguishes observation gaps from genuine zero outcomes. Reopening contributes claim development rather than mechanically creating a new insured event.
The worked examples establish why cleaning decisions are materially connected to statistical estimation. A duplicate payment changes gross and net paid measures; a duplicate claim can increase frequency and pure premium while reducing observed mean severity; and a single development-cell error can alter reserve estimates in other accident cohorts through shared development factors. Because these calculations are conditional on reconstructed data, data uncertainty is relevant alongside process, parameter and model uncertainty. A scientific conclusion about claims experience must state the data basis on which it rests.
International insurance practice contains jurisdiction-specific contractual and regulatory obligations. The U.S. actuarial professional standards, EU Solvency II data-quality provisions, IFRS accounting requirements, general statistical research and global data-quality management guidance examined here establish complementary, scope-dependent expectations. Their shared analytical relevance is the obligation to make the origins, transformations, intended uses and limits of claims data intelligible. The resulting reproducible record permits actuaries, statisticians, claims operations, financial reporting and researchers to evaluate the same numbers with the same definitions, while preserving the distinctions among their decisions.
References
Actuarial Community. (2022). Loss Data Analytics (open online textbook; September 1, 2022 preface). Open Actuarial Textbooks Project. https://openacttexts.github.io/Loss-Data-Analytics/index.html
Actuarial Standards Board. (2007; revised 2011). Actuarial Standard of Practice No. 43: Property/Casualty Unpaid Claim Estimates. Effective September 1, 2007; last revised May 2011. https://www.actuarialstandardsboard.org/asops/propertycasualty-unpaid-claim-estimates/
Actuarial Standards Board. (2010). Actuarial Standard of Practice No. 41: Actuarial Communications. Revised December 2010; effective May 1, 2011. https://www.actuarialstandardsboard.org/asops/actuarial-communications/
Actuarial Standards Board. (2016). Actuarial Standard of Practice No. 23: Data Quality. Revised December 2016; effective April 30, 2017. https://www.actuarialstandardsboard.org/asops/data-quality/
Actuarial Standards Board. (2017). Actuarial Standard of Practice No. 53: Estimating Future Costs for Prospective Property/Casualty Risk Transfer and Risk Retention. Adopted December 2017; effective August 1, 2018. https://www.actuarialstandardsboard.org/asops/estimating-future-costs-prospective-propertycasualty-risk-transfer-risk-retention/
England, P. D., & Verrall, R. J. (2002). Stochastic claims reserving in general insurance. British Actuarial Journal, 8(3), 443–518. https://doi.org/10.1017/S1357321700003809
European Commission. (2015). Commission Delegated Regulation (EU) 2015/35, Article 19: Data used in the calculation of technical provisions. EIOPA Solvency II Single Rulebook. https://www.eiopa.europa.eu/rulebook/solvency-ii-single-rulebook/article-2432_en
European Commission. (2015). Commission Delegated Regulation (EU) 2015/35, Article 265: Valuation of technical provisions—documentation. EIOPA Solvency II Single Rulebook. https://www.eiopa.europa.eu/rulebook/solvency-ii-single-rulebook/article-5928_en
European Insurance and Occupational Pensions Authority. (2022). Revised Guidelines on Valuation of Technical Provisions (published July 6, 2022; guidance applies from January 1, 2023). https://www.eiopa.europa.eu/publications/revised-guidelines-valuation-technical-provisions_en
European Parliament and Council. (2009). Directive 2009/138/EC (Solvency II), Article 121: Statistical quality standards. EIOPA Solvency II Single Rulebook. https://www.eiopa.europa.eu/rulebook/solvency-ii-single-rulebook/article-2208_en
European Parliament and Council. (2016). Regulation (EU) 2016/679 (General Data Protection Regulation), Article 5: Principles relating to processing of personal data. EUR-Lex. https://eur-lex.europa.eu/eli/reg/2016/679
Fellegi, I. P., & Sunter, A. B. (1969). A theory for record linkage. Journal of the American Statistical Association, 64(328), 1183–1210. https://doi.org/10.1080/01621459.1969.10501049
IFRS Foundation. (2017, as amended). IFRS 17 Insurance Contracts. IFRS Foundation, official issued standard information; effective for annual reporting periods beginning on or after January 1, 2023. https://www.ifrs.org/issued-standards/list-of-standards/ifrs-17-insurance-contracts/
IFRS Foundation. (n.d.). IFRS 17 Insurance Contracts—Key Terms. Official implementation resources, explanatory overview. https://www.ifrs.org/supporting-implementation/supporting-materials-by-ifrs-standards/ifrs-17/key-terms/
International Organization for Standardization. (2016). ISO 8000-61:2016: Data quality—Part 61: Data quality management: Process reference model. International Standard, November 2016. https://www.iso.org/standard/63086.html
Kuhn, M., & Johnson, K. (2013). Applied Predictive Modeling. Springer. https://link.springer.com/book/10.1007/978-1-4614-6849-3
Lindholm, M., & Zakrisson, H. (2021, online publication). A collective reserving model with claim openness. ASTIN Bulletin: The Journal of the IAA. https://doi.org/10.1017/asb.2021.33
Little, R. J. A., & Rubin, D. B. (2019). Statistical Analysis with Missing Data (3rd ed.). Wiley. https://doi.org/10.1002/9781119482260
Mack, T. (1993). Distribution-free calculation of the standard error of chain ladder reserve estimates. ASTIN Bulletin, 23(2), 213–225. https://www.casact.org/abstract/distribution-free-calculation-standard-error-chain-ladder-reserve-estimates
Rahm, E., & Do, H. H. (2000). Data cleaning: Problems and current approaches. IEEE Data Engineering Bulletin, December 2000. Leipzig University Database Group archival publication listing. https://dbs.uni-leipzig.de/research/publications/data-cleaning-problems-and-current-approaches
Rubin, D. B. (1976). Inference and missing data. Biometrika, 63(3), 581–592. https://doi.org/10.1093/biomet/63.3.581
Sambasivan, N., Kapania, S., Highfill, H., Akrong, D., Paritosh, P. K., & Aroyo, L. M. (2021). “Everyone wants to do the model work, not the data work”: Data cascades in high-stakes AI. Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. ACM; Google Research publication record. https://research.google/pubs/everyone-wants-to-do-the-model-work-not-the-data-work-data-cascades-in-high-stakes-ai/
Winkler, W. E. (2006). Overview of Record Linkage and Current Research Directions (Research Report Series, Statistics #2006-02). U.S. Census Bureau, February 8, 2006. https://www.census.gov/library/working-papers/2006/adrm/rrs2006-02.html