Insurance by BogdanovaThe Encyclopedia of Insurance and Risk Science

RESEARCH DOSSIER

Statistical Cleaning and Reconciliation of Insurance Claims Data

Abstract. Insurance claims data are generated by transactions, administrative decisions, contractual interpretations, payment systems, and reporting routines whose units and time conventions differ from those required by statistical inference. A claim record may represent a loss occurrence, a notice, a coverage component, a reserve estimate, a payment, a recovery, a litigation state, or an updated version of an earlier record. Statistical cleaning is consequently an exercise in reconstructing the insurance meaning of observations, rather than a cosmetic operation on rows. This monograph examines the foundations, formal identification problem, methods, controls, and inferential consequences of reconciling claim amounts, duplicate identifiers, reopened cases, accounting dates, payment streams, case reserves, recoveries, and exposure denominators. It develops a traceable distinction between physical source records, adjudicated economic claims, longitudinal claim states, and analysis-specific statistical units. The discussion integrates probabilistic record linkage, temporal joins, event-sourced reconstruction, accounting conservation identities, missing-data mechanisms, actuarial reserving, predictive evaluation, and regulatory data-quality requirements. Three transparent synthetic calculations demonstrate how a duplicated transaction changes gross and net paid loss, how administrative duplication distorts frequency and severity in opposite directions, and how a misclassified development payment propagates through a chain-ladder estimate. The original contribution is an integrated, decision-oriented interpretation of established methods: every proposed transformation is assessed against an explicit estimand, an immutable audit trail, and independently checkable monetary and population totals. No novel named theory, empirical industry prevalence estimate, or proprietary-system validation is claimed.

Keywords: Insurance claims data; statistical data cleaning; reconciliation; record linkage; duplicate detection; claim reopening; payments; recoveries; case reserves; claim development; bitemporal data; data provenance; exposure matching; missing data; actuarial reserving; data quality; sensitivity analysis; statistical estimands.

Introduction

A statistically useful insurance claim is the result of an identification and valuation process. It does not appear in the insurer's information systems as an invariant natural unit. A first notice of loss may lead to one file, several coverages, multiple injured parties, dozens of payment transactions, revised estimates of outstanding liability, rejected invoices, recovery actions, and later reopening. Insurance systems therefore accumulate facts about claims through time, while actuarial models typically require one observation per specified unit, valued at a defined information cutoff. The transformation between these representations is a scientific part of the analysis.

The distinction has practical consequences for every major insurance statistic. Duplicate notices can inflate claim frequency; duplicated payment transactions can inflate total paid losses; unlinked recoveries can distort gross-to-net comparisons; a reopened file can be counted as an additional new claim; a payment posted after the valuation cutoff can leak future information into a reserving triangle; and an incorrect policy join can assign a loss to exposure that never covered the event. Even when a data extract satisfies elementary checks for nulls and data types, its analytic content may be invalid.

The actuarial profession's ASOP No. 23 on Data Quality (Actuarial Standards Board, 2016) calls for data selection, review, evaluation of material limitations, and disclosures proportionate to the actuarial assignment. It explicitly distinguishes a professional data review from a formal audit. The European Solvency II data-quality requirements (European Commission, 2015, Article 19) specify completeness, accuracy, appropriateness, and documented processes for data used in technical provisions. Both sources support a central methodological proposition: the meaning and adequacy of data depend on their intended use.

The research question is therefore precise: under what documented transformations do heterogeneous operational records become a valid sample for a specified insurance estimand? The answer requires a traceable chain from raw event to analytic claim; an explicit choice among occurrence, claim, coverage, claimant, invoice, transaction, and policy-period units; temporal consistency; reconciliation with independent totals; and a sensitivity analysis of unresolved decisions.

The evidence base combines official professional standards, the publicly accessible regulatory rulebook, publisher-confirmed statistical and actuarial research, and documented data-management methods. The record-linkage framework of Fellegi and Sunter (1969) supplies a formal approach to uncertain entity resolution. The data-cleaning classification of Rahm and Do (2000) explains why integration and schema transformation must accompany cleaning. Research on stochastic claims reserving, including Mack's distribution-free derivation (1993) and the analysis of England and Verrall (2002), establishes the statistical relevance of the claims development matrix on which cleaned transactions eventually act.

The method throughout is documentary analysis and formal reasoning, supplemented by invented numbers whose arithmetic can be reproduced without access to confidential insurer records. The examples are demonstrations of mechanism, never estimates of actual market behavior. National legal definitions of an insurance claim, permitted benefit, recovery or insured event differ; their contractual and statutory particulars must be recorded rather than inferred from a universal data schema.

Raw Claims Tables and Transactions

The claim as an object with several representations

An insurance event is a factual occurrence or series of occurrences that may engage contractual coverage. A claim is an administrative and legal demand or file associated with a reported event. A coverage component is a portion evaluated under an insurance provision, deductible, limit, or responsible party. A payment transaction records an authorized disbursement, adjustment, or reversal. A case reserve is an estimated outstanding amount associated with known reported circumstances. The same information system may use the word claim for more than one of these objects. Reconciliation begins by writing their distinct definitions and keys into the data contract.

A usable conceptual decomposition is:

event→notification→claim file→coverage or claimant component→transactions and states.\text{event} \rightarrow \text{notification} \rightarrow \text{claim file} \rightarrow \text{coverage or claimant component} \rightarrow \text{transactions and states}.

The arrows describe possible administrative relations, not compulsory one-to-one mappings. A catastrophe occurrence generates many claims; one reported claim may cover several damaged property items; one motor accident may involve several claimants; one claimant may receive installments; one file may contain multiple coverage assessments. Deduplication at the event level therefore has a different purpose from deduplication at the payment-message level. The correct grain depends on the inferential target.

For an insurer's claims data environment, separate logical entities are useful even when a source vendor stores them in a single denormalized table:

Logical object Preferred key Required temporal context Typical measures Principal counting risk
Loss event event ID plus source provenance occurrence date/time event attributes, location multiple notices of one occurrence
Claim file persistent claim ID, entity namespace notice, opening and closure dates reported status, coverage decision duplicate IDs after migration
Coverage component claim ID plus coverage ID effective dates and decision version limit, deductible, share summing component and parent totals
Payment or reversal stable transaction ID plus version economic, posting, value dates signed amount and currency re-imports, retries and voids
Recovery recovery transaction ID receipt and recognition dates salvage, subrogation, ceded flows missing or sign-reversed receipts
Case-reserve movement claim/component ID plus sequence effective and recorded times outstanding level or change adding snapshots as transactions
Policy exposure policy/coverage-period key inception, termination and amendment insured units, earned exposure unjustified many-to-many joins
Analytical observation analysis-defined key valuation and information cutoff target and covariates changing denominators or leakage

A robust claim source retains both the operational claim ID and its source namespace. Identifier 12345 in two acquisitions need not denote the same claim. Claims transferred to a new administrator may acquire new identifiers; a crosswalk must distinguish migration aliases from genuinely independent files. Conversely, several payment references may legitimately belong to one claim. A unique constraint on claim_id in a transaction table would erase real payment history.

Three independent axes of identity, amount and time

Identity determines which economic entity a record describes. Amount determines what the record means financially. Time determines which state of knowledge the record expresses. The axes interact, but they cannot substitute for one another. Exact agreement of amount and date is weak evidence of identity when many benefit schedules pay standard amounts. Agreement of claim ID and transaction ID is stronger evidence when key governance is sound. A payment's bank value date, ledger posting date and accident date can each be correct while occupying different model periods.

A useful event record has the tuple

r=(s,k,v,e,te,tr,c,a,τ),r=(s,k,v,e,t_e,t_r,c,a,\tau),

where ss is source, kk is a source-defined key, vv is record version, ee is semantic event type, tet_e is its effective or economic time, trt_r is record/ingestion time, cc is currency, aa is signed amount when relevant, and τ\tau identifies the ingestion batch and transformation lineage. Not all operational sources possess every element. Missing elements are recorded as limitations rather than silently fabricated.

The event ledger is distinct from a current-state table. If a reserve snapshot is $9,000 at one date and $7,000 at a later date, those amounts represent successive valuations of one liability component; their sum is not a $16,000 expense. If a payment of $2,000 appears twice as an exact transport retry with the same authoritative transaction key, the second representation does not constitute a second cash outflow. If a transaction has an explicit posted reversal of −$2,000 and a corrected reissue of $1,800, all three events may be necessary to reconstruct the financial history, even though their net economic impact differs from a naive total of positive entries.

These distinctions reproduce the central integration problem described in Rahm and Do's data-cleaning survey (2000): single-source errors, multi-source conflicts, and schema transformations form one analytical problem. The ISO 8000-61 process reference model (International Organization for Standardization, 2016) is relevant as a general framework for organized data-quality management, although its public abstract does not prescribe an insurance claims data model or a universal reconciliation algorithm.

Source hierarchy and authoritative meaning

A production claims sample may be assembled from claims administration, payment disbursement, bank confirmation, general ledger, policy administration, reinsurance, third-party administrators, litigation management, and external loss-adjustment systems. They possess different authority for different questions. A bank settlement record may be strongest evidence that cash moved; a claim approval record establishes the purpose of the payment; a legal settlement document defines the obligation; the actuarial valuation extract supplies a particular time-stamped estimate; and the ledger determines how the insurer recorded the transaction in its accounting framework. No single source can authoritatively resolve every conflict.

The reconciliation design should specify, for each variable: operational meaning, source of authority, permissible aliases, precision and units, sign convention, valid ranges, effective-date logic, version precedence, correction process, aggregation rule, and documentary owner. This contract is particularly important when consolidating different insurers, lines, intermediaries, branches, currencies, and eras of administration software. A generic CSV column labelled paid cannot reveal whether the measure excludes allocated adjustment expenses, includes recoveries, uses gross or net reinsurance, or reports cumulative rather than incremental movement.

Conservation identities and the boundaries of equality

For a defined set of claim transactions T(c,t)\mathcal T(c,t) through a stated valuation cutoff, gross paid loss may be reconstructed as

Pcgross(t)=∑j∈T(c,t)pj,P_c^{\mathrm{gross}}(t)=\sum_{j\in\mathcal T(c,t)}p_j,

where disbursements are positive and valid financial reversals are negative. If actual recoveries are expressed as positive receipts in a separate table, then an explicitly defined net-of-recoveries paid measure is

Pcnet recovery(t)=Pcgross(t)−Rccollected(t).P_c^{\mathrm{net\ recovery}}(t)=P_c^{\mathrm{gross}}(t)-R_c^{\mathrm{collected}}(t).

This definition is not synonymous with net of ceded reinsurance: reinsurance recoverables and collections require separate legal and accounting classification. Deductibles, coinsurance, third-party recoveries and subrogation should not be collapsed into one unlabeled adjustment. Likewise, case incurred for an ordinary undiscounted gross loss basis may be represented as

Iccase(t)=Pcgross(t)+Ocgross(t),I_c^{\mathrm{case}}(t)=P_c^{\mathrm{gross}}(t)+O_c^{\mathrm{gross}}(t),

where OcO_c is the latest valid outstanding case-reserve snapshot at tt. It is neither the ultimate economic loss nor automatically the IFRS 17 liability for incurred claims. The latter operates under a wider accounting measurement framework, as the IFRS Foundation's IFRS 17 key-term explanations make clear.

For each equivalence claim there must be a basis: same portfolio, legal entity, time cutoff, claims type, currency, gross/net convention, loss adjustment expense treatment, recoveries, and recognition rules. A claim subledger equal to a bank statement in cash terms may not equal claims expense for that month because accrual, timing, and categorization differ. A failed equality is therefore a hypothesis requiring diagnosis, not immediate evidence that one source is erroneous.

Why descriptive completeness is insufficient

Data completeness must be evaluated against a relevant population. A dataset containing every row in one administration system can omit all business serviced externally or claims awaiting first notice. A transaction sample can be perfectly reconciled to a payment subledger yet miss case reserves needed for incurred loss analysis. A complete reported-claim file can omit incurred-but-not-reported obligations by construction. Under Solvency II Article 19, data used for technical provisions are assessed for the sufficiency of historical information and relevant homogeneous risk groups, as well as consistency and intended-use appropriateness. This implies that entity coverage, not merely null-field percentage, belongs in the quality assessment.

The first result of a defensible cleaning process is a documented population definition: covered contracts, insurers, geographical boundaries, lines, accidents, reporting periods, claim types, and monetary basis. The second is an immutable raw snapshot from which the current analytic representation can always be regenerated. Only after these are established does it make sense to optimize row-level cleaning rules.

Duplicated and Reopened Claims

Three different forms of duplication

Duplicate claim data arise at least in three ways. A duplicate transport record is a second delivery of the same underlying message, often after a retry or batch reprocessing. A duplicate administrative file represents the same economic claim under a new or parallel identifier, for example after migration or a repeated notification. A duplicate analytic observation occurs when a correct many-to-one source relationship is inadvertently expanded into multiple sample rows by a join or aggregation. Each case needs a different remedy.

Transport deduplication should use an authoritative message or transaction key plus source namespace and event version. Administrative-file reconciliation requires evidence about claimant, event, coverage, loss location, occurrence date, and migration history. Analytic duplication requires auditing the cardinality of joins, especially policy-to-claim, claim-to-payment, and claim-to-cover relationships. Applying a generic drop_duplicates() call to a fully denormalized table confuses these levels.

The classical Fellegi–Sunter linkage theory (Fellegi and Sunter, 1969) frames uncertain matching through agreement patterns between candidate record pairs. In simplified form, the weight for pair (i,j)(i,j) with comparison vector γ\gamma is

Wij=log⁡Pr⁡(γ∣M)Pr⁡(γ∣U),W_{ij}=\log\frac{\Pr(\gamma\mid M)}{\Pr(\gamma\mid U)},

where MM denotes a true match and UU a nonmatch. Positive and negative evidence accumulate across attributes when the relevant probabilistic model is justified. The original method permits match, nonmatch, and review decisions. Modern implementations frequently adjust comparison distributions for dependencies, missingness, blocking, and class imbalance; a numerical threshold should be validated on representative adjudicated pairs rather than transferred unchanged from another portfolio.

The U.S. Census Bureau's Winkler record-linkage overview (2006) surveys developments and applications of statistical matching. For claims data, plausible blocking dimensions include insurer/legal entity, line, year and approximate loss date, with explicit exceptions for cross-border events or delayed reports. Blocking can dramatically reduce candidate comparisons, but false exclusion of a true pair from every block is an irrecoverable false nonmatch unless additional search stages are used.

Decisions must be attached to objects

Suppose notices A and B share an accident date and vehicle registration but concern different injured parties. At event level they may be associated with one occurrence; at claim-component level they are separate obligations; at transaction level their payments are distinct. Merging them as one claim because they share an accident would reduce claimant frequency and understate per-claim case complexity. Conversely, when a duplicate notice is adjudicated to be the same demand, counting it twice raises observed reported-claim frequency. Accordingly, linkage output should include match_level (event, claim, claimant, coverage, payment), decision (confirmed, probable, rejected, review), effective_basis, evidence, reviewer, and the surviving canonical identifier.

Matching is also a graph problem. If A matches B and B matches C, automatic transitive clustering can be dangerous: A and C may differ materially when pairwise rules allow weak links. Clusters should obey domain constraints such as incompatible contract periods or distinct injured persons. A stable canonical claim ID may be chosen without deleting source IDs. The crosswalk remains the record of how the administrative population was interpreted.

False matches and false nonmatches carry asymmetric costs. A false match can erase legitimate claim counts and combine unrelated severities; a false nonmatch can increase counts, duplicate monetary measures, or incorrectly split temporal histories. A risk-sensitive matching policy uses separate review thresholds for high-severity, litigation, cross-border, long-tail and multi-coverage cases. Sampling the manually adjudicated disagreements supplies estimates of precision and recall, but these are population- and sampling-frame-dependent.

Reopening is a state transition

A reopened claim is an earlier claim file that returns from a closed to an active state. Whether reopening creates a new reportable claim number depends on the source and the required reporting definition; analytically it ordinarily contributes a status transition and renewed payment/reserve exposure to the persistent claim's development history. A closure followed by reopening must not automatically be counted as two independent loss occurrences. The importance of this distinction is underscored by Lindholm and Zakrisson's claim-openness model (published online 2021), which explicitly models claim closing and reopening within reserving dynamics.

Let Sc(t)∈{unreported,open,closed,reopened}S_c(t)\in\{\mathrm{unreported},\mathrm{open},\mathrm{closed},\mathrm{reopened}\} be a descriptive administrative state. A sequence might be reported → open → closed → reopened → closed; the model observation at cutoff tt is the last valid state after applying ordered events, but the event sequence itself is preserved. The count of distinct reports during a calendar year may differ from the count of open claims at year end, claims ever reopened, and claims receiving payment during the year. Each numerator answers a different question.

Reopening is often entangled with reserve and payment conventions. A claim closed with a zero reserve can subsequently reopen and incur an additional expense. If the cleaning rule filters any row with closed = true, the historical loss may disappear or later transactions become orphans. A state reconstruction instead maintains the claim in the cohort and attributes each new cash flow to its correct original occurrence year and payment calendar year, except where explicit contractual or statistical definitions direct otherwise.

Duplicate candidates are not invalid observations

Equal payments frequently arise from standard medical tariffs, recurring disability benefits, fixed indemnities, annuity installments, and split settlements. Therefore, amount/date equality is a useful anomaly signal but a weak deletion criterion. Conversely, the same bank transaction may appear with slightly different labels or timestamps in successive extracts. A reliable deletion or consolidation rule identifies the semantic invariant that the source system intends to preserve and checks whether canceling and corrected entries are linked by explicit reversal identifiers.

The decision record should be append-only. Each merge or suppression references the source records affected, pre- and post-cleaning keys, evidence and reason code. The raw payments remain in the archive; the analytic view suppresses repeated transport representations or expresses valid reversals in a conserved signed ledger. A reconciliation report can then separately count source rows, excluded duplicates, canonical claims, accepted payment events and exceptional unmatched records, making the interpretation accessible to auditors and analysts.

Estimating identification uncertainty

If matching is uncertain, a definitive single dataset can create false statistical precision. One practical sensitivity analysis constructs two internally coherent datasets from ambiguous cases: one conservatively retains possible independent claims; another merges only candidates that pass a lower threshold under documented review rules. Differences in frequency, total paid and target-model outputs are then reported as identification sensitivity. For a deeper probabilistic analysis, linkage uncertainty may be propagated across multiple plausible partitions, but model assumptions and computational expense must be stated.

The important conclusion is epistemic. Duplicate detection is a statistical inference about the identity of underlying obligations. The data engineer's chosen canonical key shapes the analyst's observed population. Confidence in downstream models cannot exceed confidence in the entity resolution on which those models depend.

Date, Payment, and Status Reconciliation

Accident, report, booking, payment, and knowledge time

At least five clocks can accompany a claim. Occurrence time assigns the risk to an accident or event cohort. Report time locates first notice and informs reporting delay. Booking or posting time records when an amount enters an accounting system. Effective or value time describes when a financial transaction is recognized according to a specified convention. Ingestion or knowledge time marks when a data processor first possessed a record. Additional dates include coverage start and end, reserve evaluation, closure, reopening, claim adjudication and settlement approval. A model that treats these dates as interchangeable changes the statistical object under study.

For reporting triangles, define accident period a(c)a(c), valuation period vv, and development age d=v−a(c)d=v-a(c) under an explicit period calendar. Claim payments enter a cell based on their assigned accident cohort and eligible payment date. A reporting triangle instead records when claims become known, and an incurred triangle combines paid development and outstanding estimates. A dataset that uses last_modified_at to determine accident year will systematically reassign late-edited long-tail claims to recent cohorts.

A sound period calendar records calendar system, timezone, effective midnight rule, daylight-saving treatment, source precision, fiscal-period mapping, and whether a transaction is booked or economic-dated. For example, a payment instructed at 23:50 local time and posted at 00:10 the next day may fall in different months. Neither choice is inherently an error; the analyst must state the convention and reproduce the resulting balance bridge. Currency conversion similarly requires a conversion basis and date; gross amounts in mixed currencies cannot be compared or summed before unit normalization.

Bitemporal reconstruction

A later correction may say that the earlier economic fact was different from what the analyst knew at the time. A bitemporal perspective distinguishes the time when a fact applied (valid_time) from the time when the system recorded or learned it (transaction_time). This distinction is useful even where the source database does not implement a formal bitemporal schema. A retrospective estimate may use corrected knowledge, whereas a genuinely prospective backtest must use the information available at the historical decision date.

Represent the dataset as D(tv,tk)D(t_v,t_k), where tvt_v limits economic or valuation time and tkt_k limits record availability. A historical prediction at decision date tt should use a measurable feature set Ft\mathcal F_t built from information genuinely accessible by tt. If the feature table is refreshed today with a claim's final settlement code and joined to a prediction made two years ago, the resulting model is contaminated even if the correct accident date is retained. The predictive-modeling treatment by Kuhn and Johnson (2013) emphasizes disciplined preprocessing and data splitting; the insurance-specific extension is to enforce temporal availability within those splits.

The reconstruction rule should resolve superseded records by source sequence and knowledge cutoff, never by taking the lexicographically latest text field or the largest monetary amount. A corrected reserve snapshot may have an earlier valid date but a later entry date. An analyst selecting the latest recorded version for a past backtest would be using unavailable future information. The same record may appropriately produce two views: an as-reported historical view and a restated best-current view. These are separate datasets with separate purposes and labels.

Incremental financial movements and stock snapshots

Payments and recoveries are flows; case reserves are stocks at a valuation point. Combining them requires preserving this dimensional distinction. For a known claim component, let PtP_t denote cumulative valid gross payments through tt and OtO_t the selected valid outstanding case balance. Then It=Pt+OtI_t=P_t+O_t is an elementary case-incurred measure. If the reserve source stores incremental adjustments ΔOj\Delta O_j, the final reserve can be reconstructed from an initial balance and its authorized deltas. If it stores absolute snapshots, summing snapshots is a category error.

An amount labelled incurred_loss may mean gross case incurred, net case incurred, loss plus expense, or a financial-reporting estimate. Cross-system reconciliation needs explicit, named transformations, such as:

gross paid loss  →  gross paid plus allocated expense  →  less actual recoveries  →  net ceded basis.\text{gross paid loss} \;\to\; \text{gross paid plus allocated expense} \;\to\; \text{less actual recoveries} \;\to\; \text{net ceded basis}.

The ordering is descriptive, and each adjustment must be separately defined. A deductible paid by the insured may reduce the insurer's cash payment without reducing the underlying event's physical loss. Similarly, a reinsurance recovery from a reinsurer is not the same as salvage recovered from damaged property. ASOP No. 43 on unpaid claim estimates (Actuarial Standards Board, 2007, revised 2011) explicitly asks the actuary to identify gross or net treatment, covered claims, types of adjustment expense, and the specific recoverables considered.

A fully worked transaction reconciliation

Consider the following wholly synthetic history for one reported property claim. All amounts are in USD; the original occurrence, coverage and claim identifiers are stable. The reserve on the cutoff date is assumed to be the latest valid gross outstanding case balance.

Transaction date Authoritative event Signed gross loss cash flow Collected recovery Interpretation
February 3 initial claim payment +4,000 0 valid disbursement
March 10 erroneous supplemental payment +1,500 0 initially posted cash movement
March 11 explicit reversal of March 10 −1,500 0 linked financial reversal
March 12 corrected supplemental payment +1,200 0 valid reissue
March 12 retry of same payment-message ID 0 after deduplication 0 second representation only
April 2 subrogation receipt 0 +300 recognized recovery, separate flow
April 30 latest case reserve snapshot 0 0 gross outstanding 2,500

The accepted gross cumulative paid amount is 4,000+1,500−1,500+1,200=5,2004{,}000+1{,}500-1{,}500+1{,}200=5{,}200. Collected recovery is 300300, giving a defined net-of-collected-recoveries paid amount of 4,9004{,}900. Gross case incurred is 5,200+2,500=7,7005{,}200+2{,}500=7{,}700; net of only the already collected recovery, with the gross case reserve otherwise unchanged, the illustrative figure is 7,4007{,}400. These amounts use distinct bases and should not be relabeled interchangeably.

If the ingestion retry is incorrectly treated as another valid +1,200 payment, gross paid rises to $6,400, net-of-recovery paid rises to $6,100, and gross case incurred rises to $8,900 with the same reserve. If the correction process instead drops the +1,500 payment but retains its −1,500 reversal, gross paid becomes $3,700. Both mistakes arise from treating related financial events as independent rows and demonstrate why a simple positive-payment filter is not a valid accounting transform.

A formal invariance test for a transport retry is

∑j∈accepted eventsaj=∑j∈economic eventsaj,\sum_{j\in \text{accepted events}} a_j =\sum_{j\in \text{economic events}} a_j,

provided the adjudicated event mapping is correct and all source signs are transformed consistently. The equality is an audit target rather than a guarantee supplied by software. Bank-cash, general-ledger and claims-subledger totals may require explicit outstanding-in-transit, period-boundary or reclassification bridge items before such equality is expected.

Reopened status, claim reserves, and the valuation boundary

A reopened claim can receive a new reserve snapshot after an earlier closing reserve of zero. A state machine should preserve both the closure event and later reopening, reconstruct the reserve relevant at each cutoff, and retain the original claim cohort. A late reserve estimate must not be substituted into an earlier financial close if the purpose is to replay what was known then. Conversely, a retrospective restatement may deliberately incorporate corrections that were unavailable at the original cutoff; the distinction belongs in the dataset name and metadata.

Negative amounts require examination, not blanket removal. Valid negatives can arise from reversals, refunds, corrections, salvage netting, or reclassification. Invalid negatives can arise from reversed sign conventions, OCR or migration errors, or mixed expense and recovery fields. The analyst checks whether the source expects signed net flows, nonnegative gross flows plus separately represented recoveries, or absolute balances. A claim's ultimate amount may also be negative in a narrowly defined net settlement dataset; an automated clipping rule would suppress its economic meaning.

Currency and accounting reconciliation

For multinational portfolios, summing currency-denominated claims without conversion creates a dimensionally invalid result. A reproducible reporting currency transform needs the original currency code, numeric precision, transaction and/or valuation date, approved exchange-rate source, rounding convention, and whether the line is monetary, inflation-adjusted, or discounted. A currency correction changes both amount and its audit explanation. It cannot be quietly represented as a duplicate adjustment.

The IFRS Foundation's IFRS 17 accounting framework (2017, effective from 2023) measures groups of insurance contracts using fulfillment cash flows and other components as applicable. Therefore, a cleaned claim-level paid-plus-case field is an actuarial operational measure, while reported insurance liabilities can require discounting, risk adjustment, grouping, and additional cash-flow projections. The obligation to reconcile operational data with financial accounts does not imply numerical identity between these conceptually different measurements.

Data without precise dates

When date precision is limited, missing dates should be represented by ranges or an explicit imputation field with uncertainty, not fabricated noon timestamps. If only a month is known, the true event time falls within that month; assignment to accident-year totals may be stable while assignment to daily prediction features is uncertain. A conservative sensitivity test can place the ambiguous event at both extremes of its admissible range and measure changes in the relevant triangle or time-to-report distribution. Source-provenance notes must distinguish a missing source date, an invalid parser, an uncertain business rule, and a date intentionally withheld for privacy.

Claims and Exposure Joining

The population denominator is a scientific decision

Observed claim frequency is a ratio between a defined claim count and a relevant exposure measure. Its denominator may be insured vehicles, earned vehicle-years, policy-years, insured payroll, property values, benefit member-months or another line-specific quantity. A payment transaction is not an exposure unit, and a claim file is not a coverage period. Before joining, the analyst defines the covered population and target interval as carefully as the claimant identity.

For claim count NN, earned exposure EE, and gross aggregate loss LL on compatible bases:

f^=N/E,s^=L/N,π^=L/E=f^s^,\widehat{f}=N/E,\qquad \widehat{s}=L/N,\qquad \widehat{\pi}=L/E=\widehat{f}\widehat{s},

where f^\widehat f is a descriptive frequency rate, s^\widehat s is mean loss per included claim, and π^\widehat\pi is historical pure premium per exposure unit. These identities hold algebraically for the stated population when N>0N>0, but their statistical interpretation depends on claims maturity, censoring, coverage, truncation, inflation, and inclusion rules. The open actuarial Loss Data Analytics text (Actuarial Community, 2022) provides a broader educational treatment of loss modeling and applied statistical analysis.

Coverage joins and their failure modes

A claim must be linked to the appropriate contract, risk item, coverage and effective policy version. The matching rule requires both identifiers and time validity. For a loss occurrence at tct_c, the relevant coverage version may be the contract in effect at tct_c, while the premium exposure window uses earned coverage throughout a different interval. Endorsements retroactively changing insured interests, open-market policies, claims-made triggers, extended reporting periods, and delayed notifications make a single claim_date BETWEEN inception AND expiry test insufficient across all products.

A many-to-many relationship can be legitimate: one claim involves two coverages; one occurrence affects several insured locations; a liability claimant is associated with several defendants. However, naive joins multiply payment rows. If one claim payment of $5,000 is joined to three matching coverage records and then summed as $15,000, the economic total has been changed by relational duplication, not by an actual new transaction. Before monetary aggregation, a claim-to-coverage allocation rule must establish the appropriate split or select a canonical coverage. When allocation weights wckw_{ck} represent mutually exclusive shares of the same claim amount, require ∑kwck=1\sum_k w_{ck}=1 and document exceptions; when coverage records represent distinct obligations, the data model must not impose that identity artificially.

Every merge should report its cardinality: left-unmatched, right-unmatched, one-to-one, one-to-many, many-to-one, many-to-many, and duplicated keys after joining. Claim records without matched exposure are not automatically invalid; there may be canceled-policy tails, extended reporting periods, missing migration keys, or recoveries from historical cohorts. An omission decision must distinguish an orphan record, an uncovered loss, and a legitimate claim lacking a contemporary policy key.

A transparent frequency–severity demonstration

Consider a synthetic portfolio with 40 earned policy-years, five distinct valid claims, and $27,000 in total gross ultimate losses. The accepted descriptive metrics are five divided by 40 = 0.125 claims per policy-year, $27,000 divided by five = $5,400 per claim, and $27,000 divided by 40 = $675 per policy-year.

Now suppose an administrative duplicate of one already included $4,000 claim is erroneously treated as a sixth independent claim and its amount included again. The raw extract reports six claims and $31,000. Its apparent frequency is 6/40=0.156/40=0.15, an excess of 0.025, or 20% relative to the correct frequency. Its apparent severity is 31,000/6≈5,166.6731{,}000/6\approx5{,}166.67, approximately 4.32% below the correct $5,400; nevertheless its pure premium becomes 31,000/40=77531{,}000/40=775, approximately 14.81% above the correct $675. An administrative error simultaneously increases observed frequency and total loss while decreasing average severity. The direction of a model's bias therefore depends on its specific outcome.

Measure Canonical five claims Duplicate-counted six Difference
Earned policy-years 40 40 0
Distinct claims 5 6 +1
Gross ultimate losses $27,000 $31,000 +$4,000
Claims per policy-year 0.125 0.150 +20.00% relative
Average loss per claim $5,400.00 $5,166.67 −4.32% relative
Loss per policy-year $675.00 $775.00 +14.81% relative

These figures are invented and independently reproducible. They illustrate propagation of a deduplication error through identities rather than a claim that any real insurer exhibits the same magnitudes.

Reporting delay, closed-claim sampling, and selective observation

At a valuation date, claims not yet reported cannot be directly observed in a reported-claim table. Claims reported but not settled may have censored severities. Restricting severity estimation to closed files selects claims on a post-occurrence process whose duration can depend on loss size, litigation, product and severity. The resulting sample can be biased toward faster-settling claims. Statistical cleaning cannot make such a selection representative merely by correcting syntax or merging identifiers.

Let Rc(t)R_c(t) indicate report by tt, Cc(t)C_c(t) indicate closure by tt, and YcY_c represent ultimate loss. The average E[Y∣R(t)=1,C(t)=1]E[Y\mid R(t)=1,C(t)=1] need not equal E[Y]E[Y] over all insured loss events. Their difference depends on reporting and closure mechanisms and on the chosen risk population. Rubin's missing-data analysis (1976) distinguishes conditions under which ignoring a missingness mechanism is valid. The Little–Rubin treatment (2019) develops likelihood-based and other approaches for missing data. These statistical frameworks illuminate why imputing a mean severity for every open claim can misrepresent outcome dependence and uncertainty.

Particular care is required when a recorded zero has several meanings: genuine zero payment, claim closed without payment, unpaid open claim, not-yet-populated field, coverage denial, offset by recovery, or a data export default. A zero is not automatically a missing value, and a null is not automatically zero. A principled cleaning process preserves the source's missingness and legal-status semantics before deciding how to model them.

Insurance-product distinctions

In property insurance, a single catastrophe may generate multiple claims while one insured building can have several coverage components. In automobile insurance, an accident can create bodily injury and property damage claims with different reporting and litigation pathways. In health insurance, one episode of care can yield multiple provider claims, reversals, adjustments and coordination-of-benefits transactions; claim-line versus episode-level units must be distinguished. In disability or long-term care insurance, periodic benefit payments create legitimate repeated amounts and claims often remain open over long durations. Agricultural insurance can involve a season, farm, crop and peril index whose exposure unit differs from the policyholder. These distinctions are governed by the product and its administrative data generation, rather than a single global duplicate threshold.

Consequently, claim frequency across lines cannot be pooled without standardizing the event definition, covered population, exposure, and reporting horizon. Pure premium should not be compared directly across currencies, changing coverage limits, inflation regimes or uncensored development without appropriate transformations and assumptions. The U.S. ASOP No. 53 guidance on future costs (Actuarial Standards Board, 2017) identifies historical claim adjustments, business mix, coverage, operational practice, accounting and reinsurance changes as relevant to prospective property/casualty estimation. These are precisely the changes that an apparently uniform historical claims dataset may conceal.

Privacy and permitted statistical joins

Claims data can contain identifiers, diagnoses, financial details, geolocation, disputes and sensitive household information. Reconciliation should use the minimum identifiers necessary for the defined legal and analytical purpose and protect crosswalks through access controls, retention rules and review procedures. Under the EU GDPR's data-processing principles (European Union, 2016, Article 5), purpose limitation, data minimization and accuracy constrain processing of personal data when the Regulation applies. Its legal scope and permitted bases must be assessed for the actual controller and jurisdiction; a global claims warehouse does not acquire an unrestricted legal basis merely by describing its use as statistical.

Privacy measures can alter linkage error. Tokenization that changes across source systems may block legitimate matches; excessive disclosure of direct identifiers raises unnecessary risks. A privacy-aware linkage design documents the permitted comparison fields, salted or privacy-preserving keys where lawful and technically justified, access controls for adjudication, and performance checks against appropriately protected validated match cases. Statistical validity and privacy compliance are related design requirements that need joint verification.

Audit of Cleaning Decisions

Reconciliation as a reproducible scientific argument

A cleaned dataset is a claim about how source records correspond to the target population. Its construction should therefore be reproducible and open to challenge. Every substantive transformation needs the raw input, business justification, rule version, affected records, exception path, count and amount before and after, approver where required, and the date on which the rule was applied. A sound audit trail allows the same historical snapshot to be reproduced, even when new corrections are received later.

The U.S. ASOP No. 23 data-review standard directs the practitioner to consider the intended purpose, internal consistency, data sufficiency and significant limitations; it does not substitute a full formal audit of source data. A scientific claims cleaning study should likewise distinguish tests performed by the author from validations performed by claims operations, accounting, internal audit and independent assurance functions. The existence of a data dictionary alone is not proof that its assertions are true in the underlying systems.

A useful auditable pipeline has the following sequence: register raw immutable extracts; profile source keys and monetary conventions; apply deterministic ingestion deduplication; resolve administrative identity with uncertainty labels; normalize event types and signed flows; reconstruct temporal state for each valuation; join to coverage and exposure with cardinality tests; reconcile balances and population counts; build target-specific sample; evaluate errors and sensitivities; then freeze the release candidate with checksums and data lineage. The steps can be repeated whenever additional data arrive, and the process must be stable under the same input and parameter versions.

Audit measures and exception rates

Quality metrics should describe failure categories and their denominators. Let nn denote all source rows in a defined intake batch, uu the number failing uniqueness constraints, oo the number with unresolved economic ownership, and mm the number with missing required fields. Then u/nu/n, o/no/n, and m/nm/n are descriptive diagnostics, but the percentages are not interchangeable and overlapping categories need not sum to 100%. An insurer may have a very small exception rate by row count and a large exception rate by financial exposure if the unresolved records contain high-value liability claims.

For monetary reconciliation, define source total SS, transformed total TT, and documented bridge items B1,…,BKB_1,\ldots,B_K on a matched basis. The residual is

ε=S−T−∑k=1KBk.\varepsilon = S - T -\sum_{k=1}^{K}B_k.

For exact integer-cent, same-currency signed cash records under the same cutoff, ε=0\varepsilon=0 is an appropriate acceptance criterion when all required transactions are included. For converted currencies, rounded published reports or actuarial estimates, an explicit tolerance may be reasonable. A tolerance must be justified by rounding and business significance, rather than chosen to force acceptance. The reconciliation report should show both raw residual and residual relative to a meaningful magnitude; a zero relative residual can be undefined when the base is zero, so the raw measure remains essential.

Separate at least four types of diagnostics: structural (schema, keys, relationships), temporal (ordering, cutoff and effective dates), financial (conservation and ledger bridges), and statistical (distributional shifts, population selection and model influence). A table may pass structural and financial checks while failing statistical relevance because the risk mix changed. Conversely, a legitimate catastrophe may produce an extreme loss distribution that is statistically unusual but financially supported and therefore must be retained.

Outliers, heavy tails, and erroneous amounts

Insurance severity distributions are often skewed, and extreme observations can be economically central. A maximum-value check that truncates every payment above a fixed percentile transforms the tail of the insured loss distribution. A high loss can be a genuine catastrophe, a currency conversion error, a limit error, a duplicate or a book entry. The appropriate response is to investigate source provenance, coverage and payment authorization; label verified extremes; correct demonstrated errors; and conduct analyses under stated winsorization, censoring or robust-estimation assumptions only when the estimand warrants it.

A useful diagnostic vector for a questionable monetary observation may include amount-to-limit ratio, currency and exchange-rate factor, payment versus reserve movement, duplicated payment identifiers, paid-to-date compared with case outstanding, reporting lag, loss type and known catastrophe identifier. Thresholds should be line-specific and time-aware. A high-value disability lifetime benefit and a high-value automobile glass claim have different baseline plausibility. Financial anomalies and statistical outliers are hypotheses for investigation, not automatic deletion orders.

Failure taxonomy and decision log

A transparent exception register can use compact, semantically defined classes:

Exception Detection evidence Scientific risk Permitted response
Replayed source message identical authoritative message ID and version duplicate transaction count and paid amount suppress transport copy in analytic view; preserve raw
Migration alias verified identifier crosswalk duplicate claim frequency merge canonical claim identity with lineage
Ambiguous matched pair moderate probabilistic linkage score false match or missed match adjudicate or retain sensitivity alternatives
Contradictory reserve versions overlapping effective intervals or sequence conflicts invalid case-incurred snapshot resolve authoritative version; document rule
Unlinked payment no valid claim or component association omitted or misallocated paid loss quarantine for review; bridge financial totals
Out-of-period posting event, booking or knowledge date mismatch false cohort or data leakage dual-date attribution under named convention
Ambiguous recovery sign unclear source direction gross/net reversal obtain ledger meaning before transformation
Exposure join fan-out multiple matched coverage rows multiplied loss or biased denominator establish allocation/ownership; rerun amount checks
Source date uncertainty precision or timezone missing invalid reporting lag retain interval and measure sensitivity
Extreme credible claim valid documentation, unusual severity tail estimation distortion retain, tag, model explicitly

The list is an analytic classification for this monograph, not a newly named external standard. Materiality depends on use: one unmatched claim of $30 million can matter more for reserves than thousands of small stale notifications, while a cluster of duplicate low-cost claims can matter more for frequency models than for total losses. Portfolio-level error rates should be accompanied by weighted loss metrics and a breakdown by claims maturity, product, geography and source system.

Independent reconciliation layers

Reconciliation should be checked against more than one source where possible. A claims-system transaction extract can be compared to the claims subledger; the subledger to the general ledger; payment authorization to bank settlement; reserve movement to case-reserve snapshots; claim counts to reported-claim registers; and policy-linked claims to the exposure table. Agreement between two downstream systems that share the same upstream mistake does not provide independent confirmation. Effective control requires identifying which comparison sources have independent origin or processing.

Under the Solvency II documentation rule (European Commission, 2015, Article 265), the technical-provisions process documents data collection and analysis of quality, source and usage directories, assumptions, methods and validation. EIOPA's guidelines on valuation of technical provisions (2022 revision) address, among other topics, data deficiencies and adjustments. Their application depends on the EU supervisory context and is not an automatic legal mandate for every jurisdiction in this international encyclopedia.

Governance of transformations and version control

A cleaning rule should have an owner, effective date, tested examples, permitted input schema, assumptions, explicit output contract, failure mode and version. The output should record the exact hash of the transformation code and input snapshot where tooling supports it. Analysts should be able to regenerate prior valuation views rather than overwrite them. Human adjudication of material identity conflicts should be recorded as a typed decision applied by the pipeline on subsequent runs; this is preferable to an untraceable manual spreadsheet deletion.

The process should also distinguish error correction from assumption-driven statistical adjustment. Replacing USD mistakenly recorded as JPY following bank evidence is a correction of source meaning. Inflating historical claim amounts to a future cost level is a model adjustment. Imputing an outstanding amount from similar claims is an inferential procedure. Applying a cession percentage to gross claims is a contractual/accounting transform. All may be legitimate within a project, yet each changes what the resulting number represents. One generic cleaned_amount field conceals these epistemic differences.

Regulated and professional communication should report material changes and limitations in language appropriate to its audience. The ASOP No. 41 communication requirements (Actuarial Standards Board, 2010) provide a relevant U.S. professional framework for documenting actuarial results and disclosures. The author of a scientific dataset must likewise state what was checked, what could not be independently resolved, and which decisions may affect readers' interpretation of estimates.

A reproducibility checklist expressed as data assertions

The following assertions provide a compact computational audit without purporting to establish universal thresholds. A pipeline may require: raw file hashes unchanged; all source systems enumerated; stable unique event IDs under a defined namespace; no unintended many-to-many financial fan-out; each valid reversal linked to its underlying movement or documented as unresolved; currency and sign known for every amount entering a sum; valuation state selected by effective and knowledge cutoff; monetary bridge residual within its justified tolerance; analysis-specific sample count reproduced; externally reviewed exceptions retained; and outcome sensitivity assessed when uncertain records are material.

Such controls distinguish scientific reproducibility from an attractive summary dashboard. A final approved dataset is one whose records, transformations and limits can be traced and whose statistical implications have been examined. It remains a versioned representation of underlying events; subsequent claims development may legitimately change its future values.

Downstream Modeling Sensitivity

Cleaning changes the estimand's empirical support

The importance of cleaning becomes measurable when models are fitted to claims data. A frequency model observes a count process; a severity model observes amounts conditional on claim inclusion; a pure premium model observes total loss per exposure; a reserving model observes development patterns by cohort and time; and an operational model may predict settlement duration or reopening. Each target uses a different projection of the underlying claims history. A cleaning rule can improve one projection while leaving another unchanged or worsening it.

Suppose a duplicate payment is detected and removed from a claim's transaction ledger. The number of canonical claims may be unchanged, while the paid severity and incurred triangle decrease. Suppose instead that one duplicate claim file is merged with another and its legitimate separate payment events are kept. The claim count decreases while total paid loss remains unchanged; hence severity per claim increases and pure premium remains unchanged. These two corrections must not be conflated. A third case is a duplicated file with copied transaction records: merging both claim identity and redundant payments changes counts and totals. This is why a general phrase such as deduplicate claims is insufficient to describe a statistical pipeline.

For any statistic θ(D)\theta(D) computed on dataset DD, a useful documented influence measure is

Δθ=θ(Dclean)−θ(Draw).\Delta_\theta=\theta(D_{\mathrm{clean}})-\theta(D_{\mathrm{raw}}).

The difference is descriptive and conditional on the chosen cleaning decisions. It is not proof that either dataset equals an unobservable ground truth. Analysts should calculate the impact by error class, cohort and line, and report alternative plausible corrections for ambiguous cases. When a transformation modifies the observation mechanism or conditional sample, sensitivity must be assessed at both statistic and fitted-model levels.

Reserve triangle construction

For accident year aa and development age dd, let Ca,dC_{a,d} denote cumulative paid loss as of that development evaluation, formed from accepted transactions assigned to the corresponding accident cohort. The cumulative chain-ladder development factor can be estimated for mature cohorts by

f^d=∑a∈AdCa,d+1∑a∈AdCa,d,\widehat f_d = \frac{\sum_{a\in A_d}C_{a,d+1}}{\sum_{a\in A_d}C_{a,d}},

where AdA_d comprises accident years observed at both ages, and the development ages are consecutive within the chosen triangle. Estimated ultimate loss for a cohort at latest observed age dad_a is

U^a=Ca,da∏d≥daf^d,\widehat U_a=C_{a,d_a}\prod_{d\ge d_a}\widehat f_d,

with an explicit terminal age and any selected tail factor. This estimator depends on a coherent cumulative triangle, accident-period assignment, and model assumptions about development. Mack (1993) derives standard errors under distribution-free chain-ladder assumptions; England and Verrall (2002) discuss stochastic models and predictive uncertainty. Neither source implies that the numerical triangle will be correct without a separately validated data-generation and reconciliation process.

A worked triangle with one erroneous transaction

Consider three synthetic accident years and annual paid development, with monetary units in thousands of dollars. The clean cumulative triangle is:

Accident year 12 months 24 months 36 months
A1 12 18 21
A2 15 22 —
A3 20 — —

The age-12-to-24 factor is (18+22)/(12+15)=40/27≈1.481481(18+22)/(12+15)=40/27\approx1.481481; the age-24-to-36 factor is 21/18=7/6≈1.16666721/18=7/6\approx1.166667. For A2, projected ultimate is 22(7/6)=25.66666722(7/6)=25.666667. For A3 it is 20(40/27)(7/6)=2800/81≈34.56790120(40/27)(7/6)=2800/81\approx34.567901. The latest clean observed sum is 21+22+20=6321+22+20=63; projected ultimate across A1–A3 is 21+25.666667+34.567901≈81.23456821+25.666667+34.567901\approx81.234568. Consequently the simple aggregate unpaid development estimate implied by these illustrative data is about 18.23456818.234568 thousand dollars, with no tail factor beyond 36 months.

Now suppose a payment of 2 (thousand) in A2 at the 24-month development evaluation is represented twice in the ingest and is accepted as a distinct economic disbursement. A2's 24-month cumulative value becomes 24 while A1 is unchanged. The apparently measured 12-to-24 factor becomes (18+24)/(12+15)=42/27=14/9≈1.555556(18+24)/(12+15)=42/27=14/9\approx1.555556. The mature 24-to-36 factor stays at 7/67/6. The projected ultimate for A2 becomes 24(7/6)=2824(7/6)=28, and for A3 becomes 20(14/9)(7/6)=980/27≈36.29629620(14/9)(7/6)=980/27\approx36.296296. The sum of the latest observed values rises from 63 to 65; aggregate projected ultimate rises to 21+28+36.296296≈85.29629621+28+36.296296\approx85.296296. The projected unpaid amount becomes approximately 20.29629620.296296, an increase of about 2.0617282.061728 thousand, despite the original duplicate of only 2 thousand.

Derived measure (thousands USD) Correct reconciliation Duplicate +2 at A2/24 Difference
Development factor 12–24 1.481481 1.555556 +0.074074
Development factor 24–36 1.166667 1.166667 0
A2 projected ultimate 25.666667 28.000000 +2.333333
A3 projected ultimate 34.567901 36.296296 +1.728395
Total projected ultimate 81.234568 85.296296 +4.061728
Latest cumulative observed 63.000000 65.000000 +2.000000
Projected unpaid development 18.234568 20.296296 +2.061728

The result matters because the error enters not only the directly affected cohort but also a development factor applied to a younger cohort. Its influence depends on volume, maturity, other observations and the selected reserving method; another triangle could yield a different direction or scale. All calculations assume fixed selection of cohorts, no tail, no trend, and standard chain-ladder averaging. This arithmetic demonstrates mechanism but does not estimate a commercial reserving error distribution.

Statistical uncertainty versus data-reconstruction uncertainty

Model variability and data uncertainty have distinct origins. A stochastic reserving method may quantify process variation and parameter estimation conditional on the supplied triangle. It does not automatically incorporate uncertainty about which transactions were valid, which claims are duplicates, or how historical system migrations were resolved. Data-reconstruction uncertainty can be assessed by calculating results for a defined set of plausible verified or unresolved cleaning decisions. If those choices dominate the range of outcomes, a narrow conditional confidence interval from the statistical model is misleading as a statement about end-to-end uncertainty.

A comprehensive report can separate: process uncertainty in future loss emergence; parameter uncertainty of the fitted model; model-form uncertainty from selecting development and frequency structures; and reconstruction uncertainty associated with identity, classification, cutoff and exposure. These are practical uncertainty categories for analysis, not a newly proposed universal statistical doctrine. Combining them rigorously requires specifying dependencies; simply adding percentage margins may overstate or understate aggregate variance.

Leakage and retrospective revision

Operational claims histories are revised. If a model is backtested on retrospective tables after all claims were settled, fields such as final paid loss, closure date, litigation outcome or last known reserve may reveal information not available at the prediction date. A model then appears more accurate than an implementable real-time system. The appropriate test reconstructs historical features as of each evaluation date and evaluates the outcome at the specified future horizon. The Kuhn–Johnson preprocessing and data-splitting framework (2013) supplies relevant general predictive practice, but a claims model additionally requires adjudication of backdated transactions and later corrections.

Random train–test splitting at the transaction level can place multiple rows from one claim in both training and test sets, producing dependence and potential leakage. Split policy, claim or event groups according to the estimand and intended deployment unit; add temporal separation where future deployment is the goal. If the ultimate target is learned from a later valuation of the same claim, do not present it as information observed when the model was supposed to be applied. Cleaned sample versions must carry an explicit as_of and a known_by constraint.

Missingness, payment delays, and censoring

For paid claims at development age dd, the analyst observes partial cash histories. A severity model restricted to claims with final settlements may exclude complex or long-tail losses. Modeling the eventual ultimate outcome requires methods appropriate to censoring, reporting and settlement delays; naive zero replacement for unknown futures yields structural downward bias in many such settings. Rubin's missingness conditions (1976) and Little and Rubin's comprehensive statistical treatment (2019) make clear that missingness assumptions matter to valid inference. The claim-specific observation mechanism and contractual obligations must be added to these general statistical conditions.

For example, an open file with no payment at month six may represent a high-severity liability dispute, a legitimate no-loss claim, an administrative delay, or a reporting placeholder. Setting all four to zero as observed final severity would destroy their differences. The correct cleaning output distinguishes observed paid-to-date, current case reserve, status, reporting age, adjudication stage, and whether ultimate severity is censored. Downstream models can then use a framework that accounts for the relevant observability rather than hiding it in an engineered number.

Diagnostics beyond the final prediction score

The primary model report should quantify how cleaning decisions alter target distributions and predictive performance. For each correction class, compare sample size, claim-event count, paid and incurred totals, average and upper-tail severity, reporting delay, settlement duration, status mix, recoveries, exposure denominator, temporal cohort composition, and downstream estimates. For predictive models, track out-of-time calibration and error across relevant segments, with uncertainty reflecting clustering or dependence when appropriate. For reserving, compare triangles, factor selections, holdout forecast errors and sensitivity to reopening and large claims. The outcome metric must be relevant to the intended decision; no generic accuracy score can replace a correctly defined target.

A clean-looking curve may be suspicious when unusual losses were systematically removed. A stable overall average may disguise increased underrepresentation in a small line or region. Data-quality review therefore benefits from stratified comparisons and an explicit exception ledger. The cross-disciplinary study of data cascades in high-stakes AI (Sambasivan et al., 2021) concerns broader machine-learning systems, not insurance-specific empirical incidence; its documented account of downstream effects from data practices illustrates why quality decisions deserve first-class attention.

Materiality and model use

Whether a residual data defect is tolerable depends on the use. A reserving analysis concerned with gross unpaid losses may be highly sensitive to missing large outstanding claims and comparatively insensitive to a duplicated zero-payment notice. A frequency tariff may be sensitive to repeated low-severity claim files even if the total financial impact is small. A fraud-screening model may be sensitive to falsely merged identities and unreviewed adverse labels. A consumer-outcome study may be sensitive to subgroup-specific incompleteness masked by aggregate averages. Decision-oriented quality assurance should evaluate the consequences of defects for the particular target rather than declare all datasets uniformly clean or unclean.

ASOP No. 43 advises the actuary to consider the nature of claims, characteristics of data, methods and assumptions in estimating unpaid property/casualty claims. Solvency II Article 121 specifies accurate, complete and appropriate data for an insurer's internal model in its applicable supervisory context. These sources support the general scientific need to evaluate data in relation to the calculations they are intended to support, while their professional and legal scopes remain distinct.

Original synthesis and research agenda

The analysis yields four connected conclusions. First, source records and economic claims occupy different grains; errors in the mapping propagate mechanically into statistical observations. Second, time and financial bases must be reconstructed explicitly; an apparently consistent set of amounts may encode the wrong date or accounting meaning. Third, cleaning uncertainty is inferential uncertainty; a single final data table is inadequate when material identity and classification cases remain unresolved. Fourth, release quality requires evidence of conservation, valid sample selection and downstream sensitivity, not merely a passed schema validator.

These are analytical conclusions drawn from established record-linkage, data-quality, accounting and actuarial literature, supported by the synthetic demonstrations in this monograph. They are not named new theories. A future empirical program can test them using legally accessible, de-identified transaction-level data with source-of-truth adjudication and an agreed historical snapshot protocol. It should measure how different linkage decisions alter claim counts and tails, compare as-known and restated backtests, estimate reviewer disagreement, and examine reserve and price sensitivity across different insurance products. The minimum evidence for empirical novelty would include a preregistered or transparently specified target, representative data, independent adjudication, reproducible code, uncertainty intervals and an explicit statement of where findings do and do not generalize.

Conclusion

Statistical cleaning and reconciliation of insurance claims data is the controlled reconstruction of an insurer's economic and administrative claims history for a specified quantitative purpose. Its fundamental object is the relationship among events, notices, files, coverages, claimants, payments, reversals, recoveries, case reserves, legal status, valuation dates and insured exposure. Each relationship defines an analysis-specific observation and an associated error risk.

A valid methodological sequence begins by defining the population, amount basis, reporting horizon, target and allowed information set. It preserves immutable raw data, distinguishes source record identities from canonical claims, identifies real transactions separately from retries and reserve snapshots, reconstructs states through time, enforces coverage and exposure join constraints, reconciles signed financial movements with documented bridges, and reports both unresolved exceptions and their downstream influence. It retains economically credible extreme losses and distinguishes observation gaps from genuine zero outcomes. Reopening contributes claim development rather than mechanically creating a new insured event.

The worked examples establish why cleaning decisions are materially connected to statistical estimation. A duplicate payment changes gross and net paid measures; a duplicate claim can increase frequency and pure premium while reducing observed mean severity; and a single development-cell error can alter reserve estimates in other accident cohorts through shared development factors. Because these calculations are conditional on reconstructed data, data uncertainty is relevant alongside process, parameter and model uncertainty. A scientific conclusion about claims experience must state the data basis on which it rests.

International insurance practice contains jurisdiction-specific contractual and regulatory obligations. The U.S. actuarial professional standards, EU Solvency II data-quality provisions, IFRS accounting requirements, general statistical research and global data-quality management guidance examined here establish complementary, scope-dependent expectations. Their shared analytical relevance is the obligation to make the origins, transformations, intended uses and limits of claims data intelligible. The resulting reproducible record permits actuaries, statisticians, claims operations, financial reporting and researchers to evaluate the same numbers with the same definitions, while preserving the distinctions among their decisions.

References