Insurance by BogdanovaThe Encyclopedia of Insurance and Risk Science

RESEARCH DOSSIER

Actuarial Data Quality and Provenance

Abstract. Actuarial conclusions are conditional on a chain of observations, definitions, transformations, valuation dates, exposure denominators, contractual classifications, statistical assumptions, and professional judgments. Consequently, data quality in actuarial science is an inferential property: a record can be accurate as stored and yet inappropriate for a particular estimate, and an apparently complete database can exclude the population to which an insurance decision applies. This monograph develops a documentary, mathematical, and comparative analysis of actuarial data quality and provenance across mortality studies, claims reserving, premium and exposure analyses, reinsurance, and regulatory valuation. It examines completeness, accuracy, appropriateness, temporal validity, identity resolution, missingness, late reporting, repeated joins, revisions, and the dependence of materiality on the final decision. Its legal and professional analysis distinguishes U.S. actuarial standards, European Solvency II valuation requirements, international supervisory principles, and general information-quality and provenance specifications. Original analytical synthesis connects source-level defects with the estimand, actuarial model, output sensitivity, and required disclosure. Explicitly hypothetical numerical examples demonstrate denominator distortion, missing outcomes, selection effects, joins that multiply claim amounts, and valuation-vintage instability. A proposed practical documentation sequence is presented as an analytical application of existing standards, not as a new proprietary model or doctrine. The central conclusion is that satisfactory actuarial evidence requires demonstrable fitness for a specified inference, reproducible lineage from raw observations to reported figures, and disclosures that translate consequential limitations into the terms of the decision being made.

Keywords: Actuarial data quality; data provenance; data lineage; completeness; accuracy; appropriateness; exposure reconciliation; insurance claims; mortality data; reserving; missing data; record linkage; valuation date; audit trail; model risk; source verification; actuarial disclosure; Solvency II; ASOP 23; W3C PROV.

Introduction

Insurance depends on observations that were collected for many different purposes. A policy administration system records contracts and coverage movements; a claims platform records notifications, case estimates and payments; a finance ledger recognizes transactions; a mortality study reconstructs the time insured persons remained at risk; a reinsurance system assigns recoveries to treaty years and layers. These systems may contain compatible numbers while expressing different legal, temporal and statistical objects. Actuarial work must make their relationships intelligible before treating the records as evidence.

The U.S. Actuarial Standards Board's ASOP No. 23, Data Quality explicitly addresses selection, review, use and disclosure of data. Its important distinction between review and audit establishes a professional boundary: a reasonable actuarial data review is not a representation that every underlying transaction has been independently audited. The European Union's Solvency II Article 82 requires internal processes supporting appropriateness, completeness and accuracy when calculating technical provisions. The two instruments operate in different legal and professional settings, yet they converge on the need to evaluate evidence against the actuarial task rather than accept source-system status as proof of reliability.

A general literature on information quality helps explain why. Wang and Strong's data-consumer quality research identifies intrinsic, contextual, representational and accessibility perspectives. In actuarial terms, a valid payment amount becomes contextually deficient if its claim is assigned to the wrong development period; an accurate date of birth becomes an inadequate measure of exposure if eligibility is established from a different effective-date field. Similarly, the ISO/IEC 25012 data quality model is a general specification for assessing structured data characteristics, while actuarial fitness remains contingent on coverage, risk, legal definition and estimand.

This research treats data provenance as the account of how an observation or result came to exist: who or what supplied it, the operation that changed it, the time and version relevant to that change, and the connection between the original and the resulting analytical object. The W3C PROV Data Model provides a general vocabulary for entities, activities and agents. Its role here is technical and evidentiary. It helps articulate lineage, while professional and regulatory sources determine whether the lineage is sufficient for a specific actuarial conclusion.

The method combines close reading of primary regulatory and professional documents, selective peer-reviewed and foundational statistical literature, exact mathematical identities, and transparent constructed examples. All numerical scenarios below are hypothetical and intended to expose mechanisms. They are not estimates of the prevalence of data defects in insurers, not empirical claims about a country or market, and not calibrated valuation advice. This research does not reproduce subscription-only full standards where access is restricted; it uses official accessible standard summaries within their documented scope.

The research object and its limits

A dataset has an observable state, a documented history and a proposed use. Its apparent quality must therefore be assessed in relation to four different questions: What did the source record? How was it transformed? Which population and quantity does the analysis intend to estimate? What decision will use the answer? An affirmative answer to the first question cannot settle the other three.

This dossier addresses the evidence chain supporting actuarial calculations. Broader governance of model choice, capital stress and insurer-wide information systems belongs to their associated scientific subjects; the present treatment follows those concerns only where a concrete data decision causes an actuarial effect. It compares selected international instruments without implying that U.S. professional standards apply everywhere or that European legal requirements operate outside their specified jurisdictions. The date of assessment is November 14, 2026.

Actuarial Data Requirements

Define the estimand before the file

An actuarial estimation task starts with an estimand: the numerical characteristic sought in a clearly defined population, period and valuation basis. For frequency rating, a possible estimand is the expected number of covered claims per earned exposure unit during a contract period. For a mortality study, it may be the central death rate in an attained-age and calendar-year cell. For claim reserving, it may be future claim cash flows as of a valuation date under the contract and accounting boundary relevant to that calculation.

Write an elementary frequency estimator as

λ^=∑i=1nNi∑i=1nEi,\widehat{\lambda}=\frac{\sum_{i=1}^{n}N_i}{\sum_{i=1}^{n}E_i},

where NiN_i is the number of claims meeting a specified inclusion rule and EiE_i is the corresponding earned exposure. The equation can be evaluated perfectly on an imperfect dataset. If the numerator includes claims arising during the period but the denominator includes policies issued rather than exposure earned, the result answers a different question. If a reinsurance ceded-claim table substitutes for the original claim table, the numerator can change without any visible arithmetic error.

Required documentation should therefore name, at minimum, the population, unit of observation, event definition, contractual trigger, observation and valuation windows, time zone and date conventions, currency and units, population entry and exit rules, source authority, known exclusions, and intended uses. These are analytical specifications drawn from the nature of the estimand and the professional guidance in ASOP 23's data-selection provisions. They are not presented as an exhaustive statutory checklist applicable in all jurisdictions.

The distinction between source completeness and population completeness matters. A claims system may include every file created in that system while omitting claims reported directly to an administrator. An exposure system may contain every issued policy while failing to capture retrospective cancellations, reinstatements, group-membership movements or changes in insured limits. Data completeness can be asserted only relative to an explicitly specified frame.

The observation unit

Insurance data are frequently nested. One insured may have several policies; one policy may cover multiple risks; a claim may contain many transactions; a transaction may have several accounting allocations; a loss may be subject to several reinsurance arrangements. The observation unit cannot be inferred safely from the row count. A table of 150,000 claim payments is not necessarily a table of 150,000 distinct claims, and a table of monthly exposure records is not a population of independent insureds.

For a mortality rate, the defensible denominator is an appropriately defined measure of person-time or lives exposed, depending on the selected method. A record containing a life with one January observation and a life with 12 monthly observations is neither a two-life sample nor automatically a 13-life sample. Its population accounting must reconstruct the time during which each individual met the insured-risk definition. Risk start, risk end, censoring, policy lapses, disability status and benefits provisions all affect that reconstruction.

For claims severity, the counting rule differs. The relevant statistical unit may be each claim, occurrence, claimant, coverage, policy, location or aggregate catastrophe event. A separately coded legal claimant need not represent an independent occurrence. Decisions about unit definition affect correlation, uncertainty and tail estimation. These are reasons to document structural relationships before selecting an estimator.

Accuracy, completeness, appropriateness and materiality

Accuracy asks whether values meaningfully correspond to the defined attribute and whether material errors occur. Completeness asks whether the intended population, variables, periods and risk groups are sufficiently represented. Appropriateness asks whether the data-generating process and observed information support the specified valuation or prediction. These properties can conflict: a highly accurate national mortality benchmark can be less appropriate to a specialized insured population than a smaller but well-defined internal study.

The European Delegated Regulation Article 19 defines detailed conditions for technical-provision data quality, including sufficient history, homogeneous risk groups, consistency, timely recording and the origins and processing methods of external data. The regulatory definition is linked to its own purpose; it should not be generalized uncritically to every actuarial study. The EIOPA valuation guidelines provide further supervisory guidance regarding data selection, examination and methods under Solvency II.

Materiality is explicitly decision-related. A typographical issue in an unrelated descriptive field can be immaterial to reserve calculations while a date offset of one day changes whether an insured event falls inside coverage. Materiality cannot be established by a universal permitted percentage of missing records. A handful of missing catastrophe claims may dominate an extreme-loss estimate. Conversely, large numbers of noncritical missing marketing fields may have no direct effect on a specified life-expectancy calculation.

A useful review sequence asks: Which quantity is being estimated? What failure would move that quantity? At what magnitude would movement affect management, reporting, supervision or policyholders? Which feasible checks would detect that failure? The sequence derives its importance from the actuarial use of information, not from a uniform benchmark invented for all insurance activities.

Exposure time, entry, exit, and censoring

An exposure denominator can become distorted even when every original policy transaction is preserved. Let individual ii be eligible for insurance experience during a set of intervals IiI_i, and let WW be the study window. Under a continuous exposure convention, a measure of time at risk can be expressed as

E=∑iduration⁡(Ii∩W)year basis⁡,E=\sum_i \frac{\operatorname{duration}(I_i\cap W)}{\operatorname{year\ basis}},

subject to the actual contract, risk definition and accepted actuarial exposure convention. The representation is deliberately generic: some studies use fractional policy-years, others person-years, lives, sums insured, vehicles, payroll or another exposure measure. Calendar-time duration itself does not determine the appropriate denominator where coverage attaches only upon payment, waiting periods, reinstatement or other qualifying conditions.

Left truncation occurs when an individual enters observation after the start of a period, and censoring occurs when complete event experience beyond a recorded endpoint is unavailable. Those conditions must be separated from policy lapse, termination of coverage, and death, because each corresponds to a different observation or contract event. A claims event occurring after a policy ended cannot be placed in the original covered-risk numerator solely because its payment took place later. Conversely, a claim paid after policy termination may correctly relate to an earlier covered occurrence. The exact classification depends on the insured event trigger and contractual coverage conditions.

For an illustrative annual study, suppose one insured automobile was covered for the full year and another for precisely half a year, and the agreed exposure basis is earned vehicle-years with uniform accrual. Their combined denominator is 1.5 vehicle-years. If one claim met the inclusion rule, the observed frequency is 1/1.5=0.66671/1.5=0.6667 claims per vehicle-year. Dividing instead by two issued policies would yield 0.5, a distinct statistic. Both are mathematically well-defined, but the second does not answer the originally defined frequency question. The example should not be interpreted as stable actuarial experience: one observed claim gives an extremely uncertain estimate.

A more subtle defect arises when data systems store cancellation as one transaction and reinstatement as another. Adding all original policy terms without removing days out of force can overstate earned exposure, whereas selecting only the last current-status row can omit legitimately covered historical intervals. An effective-dated event history is often needed to reconstruct the state of coverage over time. This is precisely where temporal provenance supports actuarial correctness: it permits the analyst to explain which event sequence produced each contribution to the exposure denominator.

In life insurance, exposure and mortality classification may also depend on issue age, attained age, smoker status, underwriting category, policy duration and benefit type. Any transformation from operational age fields into analytical strata should name the birthday convention and observation period. A one-year error in age assignment can alter age-specific death-rate estimates and subsequent projection assumptions even when aggregate death counts reconcile.

The scientific implication is that the denominator requires the same provenance discipline as the numerator. Counting claims correctly while mismeasuring earned exposure can produce a precise-looking but systematically displaced rate. A satisfactory review tests both the underlying event population and the eligibility rules that define time at risk.

Data and actuarial assumptions

Actuarial projections also require assumptions about unobserved future processes. The observed historical rate is a statistic; the assumed future rate is a modeling input. The two may be connected through trend, credibility, judgement and scenario design, but the connecting transformation must be named. The model-related duties and disclosures in ASOP No. 56, Modeling are important when a dataset enters a pricing, reserving or projection model.

Data quality must not be used to disguise model uncertainty. Perfectly corrected past claims do not establish future inflation, court-award behavior, climate hazard, healthcare use or lapse experience. Similarly, the best model cannot repair an exclusion that is unknown and statistically unidentified. Distinguishing data error, measurement error, sampling error, assumption uncertainty and structural model uncertainty makes subsequent sensitivity analysis interpretable.

Provenance and Source Lineage

Origin, transformation and accountability

A practical provenance record describes a source entity, its creation or extraction time, the system and actor responsible, a stable version or checksum, transformation steps, and the resulting output. The W3C PROV-O ontology expresses provenance relationships in a machine-readable framework; the PROV-DM specification provides the underlying conceptual distinctions.

For actuarial work, consider the chain

D0→T1D1→T2D2→T3X→M(θ)Y^,D_0 \xrightarrow{T_1} D_1 \xrightarrow{T_2} D_2 \xrightarrow{T_3} X \xrightarrow{M(\theta)} \widehat{Y},

where D0D_0 is the immutable extracted source, TjT_j are recorded transformations, XX is the model-ready analytical table, MM is a specified model with assumptions θ\theta, and Y^\widehat{Y} is the reported result. Each arrow requires enough information to reproduce or independently scrutinize its effect. Code revision, parameter file, reference table, join predicate, run timestamp and data cutoff can be more consequential to reproducibility than the name of the spreadsheet in which a final number appears.

Reproducibility requires preservation of input versions, not merely a recipe that operates on today's updated database. A transformation rerun after a claim system changed reserve amounts may produce a different answer for legitimate reasons. A proper lineage record can identify whether the difference is attributable to a new source vintage, a correction of old records, a changed method or a new assumption.

The provenance graph also has an organizational dimension. An official source and a controlled extraction process do not guarantee that subsequent processing was authorized, tested or appropriate. Likewise, a derived actuarial table assembled from a vendor feed needs an account of vendor definitions, license boundaries, collection mechanism and adjustments. The EU's Article 265 documentation rules specifically call for a directory describing data sources, characteristics and uses in the technical-provisions setting.

A concrete lineage description

A claim-reserving source lineage may include the following events, each preserving its input-output association:

  1. An operational system emits a source snapshot as of the documented extraction timestamp and business cutoff.
  2. Schema validation maps claim identifiers, policy references, currencies, payment amounts, reserve components and claim-development dates.
  3. Transactions are normalized to a stated valuation currency and classification map, preserving original transaction amounts and exchange-rate sources.
  4. A coverage join links claim records to relevant policies through time-bounded identifiers.
  5. Claim movements are aggregated into development cells without double counting line-level allocations.
  6. The actuarial model uses a fixed triangle vintage and named assumptions to produce projected cash flows.
  7. The report records the exact input version, model revision, review decisions and limitations.

The list is an analytical description of reproducible practice. It does not assert that every insurance system follows these steps. Retaining an original value together with any standardized or corrected value is especially important because a later reviewer must be able to distinguish an observed source fact from the analyst's interpretation.

A record's history may include mistakes and their correction. Provenance does not mean that the earliest recorded value was correct. It means that the origin and transformations remain inspectable so that claims of accuracy can be evaluated, corrections reversed when warranted, and material changes traced to a decision.

Temporal provenance

Actuarial work involves several different times: event occurrence, policy coverage period, loss report, payment posting, accounting recognition, extraction, valuation, and publication. These are separate variables even if databases store them in fields with similar names. An actuarial claims triangle ordinarily organizes experience by selected accident, policy, underwriting or report periods and subsequent development; choosing another origin basis changes its meaning.

An explicit temporal model distinguishes the date on which a fact became true in the insured world from the date on which a system recorded that fact. A claim incurred in December but reported in January may be absent from a December database snapshot while still being a December occurrence. A December payment posted after financial close can also have different transactional and accounting timestamps. For repeatable valuation, the query must identify which facts were known at the cutoff and how later corrections are treated.

This is a reason to retain data vintages rather than overwrite historical extracts. Late-arriving claims, retrospective adjustments and reopened claims change observed development. A genuine experience shift can be separated from a recording-process change only when relevant historical definitions, extracts and reconciliations are available.

Provenance, validation and governance

The IAIS Insurance Core Principles situate actuarial work inside broader governance, valuation and internal-control responsibilities. The international framework is not the same legal instrument as the EU delegated regulation or U.S. actuarial standards. Their common practical implication is that a documented, traceable figure is easier to challenge, correct and govern than an orphan result detached from its inputs.

The ISO 8000-61 process reference describes data-quality management processes at the organizational level. It complements transaction-level lineage, although it is not itself a specific actuarial valuation methodology. Operationally, a mature process should identify the responsible owner of an error, its affected outputs, remediation status and subsequent validation. When these fields are absent, uncertainty about the reliability of a reserve or experience study can remain even if no obvious arithmetic inconsistency appears.

Hashes, identity and evidentiary limits

Cryptographic digests can attest that two byte sequences match. They cannot establish that recorded premiums correspond to legally enforceable contracts, that every claim is represented, or that a classification is economically meaningful. A SHA-256 digest of a faulty extraction preserves proof of the faulty version's identity. Accordingly, file checksums support integrity and reproducibility while source-to-world correspondence requires reconciliation, investigation and appropriate professional judgment.

A useful distinction is between three records: the immutable source artifact; the processing log that maps it to an analytical result; and the substantive validation evidence that tests its relationship to contracts, events or accounting balances. Treating any one of these as a substitute for the others creates an evidentiary gap.

Completeness and Accuracy Checks

What a check must demonstrate

A data check should be defined by the defect it can detect and by its blind spots. Schema checks detect violations of allowed types or values but cannot establish that all eligible policies exist. Referential integrity checks detect unmatched foreign keys but may pass a systematically miscoded population. Financial reconciliations can show that a total agrees with a ledger while leaving the allocation among development periods incorrect. Statistical outlier analysis can identify unexpected values but can also flag precisely the largest insured losses.

The ASOP 23 review guidance asks the actuary to consider available data and implications for the task, while preserving the distinction between a review and a formal audit. The Article 19 requirements supply a separate technical-provision-specific legal frame for completeness, accuracy and appropriateness. Neither should be represented as a universal claim that every field can or must be audited in all actuarial assignments.

Tests should be tied to denominators and materiality. Counts of invalid dates are less helpful than counts of affected lives exposed and the direction of the possible mortality-rate bias. The percentage of unmatched claim keys is less informative than the total outstanding and paid claim amounts represented by those keys, their development profile, and whether the unmatched records concern extreme losses.

Coverage and transaction reconciliation

Reconciliation establishes relationships between independent representations of the same phenomenon, when their reporting bases can be aligned. Typical comparisons include issued and earned premium to accounting movement records, claim payments to cash and ledger movements, claim-level totals to aggregate claims reports, insured-risk exposure to policy movements, and recoverable amounts to reinsurance allocation records.

Let SS denote a source-system total and AA the corresponding actuarial aggregation after valid scope and timing adjustments JJ. The reconciliation difference is

Δ=A−(S+J).\Delta = A-(S+J).

A small Δ\Delta supports the reconciliation at the level tested. It does not establish that each underlying record is correctly classified. Both sides may also share the same faulty origin. A valid reconciliation report therefore names the independent source, units, period, timing adjustments, exceptions, reviewer and result.

A practical coverage check should identify both expected records absent from the analytical table and present records that should be excluded. A balanced premium total could conceal an omitted high-risk group and an offsetting duplicated low-risk group. Stratification by coverage, business line, geography, product version, claims status and year often matters more than a single global completeness percentage.

Hypothetical frequency distortion

Consider a fully specified hypothetical portfolio with 10,000 earned policy-years and 120 valid covered claims. The recorded frequency is 120/10,000 = 0.012 claims per policy-year. If 12 otherwise eligible claims are missing, the target numerator becomes 132 and the frequency becomes 0.0132. The observed rate understates the reconstructed rate by approximately 9.09% when measured relative to 0.0132.

Alternatively, retain the original 120 claims but accidentally add 300 duplicate earned-policy-year equivalents through a join. The recorded frequency becomes 120/10,300, approximately 0.0116505, or 2.91% below the correct 0.012. These mechanisms can operate together and in opposite or reinforcing directions. A global record-count check alone may not detect the opposing distortions.

The numerical conclusion is strictly about the stated synthetic counts. It demonstrates that the estimand must be protected on both sides of a ratio: numerator validity and denominator validity. The appropriate real-world audit evidence would also need to establish how missing claims were identified and why the corrected earned exposure is authoritative.

Missingness and selection

Missingness may arise from random operational failures, delayed reporting, records never collected, nonresponse, suppression, legal restrictions, or systematic differences in the underlying risk. The statistical consequences vary with the mechanism. Little and Rubin's treatment of missing-data mechanisms distinguishes missing completely at random, missing at random given observed information, and missing-not-at-random cases under the relevant statistical modeling definitions.

An actuarial dataset often contains several missingness processes at once. A claims database can omit some incurred but not reported events by design at a valuation date; a medical variable may be absent because a claim is too small to trigger a report; a policy's location may be absent because the insurer migrated legacy systems. These mechanisms have different implications for identifying the quantity of interest and cannot be grouped into a generic 'missing rows' percentage.

Let Ri=1R_i=1 indicate that an outcome YiY_i is observed. A complete-case mean estimates E[Y∣R=1]E[Y\mid R=1]. It estimates the population E[Y]E[Y] without correction only under conditions that justify equality or provide a defensible adjustment. An inverse-probability estimator is sometimes written

μ^IPW=∑iRiYi/π^i∑iRi/π^i,πi=P(Ri=1∣Xi),\widehat{\mu}_{IPW}= \frac{\sum_i R_iY_i/\widehat{\pi}_i} {\sum_i R_i/\widehat{\pi}_i}, \qquad \pi_i=P(R_i=1\mid X_i),

where the expression is a normalized weighted estimator. Its use requires suitable observation probabilities, positivity for relevant population strata and correctly specified conditions under which observed covariates explain selection. Weighting cannot identify outcomes from strata that never appear and is not automatic evidence that missingness is ignorable.

The methodological relevance of this problem is well illustrated by Kahn and colleagues' data-quality assessment framework, developed for secondary uses of electronic health records. The study is not an actuarial standard and should not be treated as one. Its distinction among conformance, completeness and plausibility is nevertheless informative for health insurance datasets that repurpose clinical records originally collected for care.

Sensitivity when missing outcomes are unknown

Suppose 9,000 of 10,000 otherwise eligible hypothetical observations have an average claim cost of $1,000. The mean across the complete target group cannot be calculated from this fact alone. If the 1,000 missing outcomes would average mm dollars, the target mean equals

μ(m)=9,000(1,000)+1,000m10,000=900+0.1m.\mu(m)=\frac{9{,}000(1{,}000)+1{,}000m}{10{,}000} =900+0.1m.

At m=$1,000m=\$1,000, the mean is $1,000. At m=$1,200m=\$1,200, it is $1,020; at m=$2,000m=\$2,000, it is $1,100. These values are not predictions. They are sensitivity scenarios showing the amount of information absent from the observed mean and how the final estimate depends on an unobserved quantity.

If claim cost can be credibly bounded between LL and UU, a simple partial-identification interval is [900+0.1L, 900+0.1U][900+0.1L,\ 900+0.1U]. Such bounds are scientifically useful when plausible support restrictions can be justified. A narrow interval derived from an unjustified upper bound would be more misleading than the broad uncertainty it conceals.

Correcting records and estimating unobserved information

Data preparation involves at least two epistemically different operations. A source correction replaces a documented erroneous fact with a better-supported value; an estimation procedure supplies a value or distribution for a quantity that was never adequately observed. The first requires evidence supporting the correction. The second requires a statistical or actuarial assumption, together with uncertainty and a stated inference target. Conflating these activities can disguise an estimation model as routine cleaning.

For example, a confirmed duplicate $50,000 claims payment can be removed from an aggregate if independent transaction identifiers establish that the same payment was ingested twice. By contrast, a claim with a missing payment status cannot be deleted merely because it resembles a duplicate. Record linkage may support a probable match, but its false-match and missed-match rates may affect the resulting financial aggregate. If an analyst resolves all uncertain matches by selecting the lowest possible claim amount, the reported reserve can become systematically biased.

A simple deterministic correction table should preserve the original field, corrected field, reason, evidence source, responsible reviewer, and timing. Probabilistic adjustments require additional documentation: what feature values determine estimated missingness, why those features are relevant, whether observation probabilities are positive, and how sensitive the estimate is to assumptions. A missing value imputed from an observed group average is not subsequently an independently observed claim amount.

In a population with nonrandom missing large claims, naive mean imputation can attenuate the right tail. Suppose an illustrative sample contains 90 observed claims averaging $5,000, while ten unobserved claims are independently known to belong to a higher-severity class but have unknown amounts. Imputing $5,000 for every missing claim yields an overall mean of $5,000 by construction; it does not establish the mean of the higher-severity class. If those ten unknown claims averaged $20,000, the combined mean would instead be (90⋅5,000+10⋅20,000)/100=$6,500(90\cdot 5{,}000+10\cdot20{,}000)/100=\$6{,}500. Both numbers follow from transparent hypothetical inputs. The data themselves do not determine which scenario applies.

This distinction matters most when a missingness mechanism is linked to the severity or probability of the event. Large liability claims may receive different handling, catastrophe observations may enter external reporting channels, and medical data may be available only for particular care pathways. The analyst should seek additional direct evidence before relying on distributional extrapolation. When new observations cannot be obtained, a bounded sensitivity exercise or appropriately qualified estimate can communicate uncertainty more faithfully than an undocumented single imputed value.

A complete statistical treatment also considers whether missingness changes covariate distributions. Missing a cost outcome for an insured still leaves that insured in the population denominator if eligibility is established; missing an entire policy can remove both its exposure and its claim experience. These processes should not be represented by the same missing-rate statistic. Reconciling population counts, outcome availability and key covariates at successive selection stages is necessary to identify the direction in which inferential bias might arise.

There is no universal correction that succeeds across these situations. The defensible action is determined by the observed evidence and by the estimand. A corrected ledger item, an inferred claim amount and a scenario bound can all be useful, but they should remain distinguishable throughout the analytical record.

Timeliness and claims development

Completeness depends on time. An insurer may have complete data for all claims reported by a cutoff and incomplete information about claims already incurred. In reserving, the data-quality task therefore includes characterizing and modeling reporting and settlement development. Late receipt is not necessarily a defective process; it can be intrinsic to the insurance claim-generating mechanism. A separately identifiable defect occurs when the extraction omits transactions that were in fact recorded and should have been included as of the defined cutoff.

For life and health experience, delayed death or diagnosis reporting can similarly alter observed numerator counts. Backdated cancellations affect exposure denominators. Checks should compare successive vintages and describe what each reflects rather than simply deem the oldest extract inaccurate.

Accuracy, plausibility and anomaly triage

Numerical plausibility is contextual. A negative payment can be a reversal, not an error. An exposure less than zero usually indicates an erroneous construction or deliberately signed adjustment requiring explanation. A claim with an enormous amount can be an important catastrophe observation. Rule design should therefore distinguish impossible states from rare but genuine outcomes and from legitimate accounting adjustments.

Anomalies should retain the original source values and the evidence supporting their treatment. Replacing unusual claims with mean amounts without a documented estimation purpose modifies tail behavior and may understate aggregate risk. A corrected data table should record transformations, exception flags and the analytical consequences of each material adjustment.

Joins, Revisions, and Transformation

Joins express statistical and contractual assumptions

Joining insurance tables is an analytical operation. It specifies which policy version governs a claim, which claims belong to a catastrophe occurrence, which insured person was exposed at a given date, and which reinsurance terms apply. A database join condition is therefore partly an assertion about the real insurance relationship being reconstructed.

Suppose a claims table contains one row per claim and a payment table contains many rows per claim. A one-to-many join repeats the claim-level attributes. Summing a claim-level incurred amount after the join will count that incurred amount once per payment row. A successful SQL execution and a complete set of matched keys provide no assurance that the resulting aggregate has the intended grain.

A reliable join review therefore specifies expected cardinalities, unique-key rules, permitted unmatched records, temporal validity, scope of reference tables, and aggregate pre- and post-join controls. It also examines composite keys and identifier recycling following migrations. The effective policy version at loss occurrence may differ from the version at extract date, making an untimed latest-record join economically incorrect.

Hypothetical reinsurance example

Assume 120 unique gross claims with an aggregate paid amount of $1.20 million. Seven claims have two allocation rows because they affect multiple treaty or accounting categories. Joining claim-level paid totals to allocation rows increases the result to 127 rows. If each of those seven claims has $10,000 in gross paid losses and the analyst sums the repeated gross amount in the expanded table, the aggregate becomes $1.27 million, overcounting $70,000, or 5.83% of the correct $1.20 million.

The example assumes all seven duplicated claims have exactly two allocation rows and that no other row is altered. The correct procedure may be to aggregate allocations separately at their natural grain, or to construct an explicitly reconciled allocation of each claim amount whose shares sum to the gross amount. Which treatment is appropriate depends on whether the target is gross claims, ceded share, treaty recoverable or a separate contract-year attribution.

The key insight is structural: a join may preserve each source table's true records and still generate a false financial total through multiplication. A provenance record that captures the join keys and expected cardinality makes the failure detectable. A generic completeness score on the expanded table does not.

Loss-development vintage and cohort integrity

Loss-development triangles are sensitive to definitions of origin period, development age, claim status, paid versus incurred amounts, currency basis and valuation-date knowledge. A change in a system's claim-reporting conventions can alter observed development patterns even without a comparable change in economic losses. A triangle assembled across system migrations should identify which periods and cohorts are affected.

For a cohort with cumulative paid losses C12C_{12} at 12 months and C24C_{24} at 24 months, the age-to-age factor is f=C24/C12f=C_{24}/C_{12}, provided these values measure a consistent cohort under the chosen development convention. Consider a synthetic dataset with C12=$10C_{12}=\$10 million and C24=$14C_{24}=\$14 million. The observed factor is 1.40. If an additional $1 million of 12-month cumulative payments was wrongly assigned outside the 12-month cutoff, while the 24-month total of $14 million was correctly captured, the corrected factor is 14/11, approximately 1.2727.

The difference does not establish the correct factor for another cohort or that a reserve should mechanically be reduced. It demonstrates that a time-classification defect can be mistaken for development behavior. Analysts should distinguish changes to the underlying cohort from changes in the recording and measurement of that cohort.

A worked cross-system reconciliation

Consider a hypothetical property insurance portfolio and three source tables at the same agreed financial cutoff. The claims platform shows gross paid transactions totaling $9.8 million, including $0.2 million of payments subsequently reversed within the same reporting period. The cash ledger reports net claims cash outflows of $9.6 million. An actuarial payments extract reports $9.5 million because $0.1 million of correctly posted transactions was inadvertently excluded by a change to a transaction-status filter.

If the claims-platform total is based on positive payment transactions before reversals, the relevant reconciliation is $9.8 million less $0.2 million equals $9.6 million. The ledger therefore agrees with the claims platform once the prescribed treatment of reversals is aligned. The actuarial extract remains short by $0.1 million. A comparison of only the raw $9.8 million and the extract's $9.5 million would instead show a $0.3 million difference and mix a legitimate accounting adjustment with an extraction failure.

At the next level, the analyst must determine whether the omitted $0.1 million belongs to one origin period or several. Restoring the correct grand total without restoring the correct development cohorts may leave reserving factors wrong. In a catastrophe study, the same transactions might also need assignment to an event identifier, peril and geographic footprint. The reconciliation of gross payments is one control, and the reconciliation of event allocation is another.

A third control addresses recoveries. Suppose a treaty allocation file reports $1.5 million of expected reinsurance recoverables for these claims, while a general ledger shows $1.3 million recognized. The difference does not by itself establish an error: the figures may employ different recognition, timing, contractual or recoverability assumptions. A proper reconciliation first identifies whether the two systems are intended to represent the same measure. If they are, the $0.2 million difference must then be traced to individual claims, recoverability states, layer terms or accounting adjustments rather than plugged by an unexplained balancing entry.

This scenario shows why a reconciliation is fundamentally an assertion about equivalence. Values should agree only after agreeing on the definition of the thing being measured. A close match can occur by coincidence, and a difference can be legitimate when definitions differ. Evidentiary quality depends on the transparency of the bridge between the representations.

The transformation record supporting such a reconciliation should therefore retain exact source totals, classification rules, period treatment, known adjustments, exclusions, exception populations, dates and named responsible roles. Where reported material figures depend on manual corrections, their amounts and rationales belong in the evidence trail. A final 'reconciled' badge cannot substitute for this substantive account.

Currency, inflation and transformations

Historical paid losses may require conversion for comparative analysis. The monetary unit, exchange-rate convention, valuation date, inflation adjustment and consistency between exposures and losses must be documented. Applying a current spot rate to historic payments is different from converting each payment at its transaction date. Both may be meaningful for different actuarial tasks; a choice must be tied to its target measure and valuation basis.

Reinsurance contracts may define settlement currencies, attachment points and limits that cannot be reconstructed from a gross-claims currency conversion alone. A currency transformation is an input to the contract model, not a substitute for reading contractual terms. Similar caution applies to inflation adjustments: price-level comparability can support an experience study, but a blanket index cannot by itself capture social inflation, benefit redesign or judicial shifts.

Revisions, backfills and effective dating

Every material revision should be described as one of several distinct phenomena: new experience after the prior snapshot, corrections of earlier facts, revised contractual classifications, new reference mappings, or changes in the analytical algorithm. These distinctions matter because they have different implications for comparing model runs.

A robust revision record associates old and new values with their source versions and correction reasons. Late reported events should not automatically be overwritten into a historically 'as-known' database when the objective is to reconstruct what was available at a former decision date. Conversely, final-outcome experience studies may legitimately use later observations to assess the quality of earlier estimates.

The technical-provisions Article 264 validation requirement addresses at least annual validation and validation when relevant circumstances indicate that data, assumptions, methods or technical provisions may no longer be appropriate. A reproducible, versioned dataset is essential to determining whether a changed reserve reflects new information, new methods or a previous defect.

Reconciliation at multiple grains

A reasonable control design reconciles across claims, transactions, policies, development cohorts and accounting totals where relevant. It captures join multiplication, omissions and timing differences that can cancel in a grand total. The appropriate range of controls depends on risk and materiality.

For example, a total incurred loss reconciliation can pass despite a transposition of losses between accident years. A per-year reconciliation may detect it; a per-coverage reconciliation can uncover whether reinsurance classification is affected. A sequence of increasingly granular controls generally makes it easier to localize discrepancies, but it also requires stable definitions and agreed cutoff rules.

Fitness for Intended Purpose

A dataset can be valid for one purpose and unsuitable for another

The same insurer data may be adequate for cash reconciliation yet inadequate for estimating ultimate loss, because cash totals do not contain incurred-but-not-reported losses. It may support an annual gross-loss report while lacking contractual details needed for a net reinsurance recoverable. It may accurately identify current insureds but omit the historic exposure episodes needed for a mortality trend.

This context dependence is consonant with Wang and Strong (1996) and the professional emphasis on intended use in ASOP No. 23. Fitness should be defended with specific evidence: the relevant population, time structure, variables, transformations, remaining limitations, and sensitivity of the result. A universal label such as 'gold standard dataset' has little scientific content without that account.

From data defect to estimation impact

Let an actuarial output be g(D,θ)g(D,\theta), where DD is the data and θ\theta represents chosen assumptions or model parameters. A perturbation δD\delta D changes the output by

Δg=g(D+δD,θ)−g(D,θ).\Delta g=g(D+\delta D,\theta)-g(D,\theta).

When a smooth numerical representation is legitimate, a first-order approximation can use a gradient, Δg≈∇Dg⊤δD\Delta g\approx \nabla_Dg^\top\delta D. Many insurance data defects are discrete structural changes—omitting a catastrophe, reallocating an accident year, reassigning a coverage—so the differential approximation may be inappropriate. Exact recomputation under documented alternatives is then preferable.

The appropriate sensitivity result reports the magnitude, direction, assumptions and affected decision. A reserve increase of 1% may be material in a highly constrained solvency context, and a larger percentage may be immaterial to another explicitly defined task. No universal threshold can replace the required decision analysis.

A sensitivity exercise should change one defect or scenario at a time where possible, followed by joint scenarios where mechanisms interact. Simply reporting a best and worst endpoint without explaining whether the component changes can coexist may exaggerate or understate the plausible range.

Interaction with actuarial models

Model validation and data validation overlap, but they test different propositions. A model may correctly implement a distributional assumption on a faulty dataset; a data extraction may faithfully reproduce transactions while the model misinterprets loss-development structure. ASOP No. 56's modeling guidance addresses model purpose, assumptions, limitations and communication within its applicable professional context.

An actuarial validation report should be capable of identifying whether a defect is upstream in record selection, inside a transformation, in model specification, or in subsequent aggregation and reporting. This separation makes remediation targeted. It is also essential for assessing whether an apparently improved fit is genuine or an artifact of post hoc cleaning against the same observed outcomes.

The European Union's Article 48 actuarial-function duties explicitly include assessment of data sufficiency and quality, the appropriateness of methods, and comparison of best estimates with experience in its regulatory setting. These duties show why reviewing data and assessing model performance should be connected in the institutional workflow while retaining their distinct evidentiary claims.

Missing data and identification

Suppose a health insurance experience study lacks diagnosis codes for a subgroup systematically treated outside the principal provider network. The issue is not solved merely by increasing the number of records from the network already observed. The missing subgroup can differ in severity and cost, producing a population-selection problem. The statistical treatment depends on available covariates, plausible observation mechanisms and the comparability of outside-network experience.

The Little–Rubin missing-data framework clarifies that assumptions about the missingness process matter to inference. Imputation, weighting and scenario analysis may be useful, but each introduces assumptions and should be tested for identifiability and sensitivity. No amount of numerical sophistication identifies an unobserved group if critical variables and selection probabilities are unknowable without additional evidence.

When estimates depend strongly on unidentified quantities, an actuarial communication can honestly provide plausible ranges, conditional results and the additional information that would materially narrow uncertainty. Claiming a precise point estimate without describing these conditions would overstate the evidentiary basis.

External datasets and transferability

Third-party mortality tables, catastrophe exposure sources, medical classifications and economic indices can supplement internal experience. They may also be measured for different populations, time periods or definitions. An external dataset's sample size and source reputation cannot alone establish that its risk characteristics are transferable to the insured portfolio.

The Article 19 external-data provisions require consideration of data origins and methodologies, historical changes and portfolio representativeness for the specified EU technical-provisions use. More generally, actuarial analysis should document the reason external data were selected, how they were adjusted, where internal and external sources disagree, and how the conclusion depends on the adjustment.

The insurer should distinguish independent validation from circular corroboration. An external benchmark derived largely from the same industry submissions can share reporting biases with internal experience. Two estimates are not independent simply because they have different file names or vendors.

Fitness across insurance branches

Life insurance experience requires correct death-event attribution, attained ages, benefit conditions, lapses and exposures at risk. Health insurance adds diagnostic coding, provider behavior, treatment timing, eligibility and service intensity. Property and catastrophe insurance demand location resolution, insured values, peril classifications, event grouping and tail sensitivity. Liability insurance presents reporting delays, reopened claims, legal changes and long-tail settlement development. Reinsurance data introduce contract-year, layer, attachment and reinstatement rules.

These distinctions influence the consequences of otherwise similar errors. A one-month delay in a short-tail property claims feed can be material to current settlement analytics, while a one-month difference in a very long-tail liability data series may be primarily a vintage-accounting question. A geocoding error matters enormously for location-driven hazard concentration, whereas a birth-date error is central to mortality exposure. Quality controls should be designed around the particular mechanism by which a variable enters an actuarial result.

Regulatory boundaries of approximation

In the European technical-provisions context, Article 20 on data limitations requires documentation of limitations, remediation and responsible functions, together with suitable retention of data before adjustments. The Article 21 approximation conditions constrain when insufficient data may justify approximation. The conditions matter: an undertaking cannot assume that any internal data failure can be cured simply by declaring a proxy estimate.

This regulatory discussion applies to the specified Solvency II framework. It does not purport to provide a global permission rule for every insurer or any personal regulatory advice. In other jurisdictions, applicable law and actuarial professional standards should be identified directly before drawing compliance conclusions.

Disclosure of Data Limitations

Disclosure as part of the scientific result

A reserve or premium estimate is incomplete as a scientific communication if material assumptions about evidence remain unexpressed. A reader must be able to determine what was observed, what was derived, what was assumed, where the data are deficient, and how those deficiencies influence interpretation. The ASOP No. 41 communications standard provides professional guidance on actuarial communication and disclosure in its scope. ASOP No. 23 separately addresses data-related limitations and the use or reliance on information supplied by others.

Effective disclosure is specific. A sentence stating that 'the dataset may contain inaccuracies' is weak if the known issue is that historical commercial auto claims before a system migration cannot be consistently linked to coverage limits. The second description tells the reader which estimate may be affected, which comparisons are questionable and what additional testing would matter.

Data quality may be adequate for one reporting purpose despite unresolved minor defects. The disclosure should state that conclusion with its basis and boundary. Where a material issue prevents a reliable answer, the report should identify the affected quantity and scope rather than convert uncertainty into a declaration of completeness.

What a material limitation statement contains

A substantive limitation statement can be organized into six elements:

  • The affected source, cohort or variable and the relevant time and contractual boundary.
  • The specific defect or unobserved mechanism, with the date or version at which it was identified.
  • The control, reconciliation or analytical evidence that detected it.
  • The quantities and decisions potentially influenced.
  • Adjustments, alternatives, scenarios and residual uncertainty, with explicit assumptions.
  • Remediation owner, status and reproducible input or analysis version where organizationally relevant.

This is an explanatory synthesis of established professional, regulatory and documentary requirements, not a new named standard. The reporting detail should be proportional to the decision, and should preserve confidentiality and lawful processing boundaries.

The Solvency II Article 265 provisions directly require documentation of technical-provisions data sources and processes within their scope. International supervisory expectations described in IAIS ICPs and ComFrame reinforce the role of reliable governance and valuation, while their legal implementation is jurisdiction-dependent.

Communicating model and evidence dependencies

A disclosure should distinguish data limitations from actuarial judgment. If an observed claims development pattern changes following a system migration, the report may contain an original series, a reconstructed series and a sensitivity comparison. It should explain how each was created and why one is selected for the principal calculation. If external data are used to augment a sparse internal portfolio, the population transfer assumptions should be stated with the basis for their selection.

The Modeling ASOP's disclosure requirements cover model purpose, important limitations, reliance and other matters within the standard's applicability. A combined data-and-model disclosure makes clear which uncertainty originates in data, which arises from model choice, and which concerns projected future experience. This supports appropriate scrutiny without overwhelming the reader with undifferentiated caveats.

Sensitivity results belong close to the affected conclusion. If missing payment records could move an estimate materially under credible scenarios, the reader should see the scenario range and underlying assumption alongside the estimate, not in a detached general footnote that leaves the numeric conclusion falsely precise.

Provenance in the actuarial report

A full source lineage graph need not always appear in the main narrative. It must be available in a form that permits another suitably qualified reviewer to reconstruct the important chain from input to output. Report-level provenance may specify source names, valuation date, extraction version, material corrections, methodology and annex identifiers; the technical audit record can hold detailed transformations and checks.

The distinction protects readability. A scientific monograph explains why a conclusion follows, while the underlying evidence ledger provides the documentary route needed to verify it. Opaque claims of 'data cleansing' or 'AI-driven quality improvement' cannot replace information about which records were changed, how decisions were made and what effect the changes had on the estimand.

For automated extraction and processing, provenance also supports accountability. A classifier can assign claim types with measurable error, and its version may alter grouping and reserves. Traceability should retain the classifier or ruleset revision, training or reference scope if applicable, the override history and the affected portfolio. This is a practical application of provenance principles; it does not establish a universal regulatory requirement for every automated insurance process.

Comparative synthesis

The comparison of official sources reveals a coherent structure with important distinctions. U.S. ASOP 23 addresses the actuary's handling of data and emphasizes review, selection and disclosure rather than requiring an audit in all cases. ASOP 56 extends relevant responsibilities into modeling and the communication of limitations; ASOP 41 concerns the form and completeness of actuarial communication. The EU's Solvency II provisions establish legally specified technical-provision data conditions and documentation duties for undertakings subject to that regime. IAIS standards offer a framework for supervisory governance and valuation. ISO and W3C standards provide reusable concepts for data quality and lineage, while statistical scholarship explains why missingness and selection can make an apparently clean dataset inferentially unreliable.

These instruments are complementary but not interchangeable. A PROV-compatible record does not automatically satisfy a technical-provisions data-quality requirement. A professional actuarial review is not a substitute for the insurer's financial controls. A comprehensive governance process does not independently identify missing outcomes. The most defensible approach connects each discipline to the precise proposition it can establish.

Original analytical conclusions

First, data quality is inseparable from the intended actuarial quantity. File-level correctness is a limited input to inferential validity. This conclusion follows directly from the examined standards and the constructed ratio and selection examples; it is an analytical synthesis, not a newly claimed theory.

Second, provenance is necessary to distinguish an evolving insurance world from an evolving record of that world. Without preserved vintages and transformation history, changes in results cannot be attributed reliably among newly observed outcomes, corrected histories and altered methods.

Third, the unit of analysis is itself an evidence requirement. Row-level validity does not prevent duplication after joins, misplaced development periods or misallocated reinsurance amounts. Cardinality and temporal joins must be evaluated as substantive actuarial choices.

Fourth, materiality is a relationship between a possible data defect and a particular decision. It should be evaluated through the affected estimate, uncertainty and reporting purpose rather than by data-cleaning counts alone.

Fifth, disclosures should convey the consequences of defects for the reported answer. Identifying a problem, its evidentiary basis, plausible effect and remaining uncertainty is more informative than a generic statement of imperfect data.

Together, these conclusions support an integrated view of actuarial evidence as a traceable chain from the insured phenomenon through observation, transformation and inference to professional communication. They do not establish a new named model, formal doctrine or exclusive method.

Conclusion

Actuarial data quality consists of the demonstrable suitability of observations and derived information for a specified insurance inference. Its assessment requires clear definitions of covered events and populations, accurate and complete exposure and claim measures, appropriately bounded observation periods, justified external evidence, controlled transformations, reconstructable versions and a substantive account of remaining uncertainty.

Provenance supplies the traceability necessary to investigate how each result was produced. Completeness and accuracy controls test the correspondence of stored records to the analytical task. Statistical methods evaluate what missing, delayed or selected observations permit the actuary to infer. Mathematical sensitivity analysis translates defects into possible effects on frequency, severity, mortality, reserving and related outputs. Professional and regulatory requirements determine what must be assessed and communicated within each applicable framework.

The principal scientific finding is that a valid-looking number can embody a materially invalid inference. The remedy is a documented, testable relationship between the observation and the question to which the number is offered as an answer. The practical value of this relationship extends beyond correcting individual records: it allows a reader, reviewer or responsible decision-maker to understand which aspects of an actuarial conclusion are supported by observed evidence, which depend on assumptions, and where additional data or qualification would materially change the judgment.

References