RESEARCH DOSSIER
Insurance Consumer Understanding Testing and Disclosure Design
Abstract. Insurance consumer understanding is an empirically assessable capacity to identify, interpret, apply, and act on the economic and contractual terms of insurance at the relevant decision point. This monograph examines the design and validation of disclosures across policy acquisition, renewal, servicing, and claims. It integrates insurance economics, consumer cognition, experimental methodology, measurement science, and international conduct regulation. A structured narrative synthesis of selected primary experiments, regulatory instruments, official implementation resources, and research summaries distinguishes objective understanding from perceived clarity, choice behavior from preference-consistent welfare, and standardized presentation from justified product governance. Evidence from European non-life insurance format testing, Australian home contents and add-on insurance experiments, United States health insurance research, and British renewal field trials demonstrates context-dependent benefits and limitations of disclosure. The analysis argues for contract-faithful, accessible information tested against concrete transfer and decision tasks; harm-sensitive assessment of effects across consumers; and lifecycle monitoring with remediation when communication changes alone cannot resolve product or process failures. The research supplies an integrated practical assessment of established methods and evidence, without proposing a new named theory or claiming original participant data. It identifies where causal claims remain bounded by simulations, sample selection, outcome definitions, and institutional context.
Keywords: Insurance consumer protection; disclosure design; consumer understanding; insurance literacy; comprehension testing; experimental design; market conduct; product governance; risk communication; vulnerable consumers; insurance choice; claims communication; causal inference.
Introduction
Insurance is purchased before the uncertain event that gives the contract economic significance. The buyer pays a known or scheduled amount in exchange for an insurer's contingent performance under specified conditions. The useful characteristics of this exchange are distributed across the policy wording, schedule, endorsements, exclusions, definitions, claims procedures, and communications that accompany the contract. A purchaser can understand the premium and still misunderstand the transaction in the states of the world that matter most. Information design is therefore a substantive component of the distribution of risk.
The principal question of this monograph is how an insurer, intermediary, or supervisor can establish that a disclosure enables the affected population to understand insurance and use that understanding in a decision. The answer requires a chain of evidence. A faithful document must reach its intended audience in a usable form; the recipient must be able to locate and interpret the relevant information; the resulting interpretation must support reasonable action; and the institution must monitor whether the communication continues to function when circumstances, products, and customer needs change. The analytical distinction between these stages makes failures diagnosable.
This study addresses the comparative design and assessment of disclosures, rather than the entire law of insurance formation, the semantics of any particular exclusion, readability as a discipline, or the experimental evaluation of advisory services. The unit of inquiry is a specified disclosure intervention and the decision it is intended to support. Its primary scientific home is Insurance Governance and Market Conduct; Insurance Ethics and Fairness, Insurance Contract Law, and Insurance Consumer Protection supply complementary disciplinary perspectives.
The research method is a purposive structured narrative review. The evidence was selected for relevance to three questions: which design changes are associated with measurable improvement; which measurements and identification strategies justify conclusions about comprehension and decisions; and how do regulatory obligations and governance processes respond to heterogeneous outcomes? Sources include original experimental reports, author and publisher abstracts where full access was restricted, regulatory text with explicit version identification, and official supervisory or implementation summaries. Evidence was assessed at the level of the proposition supported. Numerical effects are described in the original units and with their stated endpoints. Studies are not combined into a pooled effect, because products, treatments, populations, and endpoints differ materially. Search and selection are not exhaustive. No systematic review registration, independent double screening, new fieldwork, or general legal opinion is represented.
The analysis uses hypothetical insurance examples to explain experimental and statistical principles. These are explicitly illustrative and do not prescribe products or estimate population parameters. Regulatory comparison is restricted to the identified instruments and versions, rather than a survey of every local obligation. This qualification is central to scientific transfer: a sound experimental method may generalize more readily than its numerical results.
Information Duties, Consumer Cognition and Insurance Choice
The economic content of disclosure
A consumer choosing insurance compares distributions of future financial consequences. Let p denote the premium, L a loss, and I(L,x) the indemnity under contract terms x in the relevant circumstances. In a simple indemnity setting, the consumer's financial outflow is p + L - I(L,x), before other costs and recoveries. Contract terms determine the response function rather than merely describing it. A limit may truncate the insurer's payment; an excess may allocate part of each loss to the insured; a coverage definition may exclude an entire category of event; and procedural duties may affect rights according to the contract and applicable law. Benefit and fixed-sum policies require a different payoff specification.
An effective disclosure must therefore communicate the conditions that transform a loss or insured event into a contractual payment. Listing every contractual sentence in sequence does not by itself accomplish this. For the purchaser, the question is often conditional: if this event happens, what will the policy do, what will I pay, and what must I do? For the insurer, there is a parallel question: can the target consumer perform that inference correctly using the information we provide?
The insured's ability to make the inference depends on prior knowledge, time, language, numeracy, attention, trust, and the structure of the choice environment. A financially sophisticated buyer may still fail to notice a material change in renewal terms. A buyer with modest numeracy may select appropriate protection if outcomes are compared consistently. The information burden is a joint property of the product, the disclosure, and the consumer's circumstances. Treating all failures as deficiencies of the consumer obscures solutions available in product design and distribution.
Information costs and revealed preference
In economic analysis, observed choice is frequently treated as evidence about preferences. In insurance this interpretation depends on the purchaser's knowledge of the alternatives and feasible actions. A choice of a higher-deductible policy may reflect liquidity, expected expenses, premium differences, network access, misunderstanding, or a rational willingness to retain risk. Even a description of apparent dominance must specify the relevant dimensions and assumptions.
Bhargava, Loewenstein, and Sydnor (2017) examine employee health-plan decisions in a menu that included financially dominated alternatives. Their study of dominated plan choices uses supplementary hypothetical-choice experiments to examine explanations and identifies misunderstanding as important in the studied setting. The substantive implication is that risk preferences cannot be reliably inferred from a decision when the available options were misunderstood. Dominance is a useful experimental diagnostic only when premium, cost-sharing, relevant benefits, network restrictions, tax effects, and other material attributes have been reconciled. It is not a universal ranking of insurance products.
Access also has economic meaning. Kling et al. (2012) show in their Medicare comparison friction experiment that sending individualized cost comparisons can change behavior even when the underlying information is already obtainable. The comparison cost includes search, interpretation, organizing alternatives, and acting before a deadline. Disclosure availability is an institutional condition; usable information is a behavioral achievement.
An important implication for insurers is that the relevant comparison is often between entire communication journeys. A well-written document hidden after payment may be less decision-useful than a concise explanation visible while selecting coverage. Conversely, prominent summaries can create a distorted picture if they omit critical limitations or fail to connect to complete terms. Information design and placement should be investigated jointly where the commercial journey makes them inseparable.
Expectations, persuasion, and the proper object of protection
Insurance markets contain legitimate reasons for consumers to purchase protection even when expected indemnity falls below premium. Protection can stabilize resources, prevent severe hardship, meet legal or contractual requirements, or secure services that are difficult to obtain after loss. The normative test is whether an individual can make an informed, feasible choice consistent with relevant needs and constraints. A regulator or insurer should resist using a predetermined desired sales response as the definition of consumer understanding.
This matters when disclosure is tested in the presence of marketing. A low price shown beside a prominent promotion may focus attention on immediate cash flow while shifting attention away from limitations. A warning may reduce sales but also discourage buyers for whom coverage is valuable. Experimental outcome definitions should therefore distinguish recognition of price and scope, intentional selection, mismatch, and incomplete or abandoned transactions. The objective is consumer welfare under the actual contractual trade-offs, rather than an increase or decrease in conversion.
Disclosure has additional institutional functions. It can document the content represented before agreement, provide durable reference during a dispute, explain post-sale duties, and support comparison by intermediaries or researchers. These functions may persist even where a direct behavioral intervention has little measurable effect. Scientific evaluation specifies the intended function before judging the document.
Operational Definitions and Measurement of True Understanding
An operational construct
For the purposes of this investigation, consumer understanding means the ability to derive a correct, decision-relevant interpretation from available product information and use it in an appropriate task. This includes four distinct performances: locating relevant terms, interpreting those terms, applying them to a new but specified situation, and identifying a feasible next action. This operational description organizes measurement; it is not a claim of a newly discovered psychological construct.
A consumer can correctly recall the word deductible yet calculate the wrong payment. A consumer can accurately answer a monetary scenario yet be unaware of a reporting deadline. Another can select an economically attractive product without understanding it, perhaps by chance or through outside assistance. Accordingly, immediate knowledge, transfer, action, and confidence should not be collapsed without justification.
A test blueprint starts with a mapping from contractual provisions to decision tasks. Each item must identify the relevant policy version, the condition being tested, the intended population, the correct answer and rationale, the critical misunderstanding, and the scoring rule. An item should provide enough information to make the answer determinate. An uncertain legal interpretation or an unprovided fact is unsuitable for an objectively scored coverage question. In realistic scenarios, identifying that further information is required may itself be the correct answer.
Measurement instruments and scoring
Closed questions produce reproducible scores and can distinguish particular misconceptions via carefully constructed distractors. Open questions reveal unanticipated interpretation errors. Think-aloud observation can show where a person looks, how terms are distinguished, and where a mistaken inference enters the decision. Confidence ratings capture metacognition. Behavioral tasks observe whether the person can use information to compare products, find help, or report a claim.
These instruments serve different purposes. In the randomized study of uninsured consumers, Politi et al. (2016) assessed knowledge, confidence, satisfaction, preferences, and choices separately. This separation is methodologically useful because an interface that increases satisfaction may leave knowledge unchanged. An intervention that increases confidence without improving interpretation may worsen the reliability of self-assessment.
Let C_i equal one when item i is answered correctly and zero otherwise. For a fixed set of m items, an unweighted score is the sum of C_i divided by m. A weighted score is the sum of w_i C_i divided by the sum of w_i, with nonnegative weights fixed before analysis. Weighting makes the economic relevance of errors explicit, but weights require an independent rationale. An insurer cannot legitimately declare a test passed by assigning trivial weights to commercially inconvenient exclusions.
For oversight, retain item-level results and prespecified critical errors even if a composite is convenient. Distinguish a false belief that an excluded loss is covered from a false belief that a covered loss is excluded. Their consequences differ. The first may cause an uninsured financial shock; the second may prevent a valid claim or induce unnecessary duplication. A mean score can conceal both.
A practical scoring protocol uses a locked answer key derived independently from approved product wording, tests ambiguous cases before fielding, and has open responses coded by reviewers unaware of treatment allocation where possible. Report disagreement and how it was resolved. Missing responses and requests for clarification should be recorded separately, not automatically interpreted as incorrect understanding without an explicit rule.
Validity and measurement invariance
Validity is the support for a specified interpretation of the score, not a permanent property attached to a questionnaire. Content evidence requires coverage of the decisions and contractual risks that matter. Response-process evidence requires that the respondent's route to an answer reflects the intended comprehension task rather than a clue, a memory trick, or a guess. Criterion-related evidence can be examined by relating scores to a subsequent decision or validated task, while acknowledging that affordability, preferences, and access intervene.
If a scale is designed to measure a latent ability, investigators may examine dimensionality and reliability using appropriate factor or item-response methods. Dichotomous items can exhibit ceiling and floor effects. Reliability estimated from a set of near-duplicate premium questions is poor justification for inferences about understanding across a policy. Where the design deliberately samples independent competencies, internal consistency is not the principal validity criterion; content coverage and reproducible scoring take precedence.
A translated questionnaire should be assessed for differential item performance across language groups and for whether the same construct is measured across channels or populations. In latent-variable analyses, investigators can examine configural, loading, and intercept or threshold equivalence before interpreting group mean differences. When invariance fails, item-specific findings or appropriately anchored partial-invariance models may be more defensible than a single pooled latent score. These are established psychometric principles applied to insurance, not empirical findings of this monograph.
For raw operational pass rates, fairness does not mean requiring all questions to display identical difficulty across groups. Real differences in comprehension may be the very harm of interest. The researcher must distinguish an inaccessible or culturally inappropriate measurement artifact from a genuine difference in the ability to use the disclosed insurance terms. Independent cognitive interviews after test administration help diagnose the distinction.
Durable understanding and calibration
Immediate open-document testing measures usability at the decision point. A delayed test of an analogous problem evaluates whether the consumer can reconstruct the relevant logic after time has passed. A delayed test with access to the policy captures another legitimate capacity: retrieving accurate information when needed. The test must specify which ability is intended; a claim about memory cannot be inferred from open-document use, and a claim about actionable retrieval cannot be inferred from recall alone.
Ask respondents to estimate their confidence after answering, before giving feedback. When probabilities of correctness are elicited, calibration can be summarized by grouping confidence levels and comparing observed correctness; a mean squared difference between the stated probability and the binary outcome is another interpretable measure. A high-confidence false belief about a major exclusion merits particular review. Confidence reporting may itself differ across cultures, so it is diagnostic rather than a substitute for objectively specified tasks.
Experimental Disclosure Formats and Choice Architecture
Standardization and the evidence from format development
Standardized documents reduce the need to relearn the location and vocabulary of information when comparing products. Common heading sequences and consistent units can reveal economically material differences. The European effort to develop the IPID provides a particularly relevant documented example.
The IPID consumer testing report prepared by LE Europe, Ipsos MORI, and Academy Design Partners (2016) describes qualitative focus groups and an online quantitative study in Germany, Romania, Spain, and the United Kingdom. A format that performed best on the objective tasks differed from the format favored by subjective respondents. The policy-development recommendation took objective tasks as a substantive basis and incorporated further design refinements. These results establish feasibility of iterative comparative design; they do not isolate the causal contribution of any single icon or color.
The experimental implication is significant. Testing a bundle of layout changes can select a useful production candidate but does not identify which design component accounts for an effect. A factorial design that crosses concise definitions with, for example, a comparison table can estimate the contribution of each component and their interaction, provided the sample and analysis support that specification. The chosen design should reflect whether the research decision requires a whole-format winner or a generalizable component effect.
Standardization has limits. A template that works for simple risks may obscure an unusual trigger or aggregate limit. A comparison matrix that displays identical headings for distinct risks can imply equivalence where policy definitions diverge. The design should preserve a route to the controlling wording, and exceptional product terms should be salient rather than relegated to an indistinguishable footnote. Standardization is a method of organizing difference, not a reason to erase it.
Plain language, numerical displays, and layered presentation
Plain language aims to make a proposition understandable while preserving its meaning. For instance, a deductible can be explained as an amount of covered expenditure allocated to the insured, subject to the specific method of application. However, the phrase must distinguish whether the deductible applies annually, per loss, per person, or per coverage. Precision is as important as sentence length. Definitions should be situated near the decisions where they are used.
The health insurance misunderstanding study by Loewenstein et al. (2013) examines privately insured Americans and compares a conventional cost-sharing design with a simpler copayment structure. It reports stronger evidence for improved understanding than for improved appeal or changed medical choices. A central lesson is that simpler product architecture can reduce interpretation demands directly. Explaining a complicated design and simplifying its economics are different interventions and need separate evaluation.
Politi et al. (2016) report no statistically significant differences across three active communication formats in their head-to-head format trial. Participants with adequate literacy performed better on knowledge overall. Lack of a significant difference does not demonstrate equivalence; an equivalence claim requires a prespecified margin and adequate precision. Nor does the study establish that all three active formats are superior to no information. The appropriate practical lesson is that the preferred format is an empirical question for the population and task.
Display numerical costs in context: monthly and annual amounts, per-claim and annual limits, fixed amounts and percentages, and currency and time horizon must be explicit. When a probability is supplied by a model, disclose the population, period, and uncertainty appropriate to its use. A natural frequency or diagram can be tested against a percentage representation provided that both express the same event and reference class. An insurer should never substitute a frequency for an indemnity expectation or present a claims ratio as the personal probability of obtaining a payout.
Layering places decision-critical facts in an accessible summary while providing complete detail through clearly identified further layers. Important restrictions must remain visible at the relevant choice point. In digital journeys, an expandable explanation should be discoverable without an arbitrary burden, readable on small screens, accessible by keyboard and assistive technology, and available in durable form as required. A collapsed section containing the central exclusion is a poor design even if its text is excellent.
Choice architecture and commercial context
Insurance is often sold alongside another transaction, in which attention is directed to the primary product. The disclosure may therefore compete with checkout speed, default settings, trust in a seller, and time pressure. Testing a document in isolation may miss these effects.
The Australian add-on insurance trial by BETA and ASIC (2021) used a hypothetical shopping environment with 6,243 final participants. The information-statement variants reduced hypothetical purchase by 9 percentage points in the pooled comparison, corresponding to a reported relative reduction of 24%. The result pertains to the intervention and simulated task; it cannot be translated directly into reductions in actual policies purchased, claims, or welfare. A lower purchase rate alone does not establish that every withheld purchase would have been unsuitable.
A complementary study is the FCA's (2021) research on insurance promotions, which investigated price comprehension and choices in simulated purchasing. Its documented research design illustrates why total cost and promotional framing belong in the test environment. Without reviewing the detailed result tables, this monograph makes no claim about the magnitude of particular promotional effects.
Choice architecture techniques can facilitate comprehension or exploit systematic errors. A balanced comparison offers equivalent salience to cost, limitations, and relevant benefits. Default allocation and optional add-ons should be distinguishable. Interface experimentation should evaluate both acceptance and informed refusal. Production incentives should be separated from the definition of success used in consumer-understanding testing.
Evidence from home contents insurance
Malbon and Oppewal's (2018) home contents disclosure experiment recruited 406 adults from an Australian online panel for choices among hypothetical contracts. The investigators varied disclosure availability and the choice set. They found no simple consistently favorable disclosure effect, and a substantial share of participants selected an inferior offer in some conditions. The report's maximum of 42% choosing the bad option describes particular experimental conditions, not the marketwide prevalence of bad decisions.
The study is important because it constructs a tractable environment in which policy quality differences can be assessed without a premium difference. Even so, the conditions are not a complete model of an actual insurance market. Consumer preferences, brands, richer price differences, contractual variability, channel access, and long-term outcomes can alter choices in reality. The appropriate inference is that even favorable information conditions cannot guarantee economically well-informed selections. The evidence does not establish that disclosure serves no other useful function.
Taken together, the studies support a testable design agenda: identify the relevant mechanism, compare the communication with an appropriate control, measure objective outcomes, and test whether the intervention travels to real purchasing and use. The empirical results caution against assigning a universal rank to narrative, visual, or textual approaches.
Policy Exclusions, Claims, Renewals and Price Testing
Coverage, exclusions, and contingent payments
Coverage testing must address both the presence of protection and the conditions limiting it. For a hypothetical contents policy, assume a covered event produces an eligible loss of $5,000, a per-event deductible of $500, and a payment limit of $4,000 after the deductible. Under those stipulated terms the payment is min(4,000, max(0, 5,000 - 500)) = $4,000, leaving $1,000 of the loss with the insured before any premium or other recoveries. If the payment limit applies in a different order, the result may change. An excluded event would produce no payment under the stipulated hypothetical policy.
This arithmetic example should appear alongside a test involving lower losses, a loss near the limit, and an excluded peril. Some respondents can reproduce the example without generalizing its logic. Analogous new cases and open explanations reduce that risk. Test the ability to distinguish a policy limit, a deductible, and an exclusion. A scenario with insufficient information can assess whether the respondent knows where to check rather than guesses.
The task can be decomposed into identifying the peril, checking territorial and temporal scope, finding applicable terms, calculating the payment, and identifying the action required. A contract specialist must validate that the example reflects the intended provision. If the firm cannot construct a determinate answer because the wording is internally ambiguous, the experiment has already revealed a product-document defect.
For parametric coverage the corresponding task concerns the index, observation method, threshold, data source, and payout schedule. A consumer can suffer actual loss without the stipulated index trigger occurring. Testing should ask about a loss with no trigger and a trigger with little or no actual loss, while avoiding claims about contracts that do not use such a design. Life and disability benefits likewise require product-specific duration, qualifying-event, and beneficiary tasks.
Renewal, premium and affordability
Price tests need to distinguish the payment frequency, total amount payable, compulsory fees, optional components, and material changes introduced upon renewal. If a fictional annual premium rises from $500 to $575, the absolute change is $75 and the proportional increase is 15%. This is a price comparison, not proof of value deterioration: the policy terms, insured exposure, inflation, market alternatives, and service may also change.
The FCA's (2015) renewal field trials investigated the effect of including the previous premium. The official research summary reports 11-18% more home insurance consumers switching or negotiating, with no comparable effect among the two participating motor insurers. The endpoint is action at renewal; it is not a direct test of durable understanding or resulting welfare. The difference across settings indicates that both baseline market conditions and intervention timing matter.
A useful renewal test provides a versioned prior schedule, renewal notice, and change summary. Tasks require identifying premium changes, changes in limits or exclusions, the renewal date, cancellation steps, and whether there is a risk of a gap in coverage. Evaluate comprehension before a decision deadline and measure outcomes among customers who retain, switch, negotiate, or intentionally cancel.
Affordability and understanding are separable. A consumer may understand a higher deductible but accept it because the alternative premium exceeds available funds. In affordability-sensitive products, decision quality must therefore be evaluated against feasible budgets and realistic consequences. Treating all higher-risk choices as errors can erase genuine constraints and induce unsuitable recommendations.
Claims and servicing
Claims communications occur when consumers may face disrupted routines, losses, uncertainty, and reduced attention. The content task shifts from choosing among contracts to exercising rights, supplying necessary information, and understanding the insurer's response. The communication should identify the next step, source of help, documentation needed, relevant deadlines, and the distinction between preliminary and final decisions.
Testing a denial letter requires scenarios in which the participant can identify the relevant reason, understand what evidence would be relevant, and locate available review or complaint routes. A test should not presuppose that denial is legally correct; that determination requires a separate claim review. A participant's understanding of an inaccurate rationale cannot validate the rationale.
For post-sale changes, the customer should be able to determine what changed and when, how to confirm current coverage, and how to correct or report an error. The institution should preserve communication versions so that outcomes can be linked to what the person actually received. Call-center testing can use role-play and standardized problem cases, but a trained actor's performance does not substitute for observation of real consumers with different access needs.
Decision-task blueprint
| Lifecycle stage | Objective task | Critical error | Follow-up measure |
|---|---|---|---|
| Acquisition | Compare covered loss and retained cost under alternatives | Assuming a material excluded event is covered | Preference-consistent selection and mismatch complaints |
| Renewal | Identify change, deadline and continuity options | Unintended lapse or mistaken belief that limits are unchanged | Documented switching, retention and lapse patterns |
| Servicing | Locate correct terms and update required information | Using an obsolete version or sending information to the wrong channel | Repeated contacts and resolution time |
| Claim | Explain relevant requirement and locate assistance or review | Missing an available claim or review route | Timely notices, abandonment and justified complaints |
These are examples of operational tests and monitoring indicators. They do not imply that a change in a follow-up measure was caused by comprehension without an appropriate comparison. A high claim frequency or complaint rate may reflect coverage mix, exposure, product quality, or improved awareness rather than miscommunication.
Sample Design, Comprehension Validity and Causal Identification
Research populations and ethical recruitment
Specify the target population before choosing a recruitment mechanism. The population may consist of first-time buyers, existing policyholders approaching renewal, individuals eligible for a claim, or users of a particular distribution channel. An online panel is practical for controlled comparison, but it cannot on its own establish accessibility among consumers unable to use that channel. Sampling frames, response and completion rates, exclusion rules, and recruitment incentives should be stated.
For generalizable estimates, recruit from the intended population or justify the transport of estimates from another population. Probability sampling enables design-based inference where the frame and response process support it. Nonprobability samples require explicit assumptions about selection and weighting; demographic balance alone does not prove representativeness. Oversampling groups at higher risk of communication-related harm can improve the precision of subgroup estimates. Weight overall results back to the target distribution where justified, and report unweighted subgroup counts.
Consumers with sensory disabilities, limited language proficiency, limited digital access, temporary distress, uncertain housing, financial hardship or other access needs may encounter different barriers. Researchers should recruit through accessible channels, offer reasonable accommodations, and avoid unnecessary collection of sensitive information. The aim is to identify communication barriers and remedies, never to screen consumers out of protection or infer individual risk from research participation. A consent process, an appropriate ethics review where required, data minimization, and a clear separation from underwriting or claims eligibility are essential.
Causal estimands and randomized designs
Define the intervention precisely: wording, layout, timing, delivery medium, interaction behavior, and document version. The comparator should be the existing compliant communication, another credible design, or a justified simulation control. Withholding legally mandated information in real transactions is not an acceptable experimental convenience. Where the relevant standard requires a full set of disclosures, all live arms should satisfy it.
Let Z equal one for random assignment to a candidate disclosure and zero for assignment to the comparator. Let Y(1) and Y(0) represent the potential scores under each assignment. For the study population, the average causal effect of assignment is E[Y(1) - Y(0)]. With appropriate randomization, consistent measurement, and no interference, the difference in observed arm means estimates this quantity. Report the intention-to-treat contrast even when some recipients do not open, read, or finish the material. Excluding nonreaders after assignment changes the question and can introduce bias.
If the allocation unit is a customer-service team or branch, account for clustering in both sample design and uncertainty estimates. Repeated observations from the same policyholder likewise require dependence-aware analysis. For a crossover format comparison, consider order, learning, and carryover; repeated tasks should use genuinely parallel cases. Randomization should be implemented and verified before outcome analysis.
In field testing, exposure may differ from assignment because an email is undelivered or an interface branches. Record these discrepancies. The effect of receiving or using the disclosure is a different estimand and typically requires additional assumptions. A comparison of readers with nonreaders cannot by itself establish that reading caused the difference.
Precision, sample size, and subgroup effects
A test should be planned to estimate a meaningful difference with useful precision. Consider a hypothetical binary critical-item outcome with a control correctness rate of 60% and a target rate of 70%. For two equally allocated independent arms, a normal approximation for two-sided 5% significance and 80% power gives approximately 356 to 357 participants per arm before attrition, clustering or multiplicity adjustments. This follows from the usual two-proportion planning formula using the average probability 0.65 and normal quantiles 1.96 and 0.84. It is an illustrative design calculation, not a universal minimum or empirical requirement.
For cluster-randomized designs with nearly equal cluster size m and intraclass correlation rho, the approximate design effect is 1 + (m - 1)rho. If m is 20 and rho is 0.05, the factor is 1.95. Unequal sizes and small numbers of clusters may require more careful methods. When testing high-risk groups, powering only the pooled outcome can lead to unusably wide subgroup confidence intervals.
Report absolute changes with confidence intervals and denominators. If treatment correctness is 70% versus 60%, the difference is 10 percentage points; the relative improvement is about 16.7%. A relative reduction in error from 40% to 30% is 25%. The three descriptions refer to the same hypothetical observations but communicate different quantities. Report the absolute effect as the primary scale when it is meaningful for risk decisions.
An interaction between assignment and a prespecified group estimates whether effects differ across that group definition. Separate significance in one group and nonsignificance in another does not establish a difference. Multiple comparisons, data-dependent subgroup selection, and flexible scoring increase false discoveries. Predefine primary endpoints and subgroup analyses, use appropriate multiplicity control or clearly designate exploratory results, and retain sufficiently informative confidence intervals.
Attrition threatens interpretation when participation or completion depends on the treatment and unobserved outcomes. Report missingness by arm, reasons where available, and sensitivity analyses under plausible alternatives. Noncompletion may itself be a meaningful accessibility outcome. Treating every incomplete test as a zero without examining the context can confound usability, exposure, and comprehension.
Observational and quasi-experimental alternatives
Not all communication changes permit individual randomization. A phased introduction across channels or regions can sometimes support difference-in-differences analysis. The required identifying condition concerns parallel counterfactual trends, alongside stable outcome measurement and attention to concurrent changes. A pre-post improvement alone is compatible with seasonality, changing product mix, staff training, price movements, or changes in the claims environment.
A regression discontinuity design can examine a policy that changes communication at a well-defined eligibility threshold if potential outcomes vary smoothly around the threshold and manipulation is limited. Its effect is local to cases near that threshold. Interrupted time series analysis can test for level or trend changes when ample pre- and post-intervention observations exist, but requires scrutiny of concurrent events, serial dependence, and outcome definition changes. These methods answer different causal questions rather than replacing randomization automatically.
Administrative monitoring can identify emerging harm at scale, especially with repeated channels and documents. It is valuable for surveillance and hypothesis formation. Causal claims should specify the comparison, assumptions, alternative explanations, and uncertainty rather than rely on dashboards alone.
Comparative Product and Consumer Protection Regulation
International supervisory context
The IAIS fair-treatment paper announcement (2025) describes recommendations supporting Insurance Core Principle 19 across design, distribution, communication, assistance, claims, complaints, monitoring, and review. The document's institutional role is international supervisory guidance. Local legislation, regulations, and supervisory frameworks determine enforceable requirements and remedies. The cited announcement confirms lifecycle scope; it is not relied upon for detailed procedural prescriptions.
This matters for research design because international comparisons may show a shared concern with consumer outcomes even when legal duties and enforcement structures differ. A global insurer conducting a standardized test should identify which requirements are common scientific criteria and which are jurisdiction-specific mandatory elements. The comparator must be locally lawful, the wording must match local insurance contracts, and the observed population must represent the actual target market.
European Union
Article 20 of the Insurance Distribution Directive (European Parliament and Council, 2016, consolidated January 2024) establishes comprehensible relevant information before contract conclusion and sets requirements for a standardized non-life IPID, subject to its scope and exemptions. Article 20(9) addresses consumer testing in format development by EIOPA. Article 23 establishes information communication conditions. These provisions distinguish the legal duties for distribution from the testing used to develop a standardized format.
EIOPA's editable IPID template resource explains the relationship between the consumer-tested implementing standards and Commission Implementing Regulation (EU) 2017/1469. It also explains that manufacturers may use their own files that comply with the applicable format requirements. Thus the editable files are an implementation aid, not a requirement to use specific software. The research implication is that statutory format conformity and demonstrated consumer comprehension are evidence of different kinds. Local implementing law and any subsequent amendments require separate checking for application to a specific activity.
United Kingdom
The FCA's PRIN 2A.5 consumer understanding rules (2026b, updated Handbook text) require regulated firms within scope to support retail consumer understanding, communicate in a clear and fair manner, and provide timely information. PRIN 2A.5.10 addresses testing and ongoing monitoring where appropriate, investigation of issues, and correction of deficiencies through communication or, where appropriate, product and process adaptation. PRIN 2A.5.12 is guidance on determining the appropriateness of testing, including the communication's purpose, risk of harm, vulnerability, and urgency.
Testing is risk-sensitive and context-sensitive; the provisions do not create a universal sample size, comprehension pass percentage, or obligation to run a randomized trial for every communication. The consumer understanding supervisory findings (FCA, 2026a) illustrate good and poor practices and explain how firms can use multiple signals and governance processes. The report itself distinguishes examples from new rules.
In insurance research, the UK approach is particularly useful for linking evidence of poor understanding to remediation beyond rewriting documents. A communication may accurately describe a product whose complexity, distribution incentives, or service process causes harm. Where the product or process is responsible, improving language alone is insufficient.
United States and Australia
The United States has product-specific and layered regulatory arrangements. CMS's Summary of Benefits and Coverage resource explains a standardized communication instrument for applicable health coverage, including comparable benefits, cost sharing, limitations, a glossary, and standardized coverage examples. The examples illustrate benefit application; they do not predict a particular individual's exact medical costs. The instrument is a useful model for standardized scenario construction. It should not be described as a universal disclosure requirement for every US insurance line.
In Australia, the cited contents insurance study and add-on insurance research demonstrate regulatory and consumer-protection inquiry in two different product contexts. These are experimental and policy research records, not substitutes for underlying legislation. This study therefore uses Australia for empirical comparative evidence and does not assert the current detailed legal scope of the Key Facts Sheet or add-on insurance sales duties.
The comparative conclusion is that communication evidence serves different functions under different legal structures. The European product-summary format, the UK outcome-focused framework, and US health-plan comparison requirements all support investigation of the practical use of insurance information, but cannot be treated as interchangeable rules. Scientific protocols can be shared while compliance determinations remain tied to their legal sources and product scope.
Evidence Thresholds for Product Governance and Outcome Improvements
From findings to accountable decisions
An evidence package for a disclosure change should identify a decision owner; the affected product and audience; versioned old and new communications; a contractual accuracy review; the protocol and scoring rules; recruitment and accessibility considerations; the analysis and raw counts or reproducible aggregate data; and the decision with its limitations. The evidence must be retained in a form that permits an independent reviewer to reproduce the interpretation without relying on an unrecorded presentation.
The acceptance decision should address both absolute performance and comparison with the baseline. A design that improves the average yet leaves a high frequency of harmful misunderstanding requires further investigation. A design that fails to demonstrate a superiority effect may nevertheless offer improvements in accessibility, cost, or clarity if minimum outcome requirements and noninferiority evidence are satisfied. Such conclusions require advance criteria and uncertainty bounds; they cannot be selected after inspecting favorable results.
For high-consequence items, define an unacceptable error rate or a practically important improvement through a documented harm assessment. No uniform scientific cutoff follows from the reviewed literature or the cited regulatory material. Evaluation should incorporate the likelihood of the event, potential magnitude of loss, ability to recover, availability of alternative protection, and barriers to obtaining assistance. Avoid thresholds that excuse harm merely because affected consumers are a small minority.
Suppose an experiment records zero errors in 30 observations on a critical item. Under independent Bernoulli sampling, the exact one-sided 95% upper bound solves (1-p)^30 = 0.05, giving approximately 9.5%. Zero observed errors in this small sample therefore does not establish a near-zero underlying error rate. This ordinary statistical argument shows why evidence of safety and competence demands attention to precision, rather than just point estimates.
Vulnerability, accessibility and distributional effects
Use accessible designs across the research process, including recruitment, consent, stimuli, and measurement. For online interfaces, WCAG 2.2 (W3C, 2024) supplies a technical accessibility standard. Conformance supports usability for many users but cannot establish accurate understanding of a policy provision. Combine technical evaluation with task-based testing among consumers using relevant assistive technologies and alternative channels.
Evaluate the differences in critical error rates and task completion across relevant population groups, subject to adequate sample size and privacy constraints. A positive average effect may mask deterioration for some groups, or result from differential attrition. The response to a discovered disparity should begin with an analysis of the barrier: language, inaccessible layout, numeracy demands, service availability, or contractual complexity. Where the barrier is in the product or distribution process, a revised communication may not be the appropriate sole remedy.
Segmentation should be proportionate and purposeful. Sensitive characteristics gathered for research should not become sales-targeting or exclusion tools. Reporting small cells may reveal individuals; use disclosure controls and carefully balanced granularity. The objective is to improve inclusive protection, not assign permanent comprehension labels to people whose circumstances can change.
Governance and outcome monitoring
The evidence process begins before launch and continues after implementation. In production, monitor versioned exposures, channels, requests for assistance, repeated contacts, cancellation and renewal outcomes, claims questions, and complaint themes. The FCA's (2026a) supervisory examples emphasize multiple sources of consumer insight and a connected governance approach. Surveillance measures have diagnostic value; their movements do not by themselves establish causal impact.
A complaint rate may increase because a new letter reveals an available complaint route. A reduction in contacts may indicate clearer information, but can also indicate that assistance has become difficult to obtain. Outcome interpretation must therefore be triangulated with qualitative case assessment, objective comprehension measures, and service access evidence. In particular, absence of complaints may conceal barriers to complaint submission.
Configurations of customer-facing communications change when product wording, prices, legal duties, payment conditions, user interfaces, or customer populations change. Version control should relate the disclosure to the contract and the specific experiment. Retesting should be proportionate to changed risk, rather than mechanically repeating low-risk work or indefinitely relying on an older result after a material redesign.
Where systematic confusion is detected, investigate affected versions, customers, and decisions. Correct the information and determine whether the underlying contract, sales process, technology, or service is responsible. Where harm may already have occurred, consumer remediation requires assessment under applicable law and organizational duties. Retesting after a change verifies whether the identified problem was resolved; a change log alone does not.
A reproducible assessment example
Consider a hypothetical insurer redesigning the presentation of a deductible and a water-damage restriction. The objective is to decrease false beliefs about coverage while preserving correct interpretation of the deductible. The control is the existing approved communication. The candidate uses a plain-language summary with a worked covered-loss scenario and a prominently placed exclusion. Complete terms remain available in both arms.
Before recruitment, the investigators validate the scenario against the contract and specify primary endpoints: correctness on a new exclusion-transfer question and correctness on a new deductible calculation. They define adverse secondary outcomes such as a false belief that ordinary covered water damage is excluded, failure to locate the claims contact, and abandonment. A field trial follows only after accessible prototype testing, contractual review and applicable compliance clearance.
Random assignment is stratified by prior insurance experience and channel. The primary analysis estimates assignment effects on the prespecified outcomes and reports confidence intervals. The test separately records confidence and the ability to retrieve the full policy. An optional later task measures durable retrieval. Results are reported with missingness, the exact versions tested, and any departures from protocol.
The governance decision compares both observed error rates and the estimated precision with prespecified harm-sensitive criteria. It checks whether improvement for the overall sample is accompanied by material deterioration for relevant groups. A failure on the exclusion-transfer task calls for further investigation of wording and contract structure; a failure caused by the mobile interface calls for an interface correction and verification with users. The approved design receives a controlled production identifier with a monitoring plan. The example specifies a research procedure, not observed results or a proprietary methodology.
Limits of the evidence and implications for research
The empirical corpus examined here spans simulated and actual decisions. The EIOPA study addresses document presentation and objective versus subjective tasks. The Australian add-on trial measures simulated purchase. The home contents experiment uses deliberately constructed comparative offers. The US studies address plan understanding and comparison behavior; the UK field evidence addresses renewal actions. Each supports a limited inference about a particular context. Directly aggregating these effects would obscure the differences in endpoints.
Evidence suggests that research should move beyond immediate knowledge tests toward pre-registered field designs with longer follow-up, explicit critical-harm outcomes, and inclusion of populations absent from digital panels. Important open research questions concern whether improved comprehension transfers across product versions, whether a short-term gain persists until claim time, and how accessible comparison design interacts with affordability and the actual availability of coverage. Cross-country replication requires alignment of the substantive decision task rather than literal translation alone.
A further question concerns the consequences of fundamental product complexity. If a well-designed disclosure repeatedly fails to support critical understanding, this is a reason to examine the contract's economic structure. Simplifying the product may change risk pooling, pricing, adverse selection, or the breadth of protection, so changes require actuarial and governance review. The scientific finding of poor comprehension is an input to that decision, not a complete product optimization result.
Conclusion
Disclosure design is scientifically evaluated through evidence of what consumers can understand and do in relation to the actual terms, decisions and risks of insurance. The relevant unit is the interaction between a defined communication, a defined audience, and a defined task at a specified stage of the insurance relationship.
The studies reviewed establish that accessible presentation and the delivery of comparison information can improve particular performances under particular conditions. They also show that preference for a format, availability of a document, and a change in purchasing behavior are insufficient independent guarantees of understanding or welfare. Objective scenario-based assessment, explicit outcome definitions, appropriate controls, and uncertainty-aware analysis allow the investigator to distinguish those propositions.
The scientific contribution of this monograph is the integration of the contractual, economic, experimental, and governance perspectives into a coherent account of disclosure evaluation. The central inference is that information provision is most useful when the communication accurately represents the contingent financial transaction and is tested against the consumer's actual decision needs. A validated result has a defined scope; it belongs to the tested population, communication version, decision context, and outcome until further evidence justifies transport.
An insurance institution supports consumers through well-formed contracts, usable comparisons, accessible channels, meaningful assistance, and fair administration of contingent rights. Consumer-understanding evidence should inform decisions throughout those processes. Where deficiencies emerge, the appropriate correction follows the source of the deficiency and may extend to the product itself. Communication is an instrument of protection whose performance can and should be subjected to empirical scrutiny.
The conclusions are based on selected published and institutional sources; there is no new empirical dataset or claim of a universally optimal format. Independent causal estimates, validated thresholds, and jurisdiction-specific implementation decisions require evidence suited to their own questions.
References
Behavioural Economics Team of the Australian Government (BETA) and Australian Securities and Investments Commission (ASIC). (2021). Supporting consumer decision-making about add-on insurance: Slowing down to add it up. Department of the Prime Minister and Cabinet. Project report and trial summary. Source record
Bhargava, S., Loewenstein, G., & Sydnor, J. (2017). Choose to lose: Health plan choices from a menu with dominated options. Quarterly Journal of Economics, 132(3), 1319-1372. DOI: 10.1093/qje/qjx011. Source record
Centers for Medicare & Medicaid Services (CMS). (n.d.). Summary of Benefits & Coverage & Uniform Glossary. Center for Consumer Information and Insurance Oversight. Source record
European Parliament and Council of the European Union. (2016). Directive (EU) 2016/97 of 20 January 2016 on insurance distribution (recast). Official Journal of the European Union, L 26, 19-59. Consolidated text of January 9, 2024, arts. 20, 22, 23. Source record
European Insurance and Occupational Pensions Authority (EIOPA). (n.d.). Editable template for the Insurance Product Information Document (IPID). Source record
Financial Conduct Authority (FCA). (2015). Encouraging consumers to act at renewal: Evidence from field trials in the home and motor insurance markets. Occasional Paper No. 12. Official research summary. Source record
Financial Conduct Authority (FCA). (2021). Discounts, cashbacks, and soft toys: The impact of promotions on consumer decisions in the general insurance markets. Research Note. Official research landing page. Source record
Financial Conduct Authority (FCA). (2026a). Consumer understanding: Good practice and areas for improvement. Supervisory findings. Source record
Financial Conduct Authority (FCA). (2026b). FCA Handbook: PRIN 2A.5, Consumer Duty: Retail customer outcome on consumer understanding. Rules and guidance effective from July 31, 2023; retrieved Handbook version updated June 26, 2026. Source record
International Association of Insurance Supervisors (IAIS). (2025). IAIS publishes Application Paper on fair treatment of a wide range of consumers. July 7, 2025. Official release summary. Source record
Kling, J. R., Mullainathan, S., Shafir, E., Vermeulen, L. C., & Wrobel, M. V. (2012). Comparison friction: Experimental evidence from Medicare drug plans. Quarterly Journal of Economics, 127(1), 199-235. DOI: 10.1093/qje/qjr055. Source record
LE Europe, Ipsos MORI, & Academy Design Partners. (2016). IPID consumer testing and design work. Final commissioned research report for EIOPA, August 2016. Source record
Loewenstein, G., Friedman, J. Y., McGill, B., Ahmad, S., Linck, S., Sinkula, S., Beshears, J., Choi, J. J., Kolstad, J., Laibson, D., Madrian, B. C., List, J. A., & Volpp, K. G. (2013). Consumers' misunderstanding of health insurance. Journal of Health Economics, 32(5), 850-862. DOI: 10.1016/j.jhealeco.2013.04.004. Source record
Malbon, J., & Oppewal, H. (2018). (In)effective disclosure: An experimental study of consumers purchasing home contents insurance. Monash Business School and Monash Faculty of Law; commissioned by Financial Rights Legal Centre. Source record
Politi, M. C., Kaphingst, K. A., Liu, J., Perkins, H., Furtado, K., Kreuter, M. W., Shacham, E., & McBride, T. (2016). A randomized trial examining three strategies for supporting health insurance decisions among the uninsured. Medical Decision Making, 36(1), 52-62. DOI: 10.1177/0272989X15578635. Source record
World Wide Web Consortium (W3C). (2024). Web Content Accessibility Guidelines (WCAG) 2.2. W3C Recommendation, December 12, 2024. Source record