Insurance by BogdanovaThe Encyclopedia of Insurance and Risk Science

RESEARCH DOSSIER

Insurance Sales Supervisory Mystery Shopping and Outcome Evaluation

Abstract. Supervisory mystery shopping investigates the real conditions under which prospective policyholders encounter insurance products, advice, disclosures, pricing, and transaction interfaces. Its scientific value depends on the correspondence between the question posed, the sampling mechanism, the customer scenario, the observed event, and the permissible inference. This monograph develops a comparative and methodological treatment of undercover and simulated-customer exercises in insurance distribution. It examines population definition, stratified sampling, profile construction, field and digital protocols, legal and outcome coding, evidentiary safeguards, measurement reliability, and the evaluation of supervisory remediation. European insurance-based investment product exercises, German test purchases, Hong Kong bank distribution surveillance, British consumer-outcome monitoring, and experimental evidence from Indian life insurance provide complementary institutional and empirical perspectives. The analysis distinguishes observations of a sales interaction from product-profile alignment, realized contractual outcomes, and causal estimates of market harm. It demonstrates the effects of clustering, post-contact exclusions, rating disagreement, and inadequate comparison groups using explicitly hypothetical calculations. The principal conclusion is that mystery shopping is most informative when it is embedded in a transparent, risk-based supervisory program that combines reproducible observations with contract evidence, administrative outcomes, and proportionate follow-up. The monograph presents an analytical synthesis of established methods and publicly available evidence; it reports no new field experiment, population estimate, or named theoretical system.

Keywords: insurance distribution; market conduct; mystery shopping; insurance-based investment products; suitability; consumer outcomes; supervisory sampling; audit experiments; disclosure; fair value; inter-rater reliability; supervisory evaluation

Introduction

An insurance sale is a sequence of decisions about protection, contractual rights, expenses, and future uncertainty. A purchaser must translate a present need into a policy whose value may become observable only after an insured event or a prolonged investment horizon. The seller typically possesses information about exclusions, underwriting, remuneration, and administrative procedures that is costly for the purchaser to acquire independently. Accordingly, the sales encounter is a substantive object of insurance science. It is the point at which product design, economic incentives, regulatory duties, and customer understanding interact.

Supervisory mystery shopping is the planned observation of that encounter by trained participants who act as prospective customers under predefined conditions. The object of measurement is conduct observed along an actual or realistically simulated purchasing journey. An exercise may stop after an inquiry, request a quotation, continue to a recommendation or complete an authorized purchase. Each endpoint defines a different evidentiary opportunity. Insurers' own service-quality programs may use similar methods, but the supervisory objective concerns evidence of conduct risks and regulated outcomes rather than commercial satisfaction alone.

The international perspective is necessarily jurisdiction-sensitive. A European insurance-based investment product (IBIP), a Hong Kong non-linked long-term insurance plan, and a British retail general insurance add-on activate distinct legal and product constraints. Comparisons concern mechanisms and methods unless equivalence of obligations, eligible populations, and measurements can be established. The study uses a purposive documentary review of regulatory text, official reports and announcements, and selected scholarly records available during the research session on November 19, 2026. Its legal discussion is anchored to identified editions; subsequent or national changes must be checked in an operational enforcement case. The sources include different levels of access, documented in the evidence ledger. The numerical illustrations are mathematical examples, not empirical estimates.

Three questions organize the inquiry. Which observations can support estimates about a distribution market? How should a supervisor separate legal duties, quality of advice, and indications of future consumer detriment? Which decisions and interventions can be justified by evidence collected without an actual claim or a completed contract? The answers depend on aligning the design before the first shopping contact with the eventual uses of evidence.

Mystery Shopping as Supervisory Science

Observation and its unit of analysis

EIOPA's description of supervisory mystery shopping identifies trained participants, multiple communication channels, and complementarity with other supervisory methods. Four units commonly coexist: the establishment or platform, the individual adviser or interface, the shopping journey, and the insurance product offered. They answer different questions. A single adviser may present three products within one visit. Treating those offers as three independent encounters artificially increases the amount of information. Conversely, treating three different advisers as one observation hides variation in local practice.

The appropriate unit follows the estimand. A branch-weighted failure rate describes branches; a transaction-weighted rate describes consumer encounters; a product-weighted rate describes offers. The investigator should identify whether the population includes unsuccessful inquiries, online abandonment, refusals, and unavailable channels. A report that names the denominator simply as customers mischaracterizes a sample of trained shoppers and can obscure all of these distinctions.

A complaint is retrospective evidence produced after a customer detects a problem and decides to report it. A file review observes records created by a seller. An onsite inspection evaluates arrangements and controls. A mystery-shopping visit observes what happens at a specified moment in the distribution process. Each is vulnerable to a distinct selection mechanism. Complaints overrepresent customers who can recognize and articulate harm; records may omit conversational context; announced inspections may elicit atypically careful conduct; shoppers may be recognized or behave differently from ordinary customers. Independence of these sources is an empirical question because a common management system can shape them all.

Process, intermediate outcomes, and realized outcomes

The assessment must keep three levels separate. Process measures concern actions such as asking about customer circumstances or presenting a disclosure. Intermediate outcomes concern immediate consequences, such as whether a proposed policy matches the stated need or whether a consumer can explain a material exclusion. Realized outcomes concern actual coverage, the cost and length of holding the policy, claim payment, cancellation, or welfare. The first two can often be measured within a shopping journey. The third requires additional evidence and elapsed time.

A compliance item can carry intrinsic legal significance even if an immediate adverse outcome is absent. For instance, withholding a disclosure required before conclusion is relevant to supervision even if the product happens to fit the scenario. The reverse also holds: a poor projected fit can merit intervention even where a detailed questionnaire has been completed. Counting forms measures form completion. Assessing the quality of protection requires a substantive comparison between a customer's needs and the terms offered.

The IAIS conduct and culture program situates intermediaries and conduct within ICP 18 and ICP 19 and explains their relationship to prudential supervision. This connection has economic significance. If customers have systematically misunderstood benefits or purchased inappropriate protection, a firm can face cancellation, remediation costs, legal disputes, and reputational pressure. The mechanism is plausible without implying that a given observed sales defect establishes a quantified solvency impact.

A supervisory evidence cycle

A risk-based program can begin with anomalies in complaints, lapse rates, pricing, or claim denials. Shopping helps specify the interaction that could generate the anomaly. A follow-up examination then tests whether that interaction reflects an organizational control, product limitation, incentive arrangement, individual behavior, or measurement artifact. The IAIS survey of conduct indicators supports the role of data in supervisory assessment. An evidence cycle gives observable interpretation to an indicator: for example, high early lapses can motivate a scenario specifically testing explanations of surrender penalties.

The design should designate an exploratory stream and a confirmatory stream. The exploratory stream targets suspected harm and supports discovery, whereas the confirmatory stream uses independently defined rules and sampling to verify the frequency or persistence of a pattern. Otherwise, a discovery obtained from an intentionally risky sample can be mistaken for a population estimate. The two streams may complement each other, but their denominators, selection rules, and resulting statements remain distinct.

Sample Frames and Customer Profile Design

Defining the population

A sampling frame is an operational representation of the target population. An insurance license registry can enumerate registered firms and intermediaries, but usually cannot enumerate all customer encounters. Product catalogs enumerate available offers but rarely contain complete sales volumes; digital comparison websites may list only a subset of participating firms. The researcher should record the frame date, source, exclusions, duplicate identifiers, and correspondence between licensed entities, trading names, branches, and digital domains.

For a firm-level design, the frame may contain insurers and intermediaries with a relevant authorization and current business activity. For an establishment-level design, it may enumerate distribution points and measure channel reach. For a consumer-exposure design, it should include a defensible measure of inquiry or transaction volumes. Premium volume is sometimes a useful proxy for economic importance but can overemphasize expensive policies. Coverage of 35% of gross written premium has no direct mathematical equivalence to a 35% probability sample of sales encounters.

A balanced design stratifies the frame on product family, distribution model, geography, channel, and size where these variables matter to both risks and inclusion probabilities. Strata should be defined before fieldwork and should have adequate sample support. Rare channels or high-risk products may be oversampled deliberately. With a known inclusion probability, such designs allow transparent weighting. Where selection probabilities are unknown, a targeted sample still has value for discovery, but the observed proportion applies to the sample rather than the market as a whole.

Probability sampling, selection, and precision

Let N_h be the number of eligible units in stratum h, n_h the number randomly sampled, and p_h the observed fraction meeting a criterion. For simple random sampling within each stratum, the unit-weighted estimator is

p = sum_h (N_h / N) p_h,

where N = sum_h N_h. More generally, an inverse-probability ratio estimator is sum_i(w_i y_i) / sum_i(w_i), with w_i = 1 / pi_i and pi_i the inclusion probability. Design-based uncertainty must reflect strata, clusters, unequal weights, and finite-population corrections when relevant. Nonresponse weighting requires assumptions about the relationship between response and outcomes; weights cannot restore an unobserved group for which no comparable data exist.

Consider a hypothetical market in which 80% of eligible encounters are direct sales and 20% are intermediary sales. A design samples 100 encounters from each channel and observes deficient disclosure in 10 direct encounters and 30 intermediary encounters. The unweighted rate is 20%. If the channel exposure proportions are valid and each within-channel sample is representative, the standardized rate is 0.8 × 0.10 + 0.2 × 0.30 = 0.14, or 14%. Both calculations describe coherent quantities; the sampling design determines which quantity answers the question.

At an assumed rate of 0.20 and independent random sampling of 200 observations, the 95% Wilson interval is approximately 15.0% to 26.1%. The score-interval method (Wilson, 1927) gives the center (p + z²/(2n))/(1 + z²/n) and half-width z sqrt(p(1−p)/n + z²/(4n²))/(1 + z²/n). This is an illustration for a simple binomial design; it is not an interval for a weighted, targeted, or clustered supervisory exercise.

Clustering can be substantial where advisers share templates, scripts, or management. If ten encounters per branch are sampled and an illustrative intracluster correlation is 0.10, the equal-cluster design effect is 1 + (10−1) × 0.10 = 1.9. Two hundred observations then have an approximate effective sample size of 105.3 relative to independent measurements of the mean. The approximation assumes equal cluster sizes and a common correlation; complex samples require appropriate variance estimation. Expanding the number of branches may provide more independent information than repeatedly shopping one branch.

For planning a simple random estimate of an unknown proportion within ±5 percentage points at 95% confidence, using the conservative variance p(1−p) = 0.25 gives about 385 independent observations before adjustments for clustering and incomplete visits. A comparison between two profiles requires a power calculation for the prespecified effect, allocation, correlation structure, and multiplicity. Rare but severe failures may justify larger samples or risk-targeted discovery even when rate estimation is not the immediate objective.

Customer profiles and experimental contrasts

A profile specifies the information a shopper will present, facts to reveal when questioned, material preferences, financial constraints, and a stopping rule. It should describe a credible person whose purpose is internally consistent. An IBIP scenario may specify investment horizon, emergency reserves, willingness to bear losses, ability to bear losses, existing protection, and preferences. A general insurance scenario may define a property, risk exposure, desired limits, deductible tolerance, exclusions that would defeat the objective, and ability to pay. The profile should be complete enough for ex ante evaluation without forcing the seller toward a preselected product.

An observational design holds scenarios stable within strata to examine variation across sellers. A randomized audit design varies a predefined characteristic and assigns it independently of sellers or encounter times. For example, the study could compare otherwise equivalent customers who express greater or lesser familiarity with insurance terminology. The outcome is the effect of the assigned information on the observed encounter, not the effect of being a member of a demographic group in all real markets. Customer cues that legitimately affect underwriting, coverage eligibility, or pricing should not be interpreted as unlawful discrimination without a legal and actuarial basis.

Matched shoppers should be balanced across assignment arms. A single researcher delivering all low-literacy scenarios and another all high-literacy scenarios confounds shopper behavior with the intended treatment. Repeated scripts introduce detection and interference risks. Randomizing order, time windows, and assignment, retaining the assignment log, and measuring departures from scripts reduce these risks. Pilot exercises should check realistic responses, travel and scheduling feasibility, and consistency of the elicited interaction.

Attrition and failed contacts

The first recorded event is the attempted contact. Successful booking, first meeting, receipt of information, recommendation, quote, and sale are successive stages. Suppose 400 valid attempted inquiries yield 300 meetings, of which 240 yield recommendations and 180 provide complete cost information. A report might identify 75% complete information among recommendations, 60% among meetings, and 45% among attempts if the endpoint is coded as receipt of complete information by the end of the journey. Each is a different statistic. Some stages may be ineligible for a legal duty that activates later, so legal noncompliance cannot be assigned simply by dividing 180 by 400.

A missing or aborted encounter is documented as its own outcome. Where only a subset reaches the recommendation stage, missing fit ratings are potentially related to the seller's behavior. A sensitivity analysis can bound the prevalence of a failure among all 400 if 60 failures are observed in 240 assessable encounters: the most favorable bound is 15%, and the extreme upper bound is 55% if all 160 unassessed attempts would have failed the same hypothetical binary endpoint. The bounds are deliberately broad and require that the outcome can meaningfully be defined for every attempt. Reporting funnel stages separately may be preferable to imposing an artificial endpoint.

Field, Remote and Digital Shopping Protocols

Preparation, standardization, and fieldwork

The protocol specifies research purpose, legal authorization, product and channel eligibility, sampling design, shopper assignments, materials, decision rules, safety limits, and evidence handling. A training program uses scenario rehearsals and examples of common salesperson responses to reduce variation caused by the shoppers themselves. The instruction to act naturally should coexist with standardized facts. The shopper needs a permitted range of wording and should record unscripted prompts or information changes rather than improvising material facts.

A contact record contains a pseudonymous journey identifier, sampled unit, channel, timestamp and time zone, scenario version, assignment, visit stage, attempts, disclosure materials, and any deviations. Immediately after an encounter the shopper records contemporaneous notes in a structured questionnaire. Separating factual observations from perception helps later review: a salesperson's exact statement is different evidence from the shopper's impression that the statement was urgent or persuasive.

Fieldwork can distinguish unprompted conduct from prompted conduct. For example, a protocol first allows the adviser to explain a policy as usual and later asks a prespecified question about exclusions. An unprompted disclosure and a correct response after prompting are both useful, but the information required and the exposure to consumer harm differ. Prompt order should be stable when measuring spontaneous conduct, and randomized where the research question explicitly concerns responsiveness to particular cues.

A protocol must define boundaries for identity verification, document requests, payment, medical information, and contract execution. Exercises generally stop before material financial commitments unless authorized research requires a purchase. Test buyers should not submit false documents, fabricate legally consequential health declarations, cause fictitious claims, or induce employees to bypass lawful underwriting safeguards. Any genuine purchase requires provisions for eligibility, accurate representations, premium funding, lawful cancellation, treatment of the resulting contract data, and researcher protection.

Telephone and asynchronous communications

Telephone shopping makes it easier to standardize scenario facts but adds potential effects of voice, accent, call routing, hold times, call recording restrictions, and staffing patterns. A research plan should decide whether queue abandonment is a substantive support outcome or a sampling failure. Time windows should reflect the population of interest rather than investigator convenience alone. If the service is unavailable, repeated attempts under a predefined rule prevent arbitrary escalation by individual shoppers.

Email and chat generate durable text evidence but capture only responses within selected time limits. A seller may transfer an inquiry to another channel. The journey identifier must follow that transfer so the investigation describes the actual sequence rather than isolated messages. The observation cutoff is prespecified, and a reply arriving later is documented consistently.

Questions asked in one language may not have equivalent pragmatic meaning in another. Translation should preserve substantive information and the degree of assertiveness. Jurisdiction-specific variants require a common core protocol and locally validated additions. Back-translation and scenario testing are useful for conceptual equivalence, but literal translation alone cannot establish cultural comparability.

Digital distribution and interface evidence

Digital journeys require a record of the interface state, device category, browser context, geographic eligibility, product version, and timestamp. Price, policy terms, warnings, and navigation may vary as a function of session data, customer answers, experiments, or release versions. A screenshot demonstrates a displayed state at a particular time; it does not establish the experience of every user.

The researcher should preserve the sequence of offers and consent states, including what appears by default, what is required to proceed, how a user rejects an add-on, and whether the final quote changes. A new session may produce different results, so repeat observations should follow a lawful protocol and avoid affecting production systems or collecting other customers' information. A public product page can be examined without a completed purchase; a protected portal or third-party login requires authorization. Technical observation should not bypass access controls.

For embedded insurance, the unit may be the primary transaction plus an optional policy. A useful experiment assigns equivalent users to different displayed purchase paths or compares approved interface variants. The relevant outcomes include recognition of the premium, separate acceptance, accessibility of product terms, and ability to reject an option. The researcher records whether coverage is opt-in, opt-out, or bundled, while legal classification depends on product and jurisdiction. A preselected option can be recorded objectively; a legal conclusion requires the applicable rule and evidence of the transaction state.

Automated assistants create additional measurement challenges. A prompt, model configuration where publicly observable, and conversation history can influence the result. The same scenario may yield variable responses. Repeated trials can characterize output variability under controlled inputs, while the vendor's exact model and internal mechanisms may be unknown. Claims of systematic treatment need a sampling design over relevant interface versions and time periods. Static screenshots have limited value for stochastic conversations.

Data quality during collection

A data manager reviews incomplete fields, contradictory responses, and missing artifacts without altering the underlying observations. Corrections are appended with a timestamp, editor, original value, and reason. Shopper compensation should not be tied to the discovery of deficiencies, and supervisors should avoid signaling that the expected outcome is widespread failure. A monitoring rule can pause fieldwork if detection, safety, or protocol deviations exceed prespecified limits. The purpose is to protect research quality and participant rights, not to manufacture a cleaner-looking dataset.

Suitability, Disclosure and Conduct Coding

Legal applicability and outcome criteria

The European Insurance Distribution Directive (2016, consolidated 2024) provides a legal reference for retail insurance distribution. Articles 17 and 20 establish conduct and demands-and-needs duties. Article 24 concerns cross-selling, and Article 25 addresses product governance subject to its scope. For IBIPs, Articles 26–30 add specific requirements, including investment advice suitability and the separate appropriateness assessment for relevant non-advised sales. Article 30 includes conditions for derogations; Member State choices and national law must be established before an assessment of a particular transaction.

A codebook must map each criterion to the relevant entity, jurisdiction, product, advice status, transaction stage, exemption, rule version, and proof standard. A demands-and-needs check is not interchangeable with a full IBIP suitability assessment. An appropriateness assessment has a different information scope from investment advice. The original Delegated Regulation 2017/2359 elaborates these obligations. Its 2022 consolidated wording includes sustainability preferences in the suitability framework. The precise obligations and subsequent law should be validated for the actual transaction date rather than retrospectively applied to historic fieldwork.

The codebook should distinguish legal measures from supervisory evaluation. A researcher may evaluate whether a plain-language explanation enabled understanding even if the law prescribes neither the particular wording nor the exact test. An item should identify its character as a legal requirement, guidance-derived expectation, empirical observation, or analytic outcome criterion. A separate field documents the conditional basis of the score. This prevents the statistical rubric from silently creating a new legal duty.

Structural design of the codebook

A binary item can be suitable for whether a stated document was delivered, provided the delivery trigger occurred. Ordinal scoring can represent absent, partial, and complete explanation, with precise thresholds for each state. A separate missingness category is essential. Codes for not applicable, not yet due, not observable, and evidence insufficient should remain distinct; none automatically means pass or fail.

Dimension Observable evidence Interpretive boundary
Needs assessment Questions and recorded answers concerning the defined need Asking a question does not demonstrate its use
Disclosure Delivery, timing, contents, and explanation of material terms Delivery before an early stopping point may not be legally due
Product fit Contract terms compared with scenario constraints Projected, not realized, consumer outcome
Pressure and consent Time constraints, assertions, option states, acceptance sequence Legal significance depends on context and rule
Accessibility Response to a prespecified support request A scenario tests a response, not all disabled customers
Price and value Total costs, coverage, benefits, exclusions, available alternatives Actuarial value requires additional population and contract evidence
Post-sale support Response to genuine service or authorized test inquiry No inference about claim performance without suitable evidence

A suitability rubric starts with the customer's objectives and non-negotiable constraints. A low-cost product with a coverage exclusion that removes the required protection may be inferior to a more expensive policy that actually covers the risk. An illustrative homeowner seeks flood protection, but a cheaper policy excludes flood entirely. The mismatch concerns scope before price. If the policy only covers a narrower event that the customer knowingly accepts, the observation needs to record the agreed scope and the alternatives considered. A product may be defensible for one objective and inconsistent with another, so a single undifferentiated pass rate conceals important trade-offs.

For an investment product, a promised illustration is not a contractual guarantee. Evaluation should distinguish guaranteed benefits, projections, risk-bearing structure, recurring costs, surrender values, and holding-period assumptions. A hypothetical customer with a liquidity horizon of five years may encounter a product with prohibitive early surrender terms. Even if long-term returns could meet an investment target, the horizon mismatch warrants separate coding. A claim that the product is strictly dominated requires comparable contingent benefits, risks, costs, and contractual rights; a lower quoted premium alone is insufficient.

Disclosure quality, consumer understanding, and pressure

A disclosure measure needs a temporal anchor. Documents available after policy inception may be relevant to support, but they cannot substitute for information legally required before a commitment. Where a test stops early, the researcher distinguishes a failure observed before the cutoff from a document not yet due. Explanations can be evaluated for completeness, clarity, responsiveness, and consistency with the actual policy.

Consumer understanding is a partially latent outcome. A trained shopper who knows a product description may not respond like an inexperienced customer. A knowledge check following a scripted interaction can capture whether specified information was communicated in comprehensible form, but it is vulnerable to the shopper's prior expertise. Separate participant experiments using real consumers under appropriate ethical arrangements may estimate comprehension or decision quality. Field shopping then contributes evidence about which real interface or sales script should be tested.

Pressure can arise through time limits, exaggerated scarcity, discouraging further comparison, or insistence on purchasing an unrelated product. A code should describe observed words or actions before inferring intent. A genuine deadline explicitly disclosed with a valid commercial basis differs from an unsupported urgency claim. Consent quality involves an informed and voluntary choice, the ability to reject optional products, and clarity about the transaction. Evidence of pressure at one point may justify further review regardless of whether the shopper eventually accepts the product.

Vulnerability, accessibility, and economic outcomes

A scenario involving limited product familiarity, language needs, or requested accommodation can identify barriers. It should avoid stereotypes and avoid unnecessary collection of sensitive attributes. Genuine accessibility needs can be defined as concrete service requests and evaluated by observable responses. A shopper's real identity need not be used in an analysis where the analytical construct is a scripted request for assistance.

Price-and-value analysis includes the total monetary burden, coverage effectiveness, exclusions, deductibles, premiums, benefits, service quality, and distribution costs. A shopping exercise may reveal undisclosed charges or misleading explanations. It cannot, from price quotations alone, establish expected claims payments or the fairness of the insurer's aggregate economics. Claims and renewal data, product-governance records, actuarial assumptions, and the distribution of benefits by target market are relevant corroboration. The FCA's insurance outcomes-monitoring review illustrates the importance of complementing process metrics with product performance, customer experience, and follow-up actions.

A composite score may support prioritization, but the weighting scheme embeds policy choices. Severe individual failures can be diluted by many routine successes. Reporting a severe-issue flag alongside dimension-specific scores and an overall index prevents that loss of information. Weight sensitivity should be examined, and scores should never replace the text of the applicable law or a review of the underlying evidence.

Ethics, Evidentiary Admissibility and Rater Reliability

Mandate, proportionality, and authorization

Undercover evaluation changes the ordinary information relationship between the investigator and the observed person. It can be justified by the need to observe unaltered conduct, but its justification depends on authorized purpose, necessity, proportionality, and safeguards. The authority to conduct a test varies between jurisdictions, institutions, and methods. A general supervisory mandate does not automatically permit covert recording, use of false identity documents, access to private systems, or creation of contractual liabilities.

Before launching an exercise, legal and ethical review should document the power relied upon, who can act under it, the permitted extent of covert representation, data protection obligations, evidence retention, and the treatment of third parties. Employees' personal information should be limited to what is necessary. A provider's public-facing role does not eliminate privacy or employment rights. Research contractors should be trained and supervised, and their access to information should be controlled.

The least intrusive design capable of answering the research question is usually preferable. If an inquiry can establish the observable condition, purchasing a policy may be unnecessary. Where the question concerns the final order screen or cooling-off process, a controlled purchase may be necessary; it then requires additional authority and a clear post-purchase plan. The use of scripts should create an ordinary opportunity for the seller to act rather than deliberately provoke an otherwise unlikely violation.

From an observation record to case evidence

A structured field note is an evidentiary artifact. Its strength depends on clear authorship, contemporaneity, completeness, consistency, and independent corroboration. Where recording is lawfully authorized, preserve the original media and metadata with restricted access. Store derivatives separately. An integrity hash can detect later alteration of a file; it cannot prove the truth of the recorded event or the legality of its acquisition.

An evidence register connects each coded finding to a pseudonymous journey, artifact, rule version, coder identification, and decision history. When a researcher paraphrases an oral statement, the record should say so. A screenshot should distinguish the display time from the time of the actual transaction. Document version mismatches can be material: a policy condition retrieved after the encounter may differ from the version offered.

Admissibility, relevance, weight, and sufficiency are separate legal and evidentiary questions. A finding used for risk prioritization may have a lower evidentiary burden than an adjudicated sanction. A formal case requires applicable powers and procedures, accurate attribution to the responsible entity, assessment of alternative explanations, preservation of potentially exculpatory materials, and a meaningful opportunity to respond where the law provides it. This monograph specifies methodological safeguards; it does not state a universal admissibility rule.

Rater reliability and validity

The shopper who collects observations and the evaluator who determines the outcome perform distinct functions. Independent coders should apply a common version of the rubric to a prespecified overlap sample. Blinding to firm identity, treatment assignment where feasible, and other coders' results reduces expectation effects. Adjudication resolves disagreements for the final dataset but the original independent codes should be retained for reliability analysis.

Cohen's (1960) nominal agreement coefficient measures agreement after adjustment for agreement expected from the raters' marginal distributions. For two coders, kappa = (P_o − P_e)/(1 − P_e). In a hypothetical sample of 100 cases, both coders mark a deficiency in 20 cases and both mark no deficiency in 60; the first alone marks a deficiency in 10 and the second alone in 10. Observed agreement is 0.80, each coder's positive rate is 0.30, expected agreement is 0.30² + 0.70² = 0.58, and kappa is approximately 0.524. Reporting only 80% agreement obscures the effect of the common negative rating.

Kappa depends on prevalence and on differences in rating propensities; in a dataset dominated by negative cases, positive agreement can be poor even where raw agreement is high. Report the complete contingency table, positive and negative agreement, and interval estimates when feasible. Ordinal items can use weighted kappa with justified weights; continuous ratings can use an appropriate intraclass correlation or agreement analysis. Reliability for an aggregate score cannot establish that each critical item is dependable.

Validity concerns whether the rubric measures the intended construct. Coders could agree perfectly on whether a form was delivered and still fail to measure whether protection was suitable. A validity assessment includes mapping items to the legal or scientific construct, comparing judgments with actual policy terms, testing response to known controlled examples, and examining errors for particular customer profiles. Validation cases need sufficient diversity to cover borderline and rare severe conditions.

Measurement error may differ by channel. Recorded text in online chats supports exact wording review, while a physical visit may rely on notes if recording is prohibited. A cross-channel difference in failure rates can therefore arise from observability rather than sales conduct. Reporting channel-specific evidence quality and using common observable endpoints can support more defensible comparisons.

Governance and publication of results

The supervisory organization should allocate distinct responsibility for sampling, field management, legal review, scoring, statistical analysis, and case referral where resources permit. An independent review of contested severe findings reduces the risk of institutional confirmation bias. Publication of aggregate findings should disclose the scope, sampling, material limitations, and status of enforcement decisions. Firm identities should be disclosed only when justified and lawful. Debriefing observed individuals or firms should be decided under applicable law and investigative needs.

EIOPA, BaFin and International Applications

European coordinated exercise

EIOPA's 2025 coordinated IBIP study covered eight EU Member States, reporting 568 visits and 454 accepted for detailed analysis. Participating undertakings represented at least 35% of gross written premium, and the countries included at least two relevant channels each. This selection describes research reach rather than a demonstrated random sample of consumer journeys. The report expressly limits its intended use for determining regulatory compliance.

Its official results announcement reports that 84% of proposed products matched at least one previously specified objective or need. The logical operator is important: a product consistent with one criterion may be inconsistent with another. The 2025 study also identifies weak or absent associations between process thoroughness and product fit. The analytical implication is a need to evaluate procedure and alignment jointly. The evidence does not establish that gathering less information would cause more suitable advice.

The valid-visit ratio, 454/568, is approximately 79.9%. The difference of 114 observations reflects eligibility exclusions as defined by the study. An investigator must not simply assign unsuitable recommendations to all excluded cases or treat them as successful recommendations. The excluded cases affect the interpretation of results conditional on accepted journeys.

The national coordination makes a shared methodology useful for convergence. The national market mix, legal environment and channel composition remain relevant. Pooling selected countries cannot establish an EU-wide frequency absent defensible market weights, known inclusion probabilities, and appropriate coverage. Mutual learning can therefore be based on common symptoms and design lessons without overstating geographic representativeness.

German supervisory exercise

The indexed official BaFin advice investigation reports 72 consultations involving six insurance companies and their distribution partners during March–June 2024, including contract purchases. Direct retrieval of BaFin's page was blocked during the research session; therefore, the discussion is restricted to this indexed description and the corroborating public German industry response of June 17, 2025. The latter acknowledges the limited sample and concerns involving assessment, documentation and suitability. It is a stakeholder response rather than independent verification of every finding.

Contract purchase potentially permits observation of late-stage contractual documentation and cancellation arrangements that inquiry-only designs cannot reach. Because German fieldwork was part of the coordinated exercise, its count should not be added to the EIOPA total as an independent sample. The precise overlap and weighting treatment would require the original exercise data. The limited accessible evidence here does not support firm-specific enforcement or comparative rankings.

The contrast is methodologically important. A completed purchase can expand the scope of observation, but differences in stopping rules make rates calculated at the final contract stage incomparable with rates from a pre-purchase design unless the assessment aligns those stages. A national supervisor may use an authorized purchase specifically to investigate final advice records and costs; the broader coordination may serve a descriptive or exploratory purpose.

Later European developments

EIOPA's third IDD report announcement of March 2026 places sales-process issues and digital distribution within the wider regulatory evaluation. Its second coordinated exercise notice of April 27, 2026 announces online non-life insurance shopping across ten Member States, with results expected in the first half of 2027. That notice provides evidence of the planned program, not evidence of its findings. This study makes no assertion about the results of the second exercise.

The move from investment-oriented advice to online non-life distribution changes relevant outcome variables. For non-life policies, excluded perils, limits, deductibles, the connection between answers and coverage, and optional add-on selection can be central. Digital evidence may make disclosure states reproducible while dynamic interfaces and personalized quotes require explicit version control. The earlier IBIP findings cannot be assumed to predict the later exercise's results.

Hong Kong banking distribution

The HKMA's 2020 mystery shopping program report describes fieldwork covering 351 branches of 20 retail banks, with 352 samples across investment and insurance activity; not every sample concerned insurance. It reports weaknesses in some non-linked long-term insurance recommendations and financial needs assessments, including affordability mismatches. The study's stopping boundary meant some final-transaction steps could remain unobserved.

The institutional focus is authorized banking institutions rather than all insurance intermediaries. This comparison highlights the importance of identifying the organization responsible for the encounter and the other parties involved in distribution. A bank, insurer and intermediary may have different duties while contributing to the same journey. An exercise can examine the sales interface and later trace the responsible control owner through records and agreements. The Hong Kong findings cannot establish the overall prevalence of deficient sales in other channels or jurisdictions.

United Kingdom monitoring and behavioral research

The FCA's Consumer Duty monitoring review examined reports from twenty larger insurance firms and regulated service providers. Its focus was firms' monitoring arrangements rather than direct observation of sales. For supervisory science, the relevance is that this evidence addresses organizational capacity to detect and remediate deficiencies. It complements shopper observations rather than becoming a mystery-shopping sample itself.

The FCA's 2014 add-on insurance experiment summary reports that the add-on sales mechanism weakened consumer comparison behavior; the authors' views disclaimer applies. This behavioral evidence supports the design of investigations into the timing and salience of an optional insurance offer. Its controlled experimental setting differs from an inspection of an actual firm. The strength lies in isolating a mechanism; real-market measurement is needed to document the mechanism's occurrence and scale.

Indian life insurance field experiments

Anagol, Cole, and Sarkar (2017), in their Indian life insurance study, report evidence from field experiments on agents' advice and describe a substitution response when commission disclosure applied to one product category. The published abstract supports these broad results; the full article's detailed estimates and identification assumptions were not examined here. The evidence indicates why remuneration and product substitution deserve attention together in a shopping design.

A commission-driven mechanism can be tested by recording proposed product categories and examining incentive schedules during an authorized follow-up. The observed selection of a high-remuneration product alone does not prove that the remuneration caused the sale; other product attributes and constraints may be relevant. A policy intervention targeted at one product can displace a practice to a different product or channel. This possibility justifies measurement of the full set of relevant offers following remediation.

Comparative interpretation

Institution or study Primary observation Main limit for comparison
EIOPA 2025 Coordinated IBIP shopping journeys Selected markets and valid-visit criteria
BaFin 2024 fieldwork German consultations including purchases Restricted access to original report in this research
HKMA 2020 Bank sales of investment and insurance products Mixed-product sample and limited transaction completion
FCA 2024 Firm outcome-monitoring reports Organizational review, not direct shopping
FCA 2014 Behavioral experiment on add-on insurance Experimental mechanism, not market inspection
Anagol et al. 2017 Field experiments on Indian life insurance advice Abstract-level access; specific market and products
EIOPA 2026 Announced non-life digital exercise Findings pending as of the announced timetable

The comparative conclusion is not a league table of jurisdictions. Each program provides a different observation of the relationship among product, distribution, incentives, and customer outcomes. A regulator can derive common research questions while preserving jurisdiction-specific duties and limits. A comparative statistic becomes meaningful only when its target population, endpoint, design, product category, and measurement quality are harmonized or explicitly standardized.

Causal Inference, Reporting and Regulatory Impact

Descriptive findings and causal effects

A descriptive visit rate estimates a feature of the observed sample and, with a defensible design, its target population. A causal effect concerns what would change under an intervention relative to a specified counterfactual. The two are linked through research design rather than through the magnitude of a rate.

In potential-outcomes notation, let Y_i(1) describe the outcome of eligible encounter i under a treatment, and Y_i(0) under a comparison condition. The average treatment effect for the defined population is E[Y_i(1) − Y_i(0)]. Random assignment of a shopper cue can identify an intention-to-treat contrast under consistency, relevant measurement, limited interference, and adequate handling of missing outcomes. It does not identify the causal effect of all characteristics correlated with that cue in ordinary markets.

For example, a hypothetical experiment assigns 200 interactions to each of two scripted levels of product familiarity. If 40 of 200 unfamiliar encounters and 20 of 200 familiar encounters yield a specified poor-fit offer, the risk difference is 10 percentage points and the risk ratio is 2. Under independent Bernoulli sampling, the approximate standard error of the difference is sqrt(0.20 × 0.80 / 200 + 0.10 × 0.90 / 200) = 0.0354. The approximate 95% interval is 3.1 to 16.9 percentage points. This calculation assumes the assignment and independence conditions; actual branch-clustered randomization requires inference at the appropriate assignment level.

Selection after treatment complicates interpretation. If a retailer refuses to meet some assigned profiles, estimates conditional on receiving advice may refer to selected subpopulations in each arm. A stage-specific analysis should use a common assignment denominator for early outcomes and explain whether later comparisons condition on a post-assignment event. The selection mechanism may itself reveal an access barrier. Reporting both stages provides insight without treating the later conditional estimate as the unconditional causal effect.

Multilevel measurement and heterogeneous effects

Encounters are frequently nested within adviser, branch, firm, product, time period, and channel. A logistic mixed-effects specification for a binary deficiency could include fixed effects for assigned profile and channel, and random intercepts for firm and shopper. This may describe between-firm variation and separate systematic scenario effects from some nuisance variation. Its estimation relies on model assumptions, and small numbers of firms make variance components and rank estimates unstable. A cluster-robust estimator or randomization-based analysis may be better aligned with the sampling or assignment design.

The researcher should prespecify the primary outcome and contrast and distinguish exploratory subgroups. Comparing many dimensions, firms, and profiles creates multiplicity. A program may control familywise error for confirmatory tests or a false discovery rate for a defined screening family. These choices do not remove the need for substantive assessment of severity. A single serious observation with credible evidence may warrant investigation even when a frequency comparison cannot meet a statistical threshold.

When outcome-specific results are sparse, publish uncertainty rather than rank firms by raw percentages. A rate of zero failures in a sample of twenty does not establish a true rate of zero. For independent trials, the exact one-sided 95% upper bound after zero events is 1−0.05^(1/20), approximately 13.9%. Clustered or nonprobability data have different inference constraints. The illustration demonstrates the value of reporting the sample size and design rather than only percentages.

Triangulation and evaluation of harm

A supervisory evaluator can connect a shopping observation to administrative evidence at an appropriate level. If several shoppers receive a policy that excludes the loss they seek to insure, the supervisor may inspect actual customer files, product governance, complaints, and claims coded under that exclusion. The administrative data test whether the observed mechanism reaches real policyholders and produces contractual consequences. The researcher should avoid double-counting the same event when it appears in multiple systems.

A monetary detriment estimate requires a defined affected population and a credible counterfactual. Suppose an optional policy costs $120 and a controlled file review identifies 500 documented purchases without required informed consent. Multiplication gives $60,000 of gross premiums for those purchases. It does not automatically establish $60,000 of net welfare loss or legally payable redress: benefits received, applicable interest, cancellation rights, statutory remedies, and case circumstances matter. The example separates measurable expenditure from the legal and economic concept of loss.

For claim-related outcomes, the denominator may be covered claims, submitted claims, eligible policies, or policyholder-years. An exclusion can affect all policies even where few claims are submitted. Complaint rates depend on awareness, access and incentives to complain. A low frequency of complaints can coexist with limited understanding of poor coverage. The outcome evaluator therefore uses multiple indicators and a documented causal hypothesis, while avoiding unsupported extrapolation from shopper counts to affected policyholders.

Proportionate supervisory response

A useful reporting ladder separates the initial signal, documented finding, corroborated systemic issue, and formal case decision. It is an organizational description of evidence handling rather than a new legal standard. An initial signal could trigger a repeat visit or request for records. Recurring evidence might justify thematic inspection of product governance or incentive controls. A formal enforcement determination rests on the relevant authority, proof and procedure.

The response should be directed toward the mechanism causing harm. If an interface systematically conceals the cost of an optional policy, redesigning it and testing recognition may be appropriate. If a training problem causes incorrect explanations, revised materials and observed competence may be relevant. If incentives reward sales incompatible with customer needs, changing the scorecard and monitoring substitution across products may be necessary. Remediation must extend to affected policyholders where the applicable law and evidence require it.

Thresholds can support triage, but fixed percentages are often inadequate. The importance of a deficiency depends on the legal nature of the obligation, likely harm, affected population, evidence strength, persistence and reversibility. An investigation should record why its response is proportionate, including less severe alternative explanations and any contradictory evidence.

Testing whether remediation works

Re-shopping after a supervisory intervention is useful only when its comparison structure matches the claim. A simple before-and-after contrast can reflect seasonal changes, different shoppers, product withdrawal, staff turnover, or altered market conditions. An intervention can increase apparent compliance because staff detect a familiar shopping script. Its effectiveness for actual customers must be distinguished from improvement on the specific assessment instrument.

Where feasible and lawful, a phased intervention or randomized implementation allows stronger causal inference. A comparative interrupted time series can examine changes in level and trend with a defensible comparison series and sufficient observations. A difference-in-differences design requires a credible parallel-trends assumption, consistent measurement, and attention to anticipation, spillovers, and changes in sample composition. Essential corrective measures should not be withheld solely to create a control group.

A hypothetical baseline has 30% observed deficient advice in a treated group and 20% in a comparison group. After the intervention the corresponding rates are 15% and 18%. The difference-in-differences estimate is (0.15−0.30)−(0.18−0.20) = −0.13, i.e. a 13 percentage point relative decline. The quantity is causal only under the design assumptions. A follow-up program should also examine actual policyholder outcomes, adverse substitution, and persistence.

Scientific reporting

A report needs an explicit frozen corpus of completed and attempted observations, protocol version, legal mapping, sampling frame, dates, channels, assignment rules, stage denominators, missingness, scoring reliability, and uncertainty. It should document the distinction between discovery and confirmatory analyses and the effect of important sensitivity assumptions. Public reports can use aggregated results while restricted case annexes retain evidentiary material needed for lawful proceedings.

The same transparency is required when findings appear reassuring. A small or selected sample without observed serious failures cannot prove absence of harm. An inspection covering only successful sales does not address accessibility barriers. A completed evidence cycle includes the initial question, actual observation, reproducible scoring, follow-up evidence, response, and evaluation of whether the response improved the target outcome.

Conclusion

Supervisory mystery shopping is a disciplined method for observing insurance distribution under controlled or systematically selected real-world conditions. Its conclusions are strongest when the unit of analysis and the population correspond to the actual sampling mechanism, the scenario activates the relevant conduct obligations, and the scoring rubric distinguishes process, product alignment, and outcomes.

The international evidence demonstrates complementary uses of the method. European coordinated shopping illustrates the importance of measuring alignment independently from the thoroughness of sales procedures. German contract purchases demonstrate the potential value of observing later stages, subject to the accessible evidence limits. Hong Kong bank surveillance illustrates institutional and transaction-boundary distinctions. British monitoring and behavioral studies, and Indian life insurance field experiments, supply complementary evidence on governance and incentive mechanisms.

Three operational implications follow. First, regulatory duties must be mapped to product, jurisdiction, entity, advice status, and transaction stage. Second, statistical analysis must account for selection, clustering, missingness, measurement error, and the distinction between descriptive and causal questions. Third, the evaluation of actual consumer harm requires corroboration from policy terms and administrative outcomes. These implications follow from applying established supervisory, legal, statistical, and economic reasoning to the research question; they are not proposed as a novel theoretical system or universally binding legal standard.

The documentary scope supports the methodological conclusions without supporting market-wide prevalence of deficient sales, firm-level sanctions, or quantified aggregate consumer detriment. No fieldwork or independent external peer review was conducted. Official source access was uneven and the underlying individual-level data were unavailable. Any implementation should undertake local authorization, legal version validation, pilot testing, and a design-specific statistical analysis before relying on the results for enforcement or remediation.

References