Insurance by BogdanovaThe Encyclopedia of Insurance and Risk Science

RESEARCH DOSSIER

Cyber Insurance Penetration Testing and External Attack Surface Data

Abstract. Penetration testing provides cyber insurers with evidence about security controls under deliberately selected conditions. Its informational value comes from the documented interaction between a threat scenario, an authorized tester, a defined environment, and the controls that mediate access or impact. External attack surface data help identify and prioritize the exposures to be tested, but their relationship to insured loss requires separate analysis of asset ownership, test coverage, temporal validity, selection into underwriting, the process generating incidents, and the insurance contract. This monograph integrates original technical guidance, actual insurance application forms and specimen wording, empirical research, comparative supervisory requirements, and actuarial principles. It develops a source-critical account of the limitations of positive and negative test findings, analyzes the causal distinction between measurement and subsequent remediation, and presents reproducible hypothetical examples of coverage weighting, selection bias, information value, and loss aggregation. It concludes that penetration-test evidence is most informative when attached to a specific risk decision and verified against business exposure and policy scope. A successful security assessment cannot, by itself, establish a numerical probability of an insured claim. Likewise, a finding of an exploitable weakness cannot, by itself, establish causation, coverage, or financial severity. No live testing, proprietary claim analysis, universal effect estimate, or new named theory is asserted.

Keywords: Cyber insurance; penetration testing; external attack surface; cyber risk engineering; adversarial testing; measurement reliability; test coverage; underwriting selection; actuarial validation; risk mitigation; insurance contracts; operational resilience; cyber accumulation.

Introduction

The insurance value of a penetration test is the value of evidence about a contingent loss process. An insurer may know that an applicant has commissioned a test, may receive a summary of its findings, or may have access to carefully restricted evidence supporting particular findings. Each state represents a different information set. An underwriting institution must translate that information into a decision concerning the insured party, the applicable exposure period, the contract offered, the price charged, and the capacity committed. The transformation is scientific and institutional because technical observations become relevant to financial decisions only within specified assumptions and rules.

Consider a business that exposes a customer-facing service and relies on an external identity provider. A test demonstrates a weakness in the service under an agreed set of conditions. The weakness could be material to confidentiality, operational continuity, or both. Which consequences would arise in an actual incident depends on the surrounding controls, the data and operational functions involved, the capabilities of the adversary, and the response of the business. Whether the financial consequences become insured losses depends additionally on the policy. The initial report is evidence concerning some of these conditions; the insurer needs evidence concerning the remainder.

The approved scientific object of this dossier is the use and validation of penetration-test evidence and the external observations that inform test scope. A companion research object in the encyclopedia addresses the quality of passive attack-surface scanning as a measurement activity in its own right. Here external observations enter a more specific inquiry: how they guide authorized adversarial validation and affect the reliability of an insurance inference. Neither external measurements alone nor a discussion of offensive techniques defines the object. The approach is non-operational; no real target was scanned, challenged, or tested.

Three research questions organize the inquiry. First, what constitutes sufficiently reliable evidence of a security weakness or control response? Second, how do the boundaries of a test condition the meaning of an observation and its potential influence on underwriting and claims? Third, how can researchers distinguish the information supplied by testing from the effects of selecting which applicants are tested, improving security in response to testing, and changing insurance terms? These questions require shared terminology between security engineering, contract interpretation, and statistics.

The method is a purposive documentary synthesis rather than a systematic review claiming exhaustive literature coverage. Original documents include technical guidance, public insurance forms, policy wording, regulatory instruments, and published empirical study descriptions. Source identity and the specific material examined are recorded in the evidence companion. Where only an abstract or official indexed excerpt was accessible, claims are confined to that material. The comparative scope covers the United States, Australia, and the European Union; it makes no inference that their laws or insurance markets are identical. Synthetic examples are arithmetic demonstrations with defined assumptions and no empirical status. The parent Science, Cyber Insurance, supplies the larger framework of insured losses, security incentives, and portfolio dependence. This study supplies the narrower engineering-to-insurance analysis.

A recurring methodological distinction is between a fact and the interpretation placed upon it. The fact that a test produced a result may be supported by a report. The inference that the result lowers or raises prospective insured loss requires a model and an appropriate population. The further conclusion that a historical claim is payable requires the issued contract and relevant facts. The monograph preserves all three levels through its analysis.

Conceptual and Historical Foundations

From technical examination to risk selection

A penetration test is a bounded, authorized security assessment designed to investigate the behavior of systems and controls under specified adversarial conditions. External attack surface data describe externally observable parts of an organization's technology environment and associated signals. The two activities intersect when observations guide the definition of hypotheses or the allocation of test effort. Their units of observation differ: a scan may record a service or configuration indicator, whereas a test may record the behavior of a control or a demonstrated consequence under agreed conditions.

Scarfone and colleagues (2008) set out the complementary nature and limitations of assessment methods in NIST SP 800-115. Their guidance distinguishes scanning from more resource-intensive testing, recognizes risks to live operations, and places mitigation after evaluation of the findings. Although the publication predates many contemporary cloud architectures, its distinction between assessment goals, techniques, and constraints remains analytically useful. Its date and technical purpose qualify its use: it supplies a methodology, not a contemporary census of insurer requirements.

A penetration test has a beginning, a defined scope, a time budget, rules of engagement, a testing environment, and an end. The tested organization may authorize a particular degree of examination, limit the data to which testers have access, or exclude operations where interruption would endanger customers. An assessment's rigor therefore depends on the relationship between its question and its boundaries. A narrowly designed exercise can answer a narrow question with great reliability. It becomes misleading only when a subsequent user treats the answer as a comprehensive assertion.

Risk selection uses this evidence for a different objective. Where a security engineer focuses on the control, an insurer also needs the exposure that the control protects. For a small organization, the relevant concentration may be the business's dependence on a single outsourced platform. For a diversified enterprise, both the platform and the distribution of business functions across subsidiaries may matter. The same technical observation can consequently carry different financial implications. This is an ordinary application of the distinction between hazard, exposure, vulnerability, and consequence.

Romanosky, Ablon, Kuehn, and Jones (2019) examine filed contracts, questionnaires, and rates in their content analysis of cyber insurance policies. Their historical qualitative evidence shows the usefulness of treating the insurance application and the price schedule as distinct documentary objects. A question about testing establishes that the question was asked in the studied forms; it does not reveal how much the answer changed a particular underwriter's judgment. The study's three-state filing sample cannot identify the causal effect of penetration testing on contemporary global insured losses.

What the evidence measures

Five different observations deserve separate data definitions. A visible service is an observation about apparent external reachability from an observation point. A suspected vulnerability is a hypothesis about a relevant technical condition. A confirmed finding is evidence that the condition or particular failure was established under specified circumstances. A remediation verification is evidence concerning a later configuration and repeat assessment. A business-impact observation concerns the actual or simulated consequence for a defined operation. None of these should share an undifferentiated label of risk score.

Even within the confirmed category, the achieved objective matters. Demonstrating a discrepancy in authorization policy can be sufficient to establish a control defect while leaving uncertainty about access to economically valuable information. A simulated interruption may establish that certain continuity arrangements would be challenged, yet the costs of a real interruption may depend on actual workloads, alternative suppliers, restrictions during the test, and the timing of recovery. The evidentiary relation therefore has both technical and financial dimensions.

The insurance scientist also distinguishes the testing organization from the authorization holder and the named insured. A parent may commission an examination covering systems used by subsidiaries while the contract's insured definition includes only certain entities. The report may cover a technology environment owned and operated by a vendor but the applicant has only contractual use rights. The test authorization may be issued by a service provider on terms that omit unrelated tenants. These relationships determine whether findings can legitimately be attributed to the insured exposure and whether their collection was permissible.

The terminology of attack surface has sometimes encouraged an emphasis on extensive coverage of observable services. For underwriting, the most defensible unit is often a business-function–system–entity relationship. One service can support several critical activities; multiple services can belong to a single redundant function. Counting services as interchangeable observations obscures that difference. An insurer can use service counts as descriptive data, but the interpretation requires linkage to business value and dependency.

Threat and environment

The relevant comparison between a threat signal and a test result concerns conditionality. FIRST's EPSS documentation defines its 30-day output in terms of exploitation activity observed in its contributing data. In contrast, a test asks whether a specified scenario is feasible in a particular authorized setting. External threat indicators can support prioritization; their probabilities do not establish the probability that an organization experiences a covered event. Similarly, a demonstrated condition need not be frequently attempted by actual adversaries.

A well-defined risk question might examine whether a business function is protected against a specified class of disruption and how the available evidence should adjust an estimate of its expected insured loss. The question joins threat activity, susceptibility, business consequences, and contract response. It cannot be answered merely by classifying the technical severity of a finding. A technical score can organize review without being calibrated in dollars, and an actuarial estimate requires a consistent outcome variable.

The external perimeter is also historically contingent. Organizations change providers, migrate workloads, introduce new services, discontinue products, and acquire subsidiaries. Data collected during an assessment belong to a particular configuration. The loss process extends over the policy term, so an underwriter interpreting the report must establish the elapsed time and intervening changes. The scientific use of a point observation is to situate it within a process of changing exposure, rather than to treat the organization as a fixed technological object.

Measurement Reliability and Evidence Interpretation

The observation and its provenance

The reliability of a test finding concerns the correctness and repeatability of the evidence for a specified proposition. It is distinct from the extent to which the test covers possible propositions. A result can be highly reliable about one control and uninformative about most other controls. An excellent report on an isolated application can therefore have limited adequacy for evaluating the entire enterprise. This distinction resolves a frequent ambiguity in the expression independent test: independence may enhance confidence in the examiner without expanding the contractual scope, time budget, or tested environment.

A traceable evidence record should establish the assessment date, authorized scope, identity of the assessed organization, system or business function, initial conditions, type of examiner, finding category, supporting observation, restrictions, remediation state, and validation status. The report may provide an appropriately sanitized evidence description rather than publish sensitive details. An underwriter needs sufficient information to evaluate a proposition, not unrestricted access to all testing material.

For research, reproducibility means that the same input data and stated procedures can reproduce the calculation or evidence coding. Technical repetition in a later environment is a different matter. A subsequent test may fail to reproduce an initial result because of a corrected configuration, an intervening deployment, a changed permission set, or chance variation in observations. This does not invalidate the historical finding. It changes the nature of the comparison. A negative retest should identify whether the previous weakness was specifically addressed or whether the later assessment simply had different coverage.

Classification also requires explicit categories for assessed and verified, assessed with uncertain results, assessed without finding, deferred for safety, and outside scope. Combining all nonpositive categories into a single zero removes the distinction between absence of evidence and evidence of absence. For underwriting that distinction matters because a missing examination is likely to be related to resources, business complexity, or operational constraints.

Written findings can have errors of attribution and errors of interpretation. An external indicator associated with a public network address might actually pertain to a hosting provider or historical ownership. A tester may confirm the behavior of a specific component, but an insurance analyst may later associate it with a different named insured. An effective correction process allows the applicant to provide evidence of ownership, control responsibility, migration, or decommissioning, and preserves both the initial observation and the documented correction. Correcting attribution changes the evidence, not the organization's inherent conduct.

Sensitivity, specificity and inference from silence

In a stylized validation experiment, let D denote the presence of a specified defect in the population of assessable control-scenario pairs. Let T denote a positive test result. Sensitivity is P(T positive | D), and specificity is P(T negative | no D). These definitions require an independent method of establishing the reference condition. Typical exploratory assessments do not supply a complete gold standard for undiscovered weaknesses. Their sensitivity for all possible defects is generally unidentifiable from the report alone.

Assume, solely for illustration, that sensitivity is 0.80, specificity 0.95, and prevalence of the specified defect is 0.10. Among 1,000 comparable assessment units, 100 have the defect; 80 of them are identified, 20 are missed. Of 900 units without the defect, 45 yield false positives. The positive predictive value is 80/125 = 64%. The probability of a defect given a negative result is 20/875 = approximately 2.29%. If prevalence is 0.01 with the same conditional performance, the expected counts per 1,000 units are eight true positives and 49.5 false positives. Positive predictive value falls to approximately 13.91%.

These quantities refer to detection of the defined defect, not to claim incidence. The second example has fractional expected counts because it represents population expectation. The change in predictive value comes entirely from the different base rate; no change in test sensitivity or specificity was assumed. Applying these illustrative parameters to an actual test report without independently estimating them would be an error.

A test that establishes one specific condition may have exceptionally strong positive evidentiary value if the finding has been independently reviewed. That fact does not establish the negative predictive value of the test as a whole. Undiscovered conditions may fall outside the scope, the allotted resources, or the particular approach. The asymmetry is especially important for insurance applications that ask whether a recent test was passed. The phrase should be translated into the test objectives and the defined acceptance criteria before use.

Even a sample designed for estimation encounters a search process problem. Testers adapt their effort toward promising observations. Units evaluated later are selected using information learned earlier. The observations are neither independent nor identically distributed by default. A raw count divided by the number of tested services is descriptive of the process carried out; interpreting it as an unbiased estimator of all vulnerabilities requires the sampling design and inclusion probabilities.

External data as the sampling frame

The USENIX breach forecasting study by Liu and colleagues (2015) supplies a historically important demonstration that external features can contain predictive information. Its conference abstract reports a classifier using externally observable features and over 1,000 reported security incidents from 2013–2014, with a reported true-positive rate of 90% and false-positive rate of 10%. The response variable is recorded security incidents, not paid claims. Its results cannot be transported to modern insurance portfolios as calibrated predictions without a new validation population.

Its relevance to the present subject is methodological: external observation can contribute information even before the applicant supplies internal documentation. The penetration test uses this information to focus a bounded inquiry and examine the validity of particular hypotheses under authorization. An underwriting model should preserve the distinction between the original external feature, the test-selection event, the resulting finding, and any corrective action. A later analyst can then learn whether each stage provides additional information.

A sampling frame based solely on the external perimeter omits internal systems, managed endpoints, cloud control-plane dependencies, personnel access and physical processes that are not externally visible. Some external observations lack business relevance; some invisible dependencies may be essential. The correct response is to document the sampling frame and incorporate additional evidence where it matters to the insurance decision. Data gaps are variables of uncertainty, not zeros in an automatically harmonized score.

Time, quality and uncertainty

The age of a report matters through changes relevant to its findings. A recent test may be outdated after a substantial technology migration; an older finding may remain pertinent if evidence establishes that the implicated control and context have not changed. A simple fixed expiration date can support administrative consistency, but the scientific interpretation requires the date of the assessed configuration, the magnitude of intervening changes, and the relevance of the observation to the future exposure period.

A useful sensitivity calculation illustrates temporal uncertainty. Suppose that at inception a validated finding is assumed relevant to a defined control state with probability 0.9. Assume hypothetically that a relevant unobserved change occurs independently each quarter with probability 0.2 and that the occurrence invalidates that particular observation. After three quarters, the probability that the original observation remains applicable under these assumptions is 0.9 times 0.8 cubed, or 0.4608. This is not an empirical decay model. It demonstrates why the observation date and change history should remain available alongside the finding.

A report that assures management about a test result may omit details irrelevant to management's purpose. For an actuary, a useful abstraction might be an evidence quality category and the dates of assessment and verification. That category needs a documented definition and a process for recoding when underlying evidence changes. The data documentation must make the abstraction reversible for authorized review; otherwise the insurer cannot investigate changes in model performance caused by changes in report formats rather than security posture.

Test Coverage and Insurance Risk Engineering

Exposure-weighted scope

A test's scope is a mapping between objectives and testable units. Units may be business applications, external service groups, identities, infrastructure boundaries, recovery functions, or defined scenarios. The unit must be fixed before coverage is calculated. A numerator expressed in systems tested and a denominator expressed in business functions has no stable interpretation. In complex environments a function may be examined across several systems and scenarios; a coverage record should distinguish these dimensions.

For a reproducible illustration, consider four function groups with exposure weights of 0.40, 0.30, 0.20 and 0.10. Assume the proportions of an agreed scenario set actually examined are 0.75, 0.50, 1.00 and 0.00. The unweighted average of these proportions is 56.25%; the exposure-weighted average is 0.40(0.75) + 0.30(0.50) + 0.20(1.00) + 0.10(0.00) = 0.65. The 65% result describes weighted planned coverage only. It is not a measure of security effectiveness or the probability of detecting an arbitrary real-world attack.

Weights should represent an explicitly stated purpose. A security engineer may weight functions by criticality to operations. An insurer estimating interruption loss may use expected contribution to gross profit or contingent service dependencies. A liability underwriter may emphasize sensitive records, legal exposure or the contractual limit distribution. Replacing one weighting system with another silently can make a comparison meaningless even if every test is identical.

Coverage should also be recorded separately for functions excluded by the authorization and functions nominally in scope but not investigated. Where a high-risk operation is excluded because testing is unsafe, the insurer may consider alternative evidence. A safe simulated exercise can contribute evidence about response personnel or recovery arrangements. Its inferential limitations should reflect the difference between simulated and production environments. Conflating simulation with live performance overstates the degree of verification.

Assessment forms and the information actually requested

The original Emergence enterprise proposal form (2024, Assessments, p. 6) asks about scan coverage, independent test frequency, remediation of serious findings, and simulation software. The form provides direct documentary evidence that these topics enter at least one Australian underwriting information process. It does not establish a numerical schedule for translating answers into premiums.

The CFC Australian application form (copyright through 2019) separately requests information about perimeter scans, penetration tests, and testing providers. This distinguishes the types of information at the questionnaire level. The original questions are historically dated examples, not evidence of uniform current practice. The two forms also illustrate the limited information content of a single affirmative response: the frequency of testing, the assessed scope, the status of corrective action, and the provider's identity are separate facts.

A scientifically useful response to a questionnaire would attach structured evidence to the particular proposition requested. If a question concerns whether testing occurred, the assessment dates and scope answer that question. If it concerns whether significant findings were remediated, the relevant evidence is a dated closure record and, when appropriate, a verification result. If the question concerns continuing operation of controls, an annual report may need corroboration by change and operational records. These distinctions avoid converting one documentary fact into several unsupported assurances.

Insurers can respond differently to incomplete evidence: seek clarification, commission further authorized review, price uncertainty, modify terms, or decline a risk. The economic consequence of information quality depends on the cost of acquiring the evidence and the decision it could change. Requiring an expensive assessment even where its results cannot affect any underwriting decision imposes costs without corresponding informational value. Conversely, a carefully targeted inquiry may be valuable even if highly comprehensive testing would be disproportionate.

Remediation, retesting and the causal chain

A test may identify a defect; the insured may subsequently correct it. These are distinct interventions. The security consequences of ordering a test depend partly on what happens after discovery. The report may change the behavior of management, the terms of insurance, or the availability of coverage. Modeling all of these effects as a direct reduction in hazard due to testing confuses information with action.

A completed remediation record should relate the initial finding to a specific corrective action or an explicit risk acceptance decision, identify who owns the control, and document the later validation. The underwriting distinction between remediated and accepted findings is financial as well as technical. Acceptance of a residual condition may be reasonable if a compensating measure reduces its consequences, yet the remaining uncertainty must be reflected in the risk interpretation.

The cost of testing similarly belongs to the economic analysis. A narrowly scoped review may be less expensive but leave significant uncertainty. An extensive exercise may consume management time, disrupt services or require protection of sensitive information. The research problem is not to maximize the number of tests; it is to identify a suitable evidence program under budgetary, safety and organizational constraints. For an insurer, the appropriate objective includes the benefit of reduced uncertainty and any separately established effect of risk mitigation.

The institutional boundary

The European Union's Digital Operational Resilience Act (Regulation 2022/2554, Arts. 24–26) distinguishes a set of resilience assessments from advanced threat-led testing of designated financial entities' functions. The original official indexed text establishes that penetration testing is one among multiple testing methods and that advanced testing includes live systems supporting critical or important functions. The ECB's TIBER-EU update (February 2025) describes changes enabling controlled threat intelligence-based testing aligned with DORA's technical requirements. These sources illuminate organizational design and coordination. They govern or guide the resilience of financial entities within their scope; they do not automatically become cyber-policy obligations for ordinary commercial insureds.

New York's Part 500 vulnerability regulation (amended 2023, section 500.5) connects annual internal and external penetration testing with risk-based scanning, manual examination of systems not covered by automated scans, and timely correction. Its covered-entity boundary and applicable exemptions matter. Comparing this rule with the EU testing regime illustrates variation in regulatory design. The applicable legal question is which entity and activity are subject to the rule. The insurance question is distinct: does a contract incorporate any relevant representation, condition, or obligation? The research record cannot supply that answer in the absence of actual policy terms.

Safety, independence and confidential evidence

Authorized testing requires a defensible allocation of responsibility. The applicant, tester, system owner, relevant provider, insurer and broker may hold different rights and obligations. The parties controlling the relevant systems must establish the scope of authorization and permitted test activity. The insurer's commercial interest in assessment does not itself confer permission to act on third-party infrastructure. The scope and safety record should be retained even when an underwriting summary intentionally contains little sensitive technical detail.

A report should identify conflicts of interest, especially when a party that designed a control also assesses its performance or sells remediation services. Independence is an evidentiary consideration, not a universal guarantee of correctness. The appropriate degree of independent review depends on the materiality of the proposition and the costs of validation. The information retained for pricing should generally be minimized to what is relevant for the defined purpose, with sensitive underlying records available only under lawful and controlled access.

Underwriting Selection and Actuarial Interpretation

Selection into testing and insurance

Testing frequency and observed findings are endogenous to the process in which the insurer obtains them. Larger and more complex organizations may be more likely to commission assessments and may also have greater financial exposure. Firms with mature control programs may identify more defects than firms that examine fewer systems. Applicants that fear adverse terms may selectively disclose evidence within the limits of applicable legal duties. Insurers may require enhanced assessment for risks already believed to be unusual. These processes create different routes to observed associations.

Consider the observed population of insured organizations. Entry into that population depends on the applicant's decision to seek insurance, the insurer's assessment, the available contract terms, and the applicant's willingness to accept them. Relevant test evidence can influence several stages. A dataset limited to bound business omits rejected and withdrawn applications and therefore cannot independently identify the information value of testing for the full application population. Selection effects also operate at renewal when organizations change insurers or when capacity is withdrawn.

Changes in price and coverage complicate interpretation of observed claims. Insured businesses with identical underlying economic risks can have different probabilities of generating a paid claim because they face different deductibles, reporting requirements, exclusions, or coverage sections. A test variable associated with expected payment may therefore reflect contractual selection as well as security differences. Research should specify whether the target is an incident, a covered event, a reported claim, an incurred loss, or paid indemnity.

Hypothetical data illustrate the size of a composition effect. Suppose 1,000 larger organizations have a 10% annual claim probability and 1,000 smaller organizations have a 2% claim probability. Assume 80% of larger organizations and 20% of smaller organizations are tested, and that testing has exactly zero causal effect in either group. The tested group contains 800 larger and 200 smaller organizations, with expected claims of 80 + 4 = 84, or 8.4%. The untested group contains 200 larger and 800 smaller organizations, with expected claims of 20 + 16 = 36, or 3.6%. The unadjusted claim-rate ratio is 84/36 = 2.3333, even though the assumed within-group causal effect is zero.

This example proves an identification problem, not a claim about real insurer behavior. An adjustment comparing tested and untested organizations within the same size strata recovers equal hypothetical probabilities. In real data, residual differences in technology, security resources, business dependence, prior incidents, policy terms, and test timing may remain. The additional data requirements follow from those plausible mechanisms.

Frequency, severity and the policy transformation

An underwriting estimate should distinguish the relevance of a finding to event frequency from its relevance to conditional severity. A weakness in prevention might influence the chance of an event; a weakness in recovery could affect the duration or cost of its consequences. Some findings can influence both. A test of response arrangements may be valuable even if it does not alter the likelihood of initial compromise. The estimate should identify the target outcome and the assumed mechanism.

Let N be annual insured claim count and X the covered financial amount before a simple per-claim retention d and maximum payment u. In a deliberately simplified model with independent identically distributed severities conditional on the selected risk state and a Poisson count of mean lambda, the expected annual payment is lambda times E[min(max(X - d, 0), u)]. Real cyber contracts may apply aggregate limits, multiple coverage parts and waiting periods, so this representation is an illustration of an insurance payment function rather than a model of every policy.

Suppose lambda is 0.08, conditional covered severity is a fixed $250,000, retention is $50,000, and the maximum payment is $150,000. Expected annual payment is 0.08 times $150,000 = $12,000. If a specified corrective intervention were assumed to reduce lambda to 0.06 without affecting severity or contract terms, expected payment becomes $9,000. If the intervention instead leaves lambda at 0.08 and reduces covered severity to $140,000, payment per claim becomes $90,000 and expected annual payment becomes $7,200. These alternatives demonstrate why frequency and severity pathways cannot be treated as interchangeable inputs. The assumed effects are entirely synthetic.

A benefit arising from better information is different. Assume two risk classes with equal prior probability and expected annual insurance payments of $6,000 and $18,000. A risk decision based on the prior uses $12,000. Suppose a hypothetical signal identifies the high class with probability 0.8 given high risk and identifies the low class with probability 0.8 given low risk. Conditional expected payment is $15,600 after a high signal and $8,400 after a low signal. The expected payment before acting remains $12,000; the information changes conditional estimates without itself reducing loss. This calculation has no implication that actual penetration tests achieve 80% classification performance.

Policy wording and claim causation

The Emergence CES 003 AUS wording (published October 2025) provides a concrete illustration of this transformation. Its p. 5 preamble identifies the wording and schedule, including endorsements, as determining coverage. Sections A, B and C address interruption, liability and response costs through different conditions involving events, discovery or notification, and in the interruption section a waiting period. The example establishes that a technical observation and a payable loss are different contractual objects.

A finding made before binding can be relevant to a factual inquiry following an incident. The historical report can help establish what was known, which affected systems were examined, and whether a relevant defect was later corrected. It does not, by itself, establish the manner in which the incident occurred or the legal consequence of a representation. A scientifically sound analysis distinguishes the actual pathway involved in the incident, the temporal relationship to the finding, the contract wording, governing law, and the applicable burden and standard of proof. Issued schedules and endorsements must be examined in an actual dispute.

Consider three hypothetical claims. In the first, the incident is causally connected to a documented control failure that was never corrected. In the second, a later compromise arises from a separate service beyond the test's scope. In the third, the initial finding was corrected but a subsequent configuration change creates a different weakness. The same original report may be part of the evidence in all three cases; its relevance differs substantially. It supplies no automatic outcome for any of them.

The observation that some contracts contain event-response costs and interruption coverage should not be used to infer that every preventive assessment is insured. A test commissioned before any cyber event differs from incident investigation commissioned after discovery. Whether a particular expense qualifies depends on the actual wording, the nature of the expense, and the circumstances. Likewise, a regulatory obligation to conduct assessments does not create insurance coverage for the consequence of failing to comply.

Risk differentiation and fairness

A test requirement applied uniformly in administrative terms can produce unequal access costs. The cost of external testing may be a significant share of a small organization's annual premium while a large business already commissions testing for independent reasons. Conversely, complex organizations may require more extensive assessments because the number and diversity of business-critical functions are larger. Fair differentiation requires attention to whether evidence demands are proportionate to the risk decision and whether less costly reliable alternatives exist.

An applicant's absence of a particular report can arise from budget constraints, organizational age, reliance on a provider, or a recent change in systems. The insurer may reasonably consider the uncertainty resulting from missing information, but the uncertainty is not itself a confirmed technical defect. Encoding missing evidence as the worst possible control state can exaggerate the apparent relationship between risk and the availability of documentation. A separate missingness indicator and review process can retain this distinction, although validation is necessary before any statistical adjustment is relied upon.

The economic consequences extend beyond one insurer. Repeated assessments demanded by multiple market participants can create duplicated costs and disclosure risks. Controlled reuse of properly scoped evidence may reduce duplication. A shared report can still be interpreted differently by insurers with differing policy forms and portfolios. Thus standardizing evidence fields can improve comparability without requiring uniform underwriting judgment or a single industry risk score.

Evidence Validation and Sensitivity Analysis

Study design and measurement protocol

A credible empirical investigation requires a cohort that can answer the proposed question. For predictive validation, the study population might consist of all applications to a participating insurer in a defined period, with information available at a prespecified underwriting date. For analysis of claims after binding, the study needs exposure periods and policy terms. For causal evaluation of remediation, it needs the dates when findings were identified, corrected and verified, together with a design addressing selection and time-varying confounding.

An observation table should use the legal insured or applicant as an explicitly linked entity, a time-stamped assessment identifier, the tested business function, the objective, assessment status, finding and validated remediation dates, and the known limitations of scope. A separate policy table should preserve policy periods, coverage forms, limits, retentions, and changes. An incident-and-claim table should retain event and reporting dates, loss type, coverage determinations when available, incurred amounts, paid amounts, and claim development. Linking these tables using verified entity identifiers prevents double-counting subsidiaries and repeated reports.

For actuarial data selection, ASOP No. 23 on data quality (Actuarial Standards Board, 2016) provides a U.S. professional reference concerning the use, review and disclosure of data quality. The relevant lesson is that the suitability of the information must be assessed for the intended work. Raw assessment reports collected primarily for remediation may have limited suitability for a statistical estimate unless the classification and selection processes are documented.

ASOP No. 56 on modeling (Actuarial Standards Board, 2019, effective 2020) addresses intended purpose, model structure, inputs, reliance on others and testing. The relevant actuarial application is an explicit review of whether model outputs can support the chosen underwriting use. Vendor assertions concerning ranking performance do not remove the need to understand the population, outcome and material sensitivities.

Material data gaps should appear as recorded categories. When a report is withheld for confidentiality, the reason matters. When testing was deferred because of a safety concern, the fact is distinct. When the insurer never requested testing, that is a result of underwriting selection. Researchers should establish separate missingness mechanisms before choosing imputation or weighting methods. For example, imputing a common “no findings” value for all missing reports can manufacture an apparent relationship with low loss frequency.

Predictive versus causal studies

A prediction study asks whether test evidence improves estimates of a prespecified outcome using information available at prediction time. A causal study asks whether an intervention changes the outcome under a defined comparison. The difference is operational: a variable such as a serious finding may predict claims even if acquiring the information does not itself change operations. Conversely, a remediation intervention can be beneficial even if a broad test-result category adds little predictive power to a model already containing detailed control information.

An appropriate predictive comparison might start with existing exposure and policy information, then add externally observed features, and finally add test coverage and verified findings. Performance should be evaluated on held-out periods and relevant organizational groupings. Randomly splitting repeated observations from the same insured or affiliated group risks leaking entity-specific information. A model trained on data known after the underwriting date may achieve attractive apparent results without being deployable prospectively.

Outcomes should be evaluated through multiple complementary measures. Classification discrimination describes separation of cases and noncases. Calibration concerns agreement between predicted probabilities and observed frequencies. Precision and recall describe performance under a particular threshold and prevalence. Expected financial consequences connect the statistical predictions with underwriting decisions. A change in discrimination need not improve calibration or expected profit, and a change in profit may reflect new contract terms rather than more accurate technical measurement.

For a synthetic unweighted claim prediction example, suppose predicted probabilities for four equal exposure units are 0.01, 0.03, 0.07 and 0.09, with realized outcomes 0, 0, 0 and 1. The Brier score is the mean squared prediction error: (0.0001 + 0.0009 + 0.0049 + 0.8281)/4 = 0.2085. A constant prediction of 0.05 for the same four outcomes gives (3 times 0.0025 + 0.9025)/4 = 0.2275. The hypothetical variable predictions score marginally better in this tiny sample. Four observations provide no reliable inference regarding calibration or comparison in a real population; the example simply states a reproducible scoring calculation.

Causal analysis requires defining treatment correctly. “Had a test” is a composite of different scopes, times, and reporting practices. “Remediated a verified condition by a specified time” is more precisely defined but remains selected. A difference-in-differences design around a policy change or test scheduling program might be investigated if a defensible comparison group and parallel-trends assumption exist. Neither an arbitrary calendar change nor a regulatory requirement automatically supplies a valid natural experiment. The possibility of simultaneous changes in control investments, insurer terms and threat conditions requires careful examination.

Follow-up must begin after the relevant intervention definition. If an organization is classified as having completed remediation because it eventually completed a correction during the policy term, and the entire term is counted as treated, observations from before remediation are misclassified. If an incident causes early termination or accelerates an assessment, time ordering becomes especially important. A study can address these issues using time-varying covariates, defined risk intervals and explicit censoring conditions, depending on its design.

Dependence and coverage uncertainty

Eling and Wirfs (2019) use operational-risk loss records in their cyber event cost study and, as the published abstract indicates, examine ordinary and extreme losses. The evidence supports distinguishing loss magnitude and extreme-event behavior from simple event counts. Operational-risk records do not necessarily represent claims under specified cyber insurance terms. A researcher should avoid merging the two outcome definitions without a contract transformation.

The NAIC's 2026 cyber market announcement reports 56,323 reported claims for 2025, a 20% increase. This aggregate is relevant to scale and reporting, while it supplies no exposure breakdown by test method or remediation status in the announcement examined. It cannot yield a penetration-testing-specific claim probability. It also does not give the count of claims that would have occurred in an uninsured population. The data denominator and category must accompany any numerical comparison.

Portfolio dependence adds a further validation problem. Consider 100 insureds, each facing probability 0.02 of a $100,000 payment. Under independent Bernoulli events, expected aggregate annual payment is $200,000 and variance is 100 times 0.02 times 0.98 times $100,000 squared, or $19.6 billion in squared dollars. Under an alternative model in which all insureds incur the same payments in a common shock with probability 0.02, expected payment remains $200,000 while variance becomes 0.02 times 0.98 times $10 million squared, or $1.96 trillion in squared dollars. The variance ratio is 100. These stylized models hold marginal payment probabilities equal and change only dependence.

A program of individual tests may reveal valuable evidence about shared dependencies if the relevant business and technology relationships are recorded. It cannot establish independent risk simply because each entity was assessed separately. An insurance portfolio may concentrate exposures in a provider, technology architecture, or common control configuration, even when reports refer to different named insureds. A coverage-weighted individual test metric and a portfolio dependence metric answer different questions.

Sensitivity testing can vary three uncertain elements separately: the estimated relation between findings and loss probability, the financial effects of contract terms, and assumptions concerning common shocks. Keeping these parameters distinct makes it possible to explain why an underwriting decision is sensitive to particular assumptions. A model that compresses all three into one numerical score offers little insight when its predictions change.

Source quality and research limits

The source corpus contains original technical guidance, insurer-issued forms and specimen wording, official supervisory instruments, professional standards, and scholarly research. Source relevance differs by proposition. The NIST publication supports methodological distinctions; the forms establish actual examples of questions; the wording illustrates coverage construction; the empirical articles support bounded observations about prediction or loss analysis; the actuarial standards inform professional validation. Treating any one of these as sufficient for all propositions would exceed its evidentiary scope.

The EU regulatory direct page did not render in this research environment; statements rely on official indexed excerpts and the ECB's directly accessible announcement. PDF screenshots were attempted for all analyzed documents; some insurer form and supervisory screenshots were unavailable, so claims are restricted to clearly retrieved text. The cost study was accessible as an abstract; the external forecasting results are likewise taken from the original conference abstract. These limits prohibit detailed assertions about unseen methodology or findings. No confidential test report, proprietary insurer acceptance data, matched claim cohort, or issued insured-specific policy was obtained.

The numerical examples in the preceding chapters are intentionally synthetic and their assumptions are stated. They do not provide parameter estimates. The existing evidence cannot establish the average causal reduction in paid cyber claims from penetration testing, nor a universal optimal test frequency. Such quantities would require a different empirical design and access to matched exposure, test, mitigation, contract, and claims records.

Integrated Scientific Synthesis and Research Agenda

The relationship of engineering evidence to the insurance decision

The scientific contribution of this research is an integrated interpretation of penetration-testing evidence in insurance, based on known distinctions and original worked examples. Its central conclusion is that the evidentiary strength of a test result depends on the question for which it is used. The same report can be strong evidence about a particular control, moderate evidence about the tested function's risk, and weak evidence about annual insured aggregate loss. The interpretation changes because the latter questions involve additional conditions.

The underwriting process can therefore distinguish a sequence of well-defined propositions. First, a technology observation belongs to a specified entity, date and environment. Second, an authorized assessment provides evidence about an identified objective under documented conditions. Third, business exposure converts the objective into a potential financial consequence. Fourth, incident dynamics determine whether and how that consequence could arise. Fifth, the insurance wording determines the indemnifiable component. Finally, the portfolio's dependence structure determines how individual payments interact. The sequence is an explanatory synthesis of established insurance and security concepts; no named framework or new theory is instituted.

One implication is that the information gained from testing should be recorded before any underwriting action or remediation is applied. This distinction supports later investigation of the predictive contribution of testing, the effect of correction, and the effect of contract selection. Without it, an apparent improvement might arise from removing weak risks from the insured portfolio rather than improving their security. A different apparent improvement could reflect a lower limit or greater deductible rather than fewer incidents. Each effect has different economic and societal implications.

Another implication is that coverage information can be as valuable as the finding itself. An insurer that knows the scope of a test can identify unexamined material business functions and ask what other evidence supports an estimate. The report's negative findings can be interpreted within that scope. Without coverage information the apparent reassurance offered by silence may be greater than the evidence warrants. A narrow report with precise scope can support a more defensible decision than a broadly described exercise with ambiguous provenance.

Loss prevention and risk financing

The economic purpose of underwriting risk engineering encompasses both the selection and financing of risk and the possibility of reducing it. A testing program commissioned for selection concerns the insurer's uncertainty about the loss process. A program integrated with corrective action may alter that process. A program intended to demonstrate a regulatory or contractual requirement concerns evidence of performance. The three purposes can coexist, but the costs, benefits and relevant success criteria differ.

Risk prevention has value when correction changes the probability or magnitude of unwanted outcomes. Financing has value when the contract absorbs specified consequences and stabilizes an organization's financial position. A policyholder's recovery may be improved by a tested restoration process even where reporting or coverage terms mean the insurer does not pay for the underlying deficiency. Similarly, the insurer may improve its selection accuracy without changing aggregate security in the wider economy. Evaluating only insurer claims could miss those effects.

The distinction matters for public policy. Where smaller firms are excluded because they cannot obtain costly reports, the insurer may reduce uncertainty within its own portfolio while the economy retains a larger uninsured exposure. Where inexpensive, well-scoped evidence is accepted, access to coverage may improve without necessarily weakening underwriting quality. Whether either outcome occurs is an empirical question. The research supports identifying the mechanisms and measurement requirements, not asserting a universal optimal policy.

Research priorities

A high-value empirical project would collect time-stamped testing and remediation records across a defined application cohort, including approved, declined and withdrawn applications. It would link those records to contemporaneous exposure measures and preserve offered policy terms. The initial analysis could evaluate evidence coverage and missingness. A second analysis could estimate incremental predictive value conditional on prespecified business characteristics and coverage. A third analysis could investigate remediation effects only where an adequate identification strategy is available.

A complementary data project would compare assessments conducted under different objectives using the same reference conditions in consenting environments. Such a study could measure agreement on narrowly defined findings, evaluate interpretation of uncertain cases and explore how test scope affects the informativeness of negative results. It would protect confidential data and avoid any unauthorized examination. Its conclusions would concern the sampled conditions and tested procedures rather than all possible methods.

A third project would compare the predictive significance of external data collected before test selection with the incremental significance of validated test observations. It would preserve the versions of both sets of signals, avoid retrospectively selecting only successfully identified findings, and distinguish newly observed facts from subsequently changed configurations. The comparison would answer a question essential to underwriting economics: what additional information does authorized testing supply after considering other available evidence, and at what cost?

A fourth project would link assessed business functions to insurance contract and event classifications. It could compare the relevance of test evidence for first-party interruption, liability, and response expenditure, while recording dependencies affecting multiple insureds. This would help determine where technical data inform frequency, severity, or aggregation rather than assuming a homogeneous cyber outcome.

The international research agenda requires preserving local legal and market context. Regulatory testing of financial entities, voluntary commercial testing, insurance application questions and mandatory policy conditions each arise in different institutional arrangements. A comparative study should define the relevant subject population, effective instrument, form version and outcome before constructing a pooled comparison. Such precision supports international scientific synthesis without erasing jurisdictional distinctions.

Practical assessment of information quality

A research or underwriting record can be reviewed by asking whether the report established the assessed entity, documented authorization and restrictions, used a scope relevant to the insurance question, supported material findings, distinguished untested from tested conditions, recorded remediation and verification, and retained temporal and financial context. The record can then be evaluated against the decision for which it is intended. An underwriting choice with large capital consequences requires stronger evidence of the proposition than a preliminary triage question.

This review also makes clear where an apparent conflict belongs. A disagreement about ownership is resolved through attribution evidence. A disagreement about test accuracy is resolved through methodological review. A disagreement about likely loss requires actuarial evidence. A disagreement about indemnity requires interpretation of the issued contract in light of the incident and applicable law. Treating all four conflicts as disputes about the same score obscures the appropriate remedy.

The best-supported scientific position is conditional rather than categorical: penetration testing can supply material information about security controls and may trigger beneficial prevention when its findings lead to effective corrective action. Its relationship to insured claims is a distinct empirical object. The analytical contribution is the explicit tracing of that relationship through exposure-weighted test scope, observation reliability, selection, remediation, insurance terms and portfolio dependence.

Conclusion

Penetration testing and external attack surface data intersect at the point where observable technology conditions become the object of bounded, authorized security assessment. Their relevance to cyber insurance depends on the reliability and scope of the resulting evidence, the business functions exposed, the actions taken in response, and the contract defining the financial obligation. A reliable report can strengthen underwriting without independently establishing the probability of a claim; a corrective intervention can change loss experience without the report itself constituting a priced exposure. The distinction is essential to interpretation.

Measurement reliability and test coverage must be represented separately. Selection into testing, application, binding and renewal must likewise be distinguished from a genuine mitigation effect. The quantitative demonstrations establish that varying prevalence, sample composition, contract structure and portfolio dependence can materially change an insurance inference while holding other conditions fixed. No synthetic parameter is presented as observed market data.

The documentary comparisons illustrate that assessment evidence enters actual insurance information processes and that regulatory testing frameworks have specific institutional subjects. Contractual payment requires its own evidence, including the applicable policy and facts of the event. The resulting research supports careful use of test evidence, explicit validation, proportionate assessment and documented uncertainty. It also identifies the empirical prerequisites for future quantitative study: matched longitudinal testing, exposure, remediation, contractual and claims data. The inquiry is complete within its documentary and analytical scope; it leaves the estimation of causal effects and validation of particular vendor systems to empirical research capable of identifying those quantities.

References

  • Scarfone, K., Souppaya, M., Cody, A., & Orebaugh, A. (2008). Technical Guide to Information Security Testing and Assessment. NIST Special Publication 800-115. National Institute of Standards and Technology. doi:10.6028/NIST.SP.800-115. NIST technical guide

  • Romanosky, S., Ablon, L., Kuehn, A., & Jones, T. (2019). Content analysis of cyber insurance policies: how do carriers price cyber risk? Journal of Cybersecurity, 5(1), tyz002. doi:10.1093/cybsec/tyz002. Cyber policy content analysis

  • Liu, Y., Sarabi, A., Zhang, J., Naghizadeh, P., Karir, M., Bailey, M., & Liu, M. (2015). Cloudy with a Chance of Breach: Forecasting Cyber Security Incidents. 24th USENIX Security Symposium, 1009–1024. Conference paper

  • Forum of Incident Response and Security Teams (FIRST). (n.d.). Using EPSS. Exploit Prediction Scoring System. Accessed November 18, 2026. EPSS documentation

  • Emergence Insurance Pty Limited. (2024). Cyber Enterprise Solution Proposal Form. CES Proposal Form 0124, p. 6, Assessments. Enterprise proposal form

  • CFC Underwriting Limited. (2019). Cyber Private Enterprise: Insurance Application Form, AUS. Copyright 1999–2019. Cyber security controls section. Private enterprise application

  • Emergence Insurance Pty Limited. (2025). Cyber Enterprise Solution CES 003 AUS: Important Information and Policy Wording. Published October 2025, p. 5, Sections A–C. Enterprise policy wording

  • New York State Department of Financial Services. (2023). Cybersecurity Regulation, 23 NYCRR Part 500, as amended November 2023, § 500.5. New York cybersecurity regulation

  • European Parliament & Council of the European Union. (2022). Regulation (EU) 2022/2554 on digital operational resilience for the financial sector. Official Journal L 333, 27 December 2022, 1–79, Arts. 24–27. EU regulation

  • European Central Bank. (2025, February 11). TIBER-EU Framework updated to align with DORA. Market Infrastructure and Payments news. TIBER-EU update

  • Actuarial Standards Board. (2016, December). Actuarial Standard of Practice No. 23: Data Quality. Revised standard. ASOP 23

  • Actuarial Standards Board. (2019). Actuarial Standard of Practice No. 56: Modeling. Effective October 1, 2020. ASOP 56

  • Eling, M., & Wirfs, J. (2019). What are the actual costs of cyber risk events? European Journal of Operational Research, 272(3), 1109–1119. doi:10.1016/j.ejor.2018.07.021. Article abstract

  • National Association of Insurance Commissioners. (2026, October 21). NAIC Report Finds Cyber Insurance Market Has Nearly Tripled Since 2020 as Cyber Risk Grows. Press release on 2025 calendar-year report. NAIC market report release