RESEARCH DOSSIER
Probability and Risk under Limited Information
Abstract. Probability estimates are used to classify exposures, price insurance, select loss controls, assess capital and represent possible catastrophes. Their validity depends on the relationship between a defined event, the target population, the observation process, and the assumptions that connect recorded outcomes to future risk. This monograph examines probability and risk under limited information through sparse counts, zero observed events, unequal exposure, dependence, censoring, delayed reporting, selective observation, missing data, partial identification, Bayesian updating, expert elicitation, model uncertainty and structural change. Exact binomial and Poisson calculations are developed and distinguished from posterior predictive probabilities, scenario estimates and decisions under ambiguity. Original analytical examples show why more recorded observations do not necessarily correct systematic selection, why zero losses cannot establish zero risk, and why model precision may exceed the information available about tail events. The principal conclusion is that every defensible risk probability must specify its event, horizon, source population, observation mechanism, identification assumptions, inferential meaning and sensitivity to alternative plausible premises. The contribution is an integrated critical synthesis and independently derived examples, without a new named theory, doctrine or model.
Keywords: risk probability; limited information; sparse data; zero events; binomial inference; Poisson rates; exposure; censoring; truncation; missing data; selection bias; partial identification; Bayesian updating; expert elicitation; model uncertainty; distribution shift; actuarial data quality; insurance; uncertainty communication.
Introduction and Scope
An annual probability of a damaging flood may be a frequency estimate, a fitted model output, a posterior belief, or a judgment derived from physical mechanisms and historical analogues. Identical numbers may consequently express very different bodies of evidence. In insurance, where a small change in event probability can interact with considerable financial severity and dependence, the difference is material. A claim of numerical precision is credible only when the underlying process of observation and inference can be examined.
Knight's (1921) analysis of risk and uncertainty provides an important historical distinction between calculable probabilities and forms of uncertainty resistant to measurement. Modern methods represent some uncertain judgments probabilistically, while retaining the requirement that their assumptions be justified. Kaplan and Garrick's (1981) scenario-based interpretation of risk connects possible events, probabilities and consequences. This dossier investigates the evidentiary status of the probability component and the consequences of weakness in its foundations.
The scientific object is the warrant for assigning probability to adverse events with incomplete observations. The central research questions concern the effect of sparse samples, mechanisms by which records become visible, and the inferential limits of assumptions, analogies and expert judgment. The treatment is international and mathematical. The worked cases are constructed illustrations rather than estimates drawn from actual insurers or jurisdictions.
A useful distinction separates three levels. Observation supplies selected records of exposures and events. Inference uses a stochastic or structural model to connect these records to an underlying event law. Decision analysis converts the resulting uncertainty into choices with potential costs, losses and distributions of consequences. The three levels must be traceable to one another if a scientific probability is to inform a consequential action.
Probability, Evidence, and the Target Event
Let A denote an event, P a population, T a time horizon, C a set of exposure and operational conditions, and I the available information. A probability statement is adequately specified as Pr(A | P,T,C,I). In practice, the stated quantity must also identify whether it is a true parameter within an assumed model, a fitted point estimate, a posterior probability, or a prediction for a new exposure. A rate per policy-year cannot be silently presented as a probability per policy, and a claim count cannot be treated as an incident count when one incident produces multiple claims.
In repeated Bernoulli sampling, x/n estimates p when n trials share the relevant event definition and distribution. Independence and identical distribution are assumptions about the underlying process. The estimate's uncertainty is not exhausted by numerical standard error because records may omit events or concentrate exposure in shared hazards. Under the observation indicator O, the probability Pr(A | O=1) generally differs from Pr(A); a narrow interval for the first need not estimate the second.
The separation of frequency and financial consequence is equally necessary. An expected insured loss requires a severity distribution conditional on an event as well as an event rate. Quantile loss, ruin probability and liquidity exposure additionally depend on tail shape, contract terms, dependence and the insurer's available resources. A defensible estimate for one component does not validate the entire aggregate risk assessment.
Probability also depends on the conditions under which future events are generated. A historical frequency may be informative for a stable manufacturing process and poorly representative of a new cyber architecture or a changing climate. The relevant prediction concerns a future target population, not simply the recorded past. The ISO 31000 risk-management guidelines provide an institutional frame for integrating analysis with decision making, monitoring and communication; they do not by themselves certify any numerical probability.
Sparse Frequencies and Exact Uncertainty
Reference classes, frequency, and transportability
Before counting trials, the analyst must define the reference class. A population of insured drivers might be segmented by geography, vehicle, mileage, age, driving environment or policy period. Such choices affect denominators and the relation between past and prospective experience. Broad aggregation increases count information but risks pooling non-comparable hazards; narrow segmentation improves comparability but yields sparse observations. The appropriate classification depends on the question and the causal or predictive mechanisms, rather than solely on the number of observations available.
An exposure count is also conditional on observation completeness. A hundred policies that began coverage in January and ended in December are not equivalent to a hundred policies entered at different times and valued in March. The analyst should preserve occurrence dates, start and end of coverage, censoring and changing risk conditions. A probability per contract may differ from a hazard rate per year if policies have different durations or repeat events.
There is an unavoidable distinction between sampling more deeply from a biased source and sampling more broadly from the relevant population. If a broker reports only large claims, doubling the broker's submissions improves precision for reported large claims while retaining selection relative to all claims. Data richness is consequently multidimensional: relevance of the reference class, quality of exposure, diversity of mechanisms, completeness of detection and duration of observation matter alongside record count.
Likelihood information beyond the point estimate
A point estimate compresses a whole likelihood function into a single value. For zero events, the likelihood ratio relative to its maximum is (1-p)^n. At n=100, a proposed p=0.01 has relative likelihood 0.99^100, approximately 0.366; p=0.05 has relative likelihood 0.95^100, approximately 0.00592. These calculations reveal how a zero count shifts relative support among proposed probabilities while stopping short of assigning posterior probabilities to them.
Likelihood ratios are conditional on the event model and selection assumptions. If individual trials are heterogeneous, the probability of zero events becomes the product of (1-p_i), not the n-th power of one common term. The same observed total therefore supports different conclusions when individual probabilities vary strongly or when events depend on shared hazards. Expressing this structure explicitly prevents a homogeneous approximation from being mistaken for evidence about heterogeneity.
The absence of observed events may also be the most difficult situation for model criticism, because there is little observed variability with which to test distributional assumptions. A model that makes extreme danger nearly impossible and a model that makes it rare but consequential can fit zero-event data similarly well. Substantive mechanistic knowledge and appropriate comparison data must then carry a larger share of the scientific assessment.
The meaning of zero observations
Suppose X is binomial with n independent, identically distributed trials and unknown event probability p. The likelihood is proportional to p^x(1-p)^(n-x). When x=0, the likelihood equals (1-p)^n, attaining its maximum at p=0 while remaining positive for every p below one. An observed zero count therefore favors low probabilities within the model while leaving a range of positive values compatible with the data. Hanley and Lippman-Hand (1983) analyzed the interpretation of zero numerators in this context.
A one-sided 95 percent exact binomial upper confidence bound for x=0 solves (1-p_U)^n = 0.05. Hence
For n=100, the bound is approximately 0.02951, or 2.951 percent. The approximation -ln(0.05)/n gives 0.02996. The approximation commonly called the rule of three is an approximate bound, not a finding that event probability is three percent. The historical Clopper-Pearson confidence-limit method and NIST exact binomial guidance establish why ordinary symmetric normal intervals can be unsuitable for small or zero event counts.
Confidence interval coverage refers to the repeated-sampling properties of the construction under its model. It is distinct from a posterior probability statement about the parameter conditional on the observed sample. Its nominal coverage is also conditional on the defined observation units, independence, and inclusion of all eligible observations. A calculation that is exact for an assumed binomial experiment can be inapplicable to the actual administrative process.
Small denominators and correlation
An insurer may hold one hundred policies while the main hazard is a single shared cloud provider or floodplain. The informative number of independent hazard realizations can then be far smaller than the policy count. For equally sized clusters of m observations and approximate within-cluster correlation rho, a standard diagnostic design effect is 1+(m-1)rho; the effective sample size is approximately n divided by that factor. This diagnostic is not itself an exact rare-event model and cannot substitute for a specified common-cause mechanism.
If only twenty event-generating clusters are genuinely independent, zero observations produce the analogous upper bound 1-0.05^(1/20), approximately 0.1391. Contrasting it with the 2.951 percent bound for one hundred independent events illustrates how much inferential force is supplied by the independence assumption. It does not justify replacing the true dependence model with an arbitrary effective denominator.
What resampling can recover
Efron's (1979) bootstrap resampling analysis estimates sampling behavior using observed data. An ordinary empirical bootstrap of a sample containing only zeros will contain only zeros. The method cannot invent an unobserved catastrophe class or missing reporting channel. Parametric or hierarchical simulation can assign positive probabilities to unobserved events, but the information then enters through model structure or prior assumptions rather than through the absent observations.
Exposure, Event Counts, and Dependence
From individual hazards to aggregate claims
An insurance loss process is more complex than an event count. If N is the number of covered claims and X_i are severities, the aggregate loss is S=sum from i=1 to N of X_i. Under a simple independent compound Poisson model with rate parameter lambda E and independent identically distributed severities, E[S]=lambda E E[X]. This identity is a modeling result, not an empirical shortcut. If one catastrophe generates many claims, N may represent claim multiplicity conditional on an event rather than the number of independent hazard events.
For this reason, the choice of modeling unit matters. A catastrophe model can distinguish event occurrence, affected geographic footprint, asset values, vulnerability, coverage and loss settlement. Counting claims alone often confounds all these layers. A region with more insurance penetration may record more claims even where the physical occurrence rate has not changed; a stronger deductible may lower recorded paid-claim frequency without altering hazard occurrence.
Mixtures also alter count uncertainty. Suppose lambda itself varies by year according to changing hazards or exposure conditions. Conditional on lambda, counts might be Poisson, while unconditional counts exhibit variance greater than their mean. A model that attributes all this variation to independent Poisson noise may underestimate uncertainty in next year's count and in the capital needed for correlated large losses.
Tail information and annual exceedance
Let L denote aggregate annual insured loss under a specified portfolio and model. The annual exceedance probability at threshold z is Pr(L>z). A return-period label of 1/p years is only a restatement of the modeled annual probability p when the annual distribution is considered stationary. It is a property of the stochastic model, not a schedule of future disasters. If exposure or hazard changes, the annual exceedance distribution changes as well.
A sparse record may contain enough information for an approximate mean but very little about an extreme quantile. Even if an empirical claim rate is well estimated, the large-loss tail may be driven by data outside the observed record, insured aggregation across locations or changes in claim settlement. In that setting, decision analysis should state the role of structural assumptions and plausible severe scenarios rather than convert simulated precision into historical certainty.
For an assumed homogeneous Poisson count process with rate lambda per exposure unit, total exposure E gives N following Poisson(lambda E). With zero observed events, Pr(N=0)=exp(-lambda E), yielding a one-sided 95 percent upper limit
For E=1,000 policy-years, the bound is approximately 0.002996 events per policy-year. This is a frequency rate with a specified unit. Under the Poisson model, the corresponding one-year probability of at least one event is 1-exp(-lambda), which approaches lambda when lambda is small. Such conversion requires constant intensity and the relevant process assumptions.
Exposure normalization must follow actual risk-bearing time. One policy active for three months contributes one quarter of a year of exposure if the contractual and statistical unit is policy-year. A recent cohort with only partial development contributes less observable event time. Converting counts to annual rates without effective exposure and lag can create apparent improvements or deteriorations that reflect recording dates rather than hazard changes.
A nonhomogeneous intensity lambda(t) produces the zero-event probability exp(-integral from 0 to T lambda(t)dt) under the specified process. A stable annual rate is therefore one special case. Seasonality, trend, clustered catastrophes, and variation in insured location or occupancy demand a more explicit hazard specification.
Dependence matters for severity as well as counts. Two exposures with the same marginal probability can have very different aggregate loss distributions when their events occur together. The financial effect of a shared storm, system outage, legal judgment or infrastructure failure cannot be recovered merely by observing many policy records. A common-cause model, appropriately bounded scenarios, or explicit dependence sensitivity is needed to support an accumulation statement.
Censoring, Truncation, and Delayed Reporting
Observation time and the estimand
Survival analysis clarifies why raw counts are often insufficient. For an individual with event time T and censoring C, observed follow-up Y=min(T,C) records how long that individual remained under observation. If an event occurs at time t, its density contribution under a survival model is related to a hazard at t and survival to t. For a censored record the contribution concerns survival at the censoring time. This distinction is lost when a censored policy is recorded simply as having no claim.
The independence conditions for censoring are substantive and often conditional. Withdrawal from follow-up may depend on measured factors, such as age of an insured item, in which case stratification or adjustment may be possible. It may depend on unobserved deterioration, financial distress or anticipated loss, in which case the effect may be difficult to identify. The phrase noninformative censoring must be explained relative to the variables and inferential target.
Interval censoring introduces a further complication when the event is known to have occurred within a period but its exact timing is unavailable. Warranty failures, latent injuries, water intrusion and some cyber compromises may be discovered only through later inspections. Treating inspection date as occurrence date changes duration estimates and may create artificial clusters around reporting practices.
Development data and valuation
Insurance data often include separate accident, report and payment triangles. Each triangle answers a different question, and its latest cells are typically immature. A reliable probability estimate should be consistent with the contractual event definition and the date at which claims are valued. Ultimate claim probability requires a considered projection of unreported and unsettled events, including the possibility that legal or administrative development changes over time.
Changing business volume affects what is observable at each development duration. For example, an insurer expanding rapidly into a new class may have many recently written contracts but little mature experience. Apparent improvement in paid loss frequency could reflect shorter maturation and higher deductibles. The analyst must disentangle policy mix, coverage changes, development and underlying incidence.
Where cause-of-event classification is revised during settlement, a single physical loss may move between accounting categories. Versioned data and a reproducible mapping from historical codes to target categories are important. This is especially salient when the same event interacts with primary coverage, deductibles, retentions and reinsurance contracts.
An event-time record is right-censored when observation ends before the event is known to occur. Writing T for the event time and C for the censoring time, the recorded variables are Y=min(T,C) and delta=1{T<=C}. A censored record contributes evidence that the exposure survived through Y; it does not establish that the event will never occur.
Kaplan and Meier's (1958) product-limit estimation method uses event counts and risk sets to estimate survival with incomplete follow-up. Its standard interpretation relies on an appropriate condition of noninformative censoring. If policies with mounting loss risk leave follow-up earlier, a naive product-limit curve can overestimate survival. This assumption should be assessed with substantive knowledge about exits, withdrawals, cancellations and changes of coverage.
Cox's (1972) proportional-hazards regression formulation uses a conditional likelihood for covariate effects with an unspecified baseline hazard. A fitted hazard ratio does not alone establish absolute future event probability, and the basic proportional-hazards assumption must be considered when risk effects vary with time. Neither survival technique automatically restores outcomes that are never eligible for entry into the record.
Left truncation occurs when inclusion requires surviving until observation begins. A dataset of facilities remaining operational after a decade systematically omits facilities that failed earlier. Right censoring concerns unfinished follow-up after entry; left truncation concerns conditional entry. Their likelihood and risk-set treatments differ.
Claims often emerge through accident, reporting, adjudication and settlement stages. A claim arising during the observation window but reported years later is absent from the current file. A recent underwriting period may look unusually safe simply because its reporting process is immature. Analyses must identify accident versus report date, reporting delays, claims development, contract terms and valuation date. Changing legal and administrative practices can modify the delay distribution independently of underlying event probability.
Missingness and Selective Observation
A causal view of observation
The probability that a record is included may depend on outcome A, observed variables X, and latent variables U. A schematic selection process can be written Pr(O=1 | A,X,U). If selection depends on A even after conditioning on X, the observed event rate may diverge systematically from the target rate. If U influences both A and inclusion, adjustment using X alone may fail. These possibilities must be assessed through domain knowledge and external information, not judged only from predictive accuracy among observed records.
An observed covariate can sometimes make selection conditionally ignorable. Suppose claims from smaller insurers are less likely to be recorded but insurer size is known for the complete population and sufficiently explains selection. Weighting by inverse inclusion probabilities may recover selected population quantities under positivity and the specified conditional assumptions. If certain subgroups have zero probability of inclusion, weighting cannot reconstruct their unknown outcomes from the data. The support condition is as important as the algebra.
Misclassification and the detection process
An additional limit arises when events are recorded but mislabeled. Let Se be the sensitivity of the event detector and Sp its specificity. If true event prevalence is p, the observed positive fraction q satisfies q=Sep+(1-Sp)(1-p). When Se and Sp are known, and Se+Sp>1, p=(q+Sp-1)/(Se+Sp-1). This correction is a direct consequence of total probability. When detection accuracy is itself uncertain, estimates of p inherit that uncertainty.
Rare-event studies are particularly sensitive to false positives. If event prevalence is very small, even high specificity can produce a substantial share of false detections among recorded positives. Conversely, low sensitivity can hide materially important events. A study claiming that an advanced reporting or automated classification system improved detection should distinguish changes in true incidence from changes in sensitivity, specificity and case definition.
The observation mechanism should be part of the data dictionary and scientific report. Documenting how loss records are opened, amended, merged and excluded makes the probability claim auditable. This is not a merely technical matter: changes in how claims are registered may alter premiums, reserves, capital estimates and policyholders' access to protection.
Rubin's (1976) foundational missing-data analysis describes conditions under which an observation mechanism may be ignored for particular likelihood or Bayesian inference. Missing completely at random, missing at random conditional on observed data, and mechanisms dependent on unobserved outcomes are different scientific propositions. The label missing at random is an assumption relative to a specified dataset and model, rather than a property inferable simply from many available covariates.
Let A be an adverse outcome and O its inclusion in a database. Bayes' rule gives
Suppose Pr(A)=0.10, the inclusion probability for adverse outcomes is 0.80, and for non-adverse outcomes it is 0.20. Then Pr(O=1)=0.08+0.18=0.26. The event fraction among observed records is 0.08/0.26=0.30769, or about 30.8 percent, even though the target-population event probability is ten percent. A large sample will estimate the wrong target with increasing precision if the analyst mistakes the selected conditional probability for the unconditional population probability.
Heckman's (1979) study of selection bias shows that selection can cause specification error and that corrections require additional assumptions. Insurance applications include claims disproportionately visible above a reporting threshold, digital records available only for recently enrolled customers, settlements absent from certain systems, and loss data contributed by insurers with different reporting incentives. Model correction needs knowledge of selection probabilities or credible structural restrictions, not only a more complex predictive algorithm.
Record linkage, duplicate claims, coding changes, incomplete event definitions and missing insured-value fields can produce additional selection and measurement errors. Documentation of data provenance should specify which records are missing and why, how exposure denominators are reconciled, and whether policy cancellations or claims closures differ across risk classes.
Identifiability and Probability Bounds
Identification by auxiliary information
External evidence may partially bridge missing outcomes. An independent audit sample, a validated registry, an engineering failure catalog or a prospectively monitored cohort can supply information about inclusion probabilities or unobserved severity. This evidence must be assessed for transportability: an audit of large urban firms may not identify the missing event rate among small rural firms without additional assumptions.
Bounds are often more scientifically informative than unqualified point estimates. A point estimate may be reported alongside an identified set when a central scenario is useful for operational planning, provided that the distinction is visible. The identified set states what the data and restrictions permit; the central scenario states one selection among possibilities.
Robustness of inequalities
The simple missing-outcome bounds can also be translated into financial consequences. Suppose an insured event causes a fixed loss of 100 monetary units and the probability interval is [0.06,0.46]. Then expected event loss lies between 6 and 46 units under the stated fixed-severity premise. If severity itself is unknown, a bound on frequency alone is insufficient to bound expected loss without severity constraints.
With a contractual policy limit M and nonnegative covered payout Y, expected indemnity may be bounded above by M times the probability of an indemnifiable event, if every event generates no more than M. This upper bound can be wide, but it is still an interpretable consequence of a contract and a probability bound. Claims about net losses require further terms for deductibles, limits, exclusions, recoveries, expenses and timing.
An impossibility result is also a research result. When credible restrictions do not identify risk tightly enough for a decision, the report should say so and explain which future observations or institutional changes could make the question more tractable. Pretending the parameter is identified merely because a software package returns a number erases a central scientific finding about the information available.
A parameter is point identified when observable evidence combined with the assumptions determines one value. It is partially identified when the same observable evidence permits a set of values. This distinction should be expressed directly rather than hidden by optimization that always returns a single numerical estimate.
Suppose the proportion r of target-population outcomes is observed and the event fraction among these is q. Let the unobserved portion have unknown event probability q_m between zero and one. By total probability, p=rq+(1-r)q_m. Therefore
With r=0.60 and q=0.10, the evidence permits p between 0.06 and 0.46. The bounds are attainable by taking missing-group probability equal to zero or one and are consequently sharp under the stated restrictions. Manski's (1990) partial-identification research illustrates the broader scientific use of bounds; the specific accounting example here is independently derived.
If an independent audit justifies 0.05<=q_m<=0.15, the bound becomes 0.08<=p<=0.12. The narrower interval is explained by additional evidence or a substantive restriction. Its apparent precision must not be presented as coming solely from the originally visible records.
Analogous identification problems affect tail severity. A historical dataset whose largest recorded loss is ten million does not establish that higher loss is impossible. A physical cap, contractual limit, fitted heavy-tail distribution and expert worst-case scenario are distinct grounds for different claims. Tail extrapolation should identify where observations stop and assumed structural behavior begins.
Bayesian Updating and Prior Sensitivity
Posterior uncertainty and model specification
Bayesian sensitivity extends beyond choosing Beta priors. The likelihood may assume exchangeability of trials, a stable baseline hazard, a distribution for heterogeneous class effects or a pattern of dependence. Under data scarcity, reasonable alternative likelihoods may yield predictions farther apart than alternative priors under a fixed likelihood. Both sources should be inspected.
A useful practice is to distinguish a prior distribution on an unknown parameter from a prior predictive distribution on possible future observations. The latter reveals what the model anticipated before seeing the current sample. If an alleged weakly informative prior implies implausibly frequent catastrophes or virtually forbids events that physical mechanisms permit, its role should be reconsidered.
Posterior summaries should correspond to their decision use. A posterior mean event rate may be appropriate for a particular expected-loss calculation, while a high posterior quantile of aggregate loss may be relevant for a capital decision under a more complex severity and dependence model. Neither should be substituted for a frequentist confidence limit without explanation. Posterior predictive evaluation should ideally use genuinely later, appropriately grouped observations.
Decision sensitivity to posterior tails
For a threshold decision, expected utility may depend on posterior mass in regions associated with high consequences, rather than simply on the posterior mean of p. Suppose a protective measure makes little difference for ordinary events but greatly reduces the consequences of a rare high-severity event. The analysis must represent event frequency and severity jointly. Concentrating only on mean event frequency can suppress the features that determine a rational intervention.
The scientific report should therefore show how conclusions respond to alternative reasonable priors, plausible structural models and differing assumptions about future conditions. These sensitivity exercises document the strength of the conclusion conditional on incomplete data, rather than manufacturing an appearance of certainty.
For Bernoulli outcomes with a Beta(a,b) prior, observing x events in n trials gives posterior Beta(a+x,b+n-x). The posterior predictive event probability for one future conditionally exchangeable observation is (a+x)/(a+b+n). Its mathematical simplicity should not conceal the substantive assumption that the historical and future trial are linked by one stable parameter.
For x=0 and n=100, the uniform Beta(1,1) prior gives predictive event probability 1/102, approximately 0.9804 percent. Jeffreys' Beta(1/2,1/2) prior gives 0.5/101, approximately 0.4950 percent. These results differ despite identical likelihood information. Their difference is not a contradiction: each conditions on a different prior. Neither is a frequentist 95 percent upper limit.
Berger, Bernardo and Sun's (2009) work on reference priors develops principled prior construction under specified inferential goals. A so-called objective prior remains dependent on mathematical structure and the parameter or question of interest. With few observations, reporting the prior and varying plausible alternatives is essential to the scientific interpretation of posterior results.
Hierarchical pooling across related insurance classes can help stabilize estimates by sharing information. It can also suppress meaningful heterogeneity, particularly when the insured technologies, contracts, hazards or reporting cultures differ. The analyst should distinguish information contained in the class-specific observations from information supplied by the pooling assumptions. Posterior predictive checks, prior predictive checks and sensitivity to alternative grouping structures provide substantive diagnostic evidence.
Expert Judgments and Elicitation
Structuring an elicitation session
A well-specified elicitation begins with a target event that can be judged in the same way by all participants. The event definition should include the insured object, horizon, threshold, geographical and operational scope, and whether the judgment concerns occurrence, detection, claim reporting or financial payment. Experts often disagree because they answer different implicit questions; resolving that ambiguity improves scientific comparability before numerical aggregation begins.
Assessors can be asked for quantiles of a plausible loss parameter, probabilities of defined binary events, or distributions over physically interpretable scenarios. Elicitation protocols should give participants opportunities to explain causal mechanisms and identify conditions under which their judgments would change. Intervals can reveal uncertainty obscured by forced single-number elicitation.
Dependence between experts
Expert opinions should not automatically be treated as statistically independent evidence. Multiple catastrophe specialists may use the same vendor hazard model, and underwriting experts may be influenced by identical loss histories. Recording shared informational ancestry helps assess the extent of genuinely independent judgment. Sensitivity to alternative combination rules is advisable where common-source bias is plausible.
Performance weighting may be possible using calibration questions whose correct answers are known only after judgments have been fixed. Yet calibration questions should reflect the domain and difficulty of the target questions. Weighting experts by performance on easily resolved ordinary events may be weak evidence of ability to forecast an exceptional systemic failure. The validity of the calibration population is itself a transportability issue.
Updating expert assessments
As data arrive, expert distributions can be compared with observations and revised. A revision record should distinguish new evidence from changes in interpretation, incentives or exposure. Such provenance becomes important when an underwriter's prior belief affects model output, insurance availability or a regulatory narrative. Expert judgment is most useful scientifically when it remains both revisable and attributable.
When rare exposures have no meaningful frequency history, structured judgment can incorporate physical mechanisms, near misses, operational experience and analogues. O'Hagan and colleagues' (2006) probability elicitation handbook covers elicitation as a disciplined method. Cooke's (1991) analysis of expert uncertainty addresses elicited probabilities and approaches to assessing and combining expert views.
An elicitation must define the event and horizon before requesting a number. It should reveal the expert's reference class, access to evidence, assumptions about the system, degree of disagreement and uncertainty around each judgment. Calibration exercises on comparable resolvable questions can inform evaluation of a forecaster, although success on such questions does not prove competence for an unprecedented systemic catastrophe.
Gneiting and Raftery's (2007) proper scoring-rule analysis explains how appropriate prediction scores incentivize truthful probabilistic forecasts. Proper scores are useful only when the outcome definition is suitable and the relevant events eventually resolve. Experts drawing on the same records or vendor models may have strongly correlated judgment errors, so simple averaging of multiple opinions should not be treated as independent replication.
Elicitation records should preserve prompts, dates, expertise, information shared, explanations and revision history. Where judgments are used for claims reserves, catastrophe stress or emerging cyber insurance, the evidentiary status of a modeled probability should remain visible to decision makers.
Prediction, Model Uncertainty, and Structural Change
Calibration, sharpness, and event resolution
A probabilistic forecast can be evaluated against outcomes using calibration and scoring tools. Calibration broadly concerns agreement between stated probabilities and realized event frequencies over suitable groups of resolved cases. Sharpness concerns concentration of predictive distributions. A forecast that always states fifty percent may be calibrated in a balanced population while being unhelpful for distinctions within it. Conversely, a highly sharp forecast may be overconfident when rare mechanisms are omitted.
For very rare outcomes, traditional holdout datasets may contain few or no cases of the event. High aggregate classification accuracy or a low overall loss score can be achieved while estimating the rare-event tail poorly. Evaluation should therefore report event-specific sensitivity, appropriate scoring rules, intervals, exposure coverage, and the actual number of resolving events. Comparing forecasts only within a selected claims database can reproduce selection bias.
Prospective validation design
Out-of-time testing asks whether a prediction transfers to later experience. Out-of-region testing examines geographical transfer. Grouping by catastrophe or common infrastructure prevents closely related observations from leaking between development and evaluation datasets. Where future events are slow to resolve, a validation report should explicitly distinguish mature observed outcomes from provisional and model-developed targets.
Backtesting is most persuasive when the forecasting model, event definition, exposure denominator and data cutoff were fixed before outcomes became known. Retrospectively optimizing a threshold or severity definition on the same realized episodes can overstate predictive success. A transparent experiment record should retain unsuccessful tests and material revisions rather than only the ultimately preferred model.
Structural sensitivity
A climate-related loss model can exhibit precise simulated event frequencies under a fixed catalog and uncertain future physical conditions. A cyber risk model can depend on a shared software architecture that changes quickly. In both cases, scenario stress may be valuable even if long-run event probability cannot be measured adequately. Reporting should distinguish stochastic sampling uncertainty within a scenario, uncertainty about model parameters, and uncertainty about the scenario's applicability to the decision horizon.
Prediction asks whether information about past observations applies to a specified future risk. Contract wording, building standards, insured values, digital dependencies, social behavior and climate conditions can shift the data-generating process. The difference between sample uncertainty and structural change is substantial: collecting more historical data from the same regime may not resolve uncertainty about a new one.
Hoeting and colleagues' (1999) Bayesian model-averaging tutorial examines how selected-model certainty can understate predictive uncertainty. Model averaging can represent disagreement among candidate models, but it cannot include plausible event mechanisms that analysts have failed to identify. Model uncertainty and unknown mechanisms should both be declared in a transparent assessment.
A credible validation design should reflect intended deployment. Random splitting can leak information when policies share one catastrophe, underwriting campaign or reporting system. Out-of-time, out-of-region, exposure-clustered or event-blocked validation may give a more demanding estimate of predictive transferability. Long-tail claims also require maturity qualifications: unresolved outcomes are not known negative events.
The IPCC (2021) distinguishes evidentiary confidence from quantified likelihood in its assessment uncertainty guidance. This separation can clarify risk reporting in insurance. A model may assign a very precise annual exceedance probability while scientific confidence in the underlying event catalog or vulnerability function remains limited. Conversely, a direction of hazard change may have substantial evidence even when a future annual financial loss probability cannot be responsibly stated to several decimal places.
Stress scenarios may test resilience when no credible joint probability can be assigned. Their outcomes should not automatically be described as return-period quantiles. Probabilistic interpretation requires an additional distributional premise or defensible elicitation process.
Assumption Sensitivity and Decisions
Sensitivity tests should focus on parameters and mechanisms with leverage over decisions. For a zero-event estimate, relevant perturbations include the true number of independent hazard units, prior specifications, reporting lag, clustering, and target-period stationarity. For a selected claims sample, the observation mechanism and plausible event rate among unobserved exposures may dominate calculation precision.
A decision can remain rational despite incomplete probability. Suppose a control costs c, an event causes loss L, and the control eliminates that event. Under a risk-neutral, one-period expected-loss comparison, treatment is preferred if pL>c. If p belongs to [p_L,p_U], it is unambiguously favored under these assumptions when p_LL>c, and disfavored when p_UL<c. The interval between these cases exposes the decision significance of the unresolved information. This is an elementary implication of expected-loss bounds, not a novel normative theory.
Additional information has economic value when it may change the decision. Such value requires a model of information acquisition, how decisions adapt after learning, and the costs of collection or delay. An insurer may reasonably adopt limits, deductibles, exclusion clauses, capital buffers or preventive measures while uncertainty remains. Such decisions should be documented as conditional on evidence and institutional preferences, rather than represented as demonstrations of an exact unknown probability.
Insurance Applications
In new coverages, sparse claims may reflect genuine rarity, short exposure, product exclusions or an immature reporting system. Frequency estimation should use insured time, event and policy definitions, reporting development and the selection of buyers. External experience may be informative when underwriting class, jurisdiction, peril, limits and reporting conditions can be compared.
In catastrophe insurance, many individual policy records may share one regional storm process. Assessments should distinguish hazard event frequency, exposure, vulnerability, contract response and claims development. A two-hundred-year annual exceedance level is a quantile statement under an assumed distribution, not an assurance of a fixed occurrence cycle. Changes in climate, land use and insured value demand forecast-horizon sensitivity.
In liability insurance, the delay between incident, notification, adjudication and ultimate payment can exceed the available history. Recent periods are incompletely developed by construction. A rate based on paid claims alone measures financial processing as well as risk. Projections should distinguish occurrence from reporting and payment stages.
Risk engineering also generates useful evidence without an extensive loss record. Near misses, inspections, component failure analysis and physical constraints can justify prevention, but those sources of information must be separately identified from an observed loss frequency.
For US actuarial work, ASOP No. 23 on data quality supplies official guidance on selecting, reviewing, using and disclosing limitations of data. Its jurisdiction is specific. International comparison should preserve each country's professional and regulatory requirements. The Basel risk-reporting framework supplies a banking-sector illustration of completeness, accuracy and provenance as governance concerns; it is not represented here as a direct insurance supervisory rule.
Worked Analytical Cases
A. No observed high-severity losses
Under the hypothetical binomial model with zero events in one hundred genuinely independent, fully observed trials, the maximum-likelihood estimate is zero and the exact one-sided 95 percent upper bound is approximately 2.951 percent. Under the same likelihood with a uniform Beta prior, the predictive event probability for one further exchangeable trial is approximately 0.9804 percent. The two numbers answer different questions. Independence, complete follow-up and stability are conditions of both calculations, while the Bayesian result additionally depends on its prior.
B. Selected administrative claims
A population has event probability ten percent. Eighty percent of events and twenty percent of non-events enter a database. The event probability conditional on inclusion is 30.769 percent. Ten million such selected records would not identify the ten-percent population risk without additional selection information. Audits or independently measured inclusion probabilities could establish an adjustment, subject to their own measurement uncertainty.
C. Outcomes missing from forty percent of exposure
A sample covers sixty percent of the target units, with a ten-percent adverse rate in those observed. Without further restrictions, the full-population event risk lies from six to forty-six percent. If credible new evidence restricts unobserved-unit risk to five through fifteen percent, the resulting interval narrows to eight through twelve percent. Reporting both results makes the evidentiary role of the extra restriction explicit.
D. Prevention decision
Suppose an effective preventive action costs 12,000 monetary units and avoids a loss of 200,000 when the event would otherwise occur. The uncertain event probability lies between 0.03 and 0.10. Expected avoided loss then lies between 6,000 and 20,000. With a risk-neutral one-period objective, the evidence does not determine one action, because the cost lies inside that range. If credible evidence raises the lower probability bound above 0.06, prevention is preferred under the stated assumptions. If the action prevents only half of events, the expected benefit is halved and the decision boundary changes.
E. Common catastrophe exposure
A thousand policy-years yield no claims, but the policies are concentrated in one floodplain during several calm calendar years. The thousand individual records do not represent a thousand independent flood seasons. Hazard frequency, vulnerability, property values and contractual aggregation must be separately modeled. The zero-count formula for independent policy-level hazards is not warranted by the large count of contracts alone.
Scientific Reporting and Editorial Standards
Minimum reproducibility record
A technical report benefits from a compact but complete data lineage statement: extract version, source systems, inclusion and exclusion criteria, event coding, policy identifiers, contract effective dates, exposure basis, evaluation horizon, reporting lag, amendments and validation. Where a formula is central to a conclusion, the derivation or a reproducible computation should accompany it. This allows a reviewer to distinguish arithmetic errors from questionable assumptions and from insufficient observation.
Every important uncertainty estimate should also name its statistical meaning. A confidence bound, posterior credible interval, prediction interval, expert elicitation range, scenario threshold and identification bound are distinct objects. Their coverage or interpretation differs. A report that gives a bare percentage without the relevant class risks obscuring the very question it claims to answer.
Limits of evidence and editorial independence
A bibliographic record confirms an identifiable publication, and an abstract may confirm its general scientific subject. Technical details requiring particular proofs, tables or experimental methods should not be inferred solely from metadata. A defensible source ledger therefore states whether each claim rests on a full original text, a publisher abstract, a standards page or a previously verified catalog reference. This permits a publisher to assess which statements require further verification if editorial scope changes.
Original mathematical examples can provide analytical evidence without pretending to describe an observed insurance market. Each hypothetical should carry explicit parameters and a derivation that can be checked independently. If an example uses percentages, rates or currency values, it should clearly separate its constructed inputs from any public historical statistics.
The final scientific obligation is to report material limits in a form that changes how readers interpret the result. A scenario probability that depends on stable hazard conditions should say so; a selected sample should identify the target population that its proportion actually describes; and a sparse count should report an uncertainty range. The resulting text connects its evidence directly to interpretation and decision, without granting stronger confidence to a number than its premises justify.
A published probability result should report the target event, population, time horizon, exposure and denominator, observation dates, outcome definitions, missingness mechanism, censoring, selection, model assumptions, the reported inferential quantity and numerical uncertainty. For claims data, it should also specify the valuation date and status of case development. Where data cannot be released, metadata, derivations, synthetic examples and audit traces can still support reproducibility.
Each important research citation should support the specific statement to which it is attached. A publisher landing page can verify bibliographic identity without giving access to every proof, numerical result or argument in a paywalled paper. Such access limits are part of scientific evidence. The manuscript therefore distinguishes original derivations from claims about the cited literature and excludes unsupported assertions of full-text verification.
An editorial review should establish whether a reader can follow links without encountering opaque citation IDs or long institutional parentheticals. The approved contextual-link policy embeds concise descriptions of the relevant study, standard or source directly in the prose. Complete bibliographic entries remain in References, with the associated source records, claim correspondence and access limitations recorded separately in EVIDENCE.json.
Scientific confidence in a conclusion and its statistical probability are related but distinct. The IPCC's calibrated assessment language provides one model for articulating the difference. A statement about a rare hazard should reveal which aspects are observed, which are derived, which depend on experts, and which concern plausible but unresolved future regimes.
The overarching editorial test is whether a probability claim remains interpretable after its stated premises are varied. This criterion links mathematical scrutiny, practical insurance consequences and scientific transparency without converting routine analytical findings into an additional theoretical system.
Conclusion
Limited information places inferential obligations on every probability claim. A numerical likelihood becomes scientifically meaningful through explicit event definitions, denominators, exposure, observation mechanisms, model structure and a specified relation to future risk. Sparse counts provide evidence, but zero observed losses do not establish zero probability. Exact procedures quantify sampling uncertainty under their assumptions; censoring, selection and structural change may create further uncertainty that cannot be repaired merely by more precise computation.
The distinction between sampling variation and lack of identification is central. Additional representative observations can sharpen estimates, while data systematically missing as a function of outcomes may leave the target probability weakly or only partially identified. Bayesian methods and expert judgment supply structured ways to use additional assumptions and knowledge; their contributions should be visible rather than presented as observations. Model comparison and sensitivity analysis reveal the consequences when alternative plausible assumptions change a forecast or decision.
For insurance and broader risk science, the resulting practice is to preserve an auditable connection from source evidence to risk characterization and from risk characterization to policy, pricing, controls and capital choices. The original contribution here is a combined evidentiary interpretation of established statistical and risk-science methods with independently derived examples. No novel named theory, principle, model or doctrine is claimed.
References
Actuarial Standards Board. (2016). Actuarial Standard of Practice No. 23: Data Quality. https://www.actuarialstandardsboard.org/asops/data-quality/
Basel Committee on Banking Supervision. (2019). Risk Data Aggregation and Risk Reporting. Basel Framework, SRP36. https://www.bis.org/committees/bcbs/basel-framework/standard/srp/36/inforce/2019-12-15/published/2019-12-15
Berger, J. O., Bernardo, J. M., & Sun, D. (2009). The formal definition of reference priors. Annals of Statistics, 37(2), 905–938. https://doi.org/10.1214/07-AOS587
Clopper, C. J., & Pearson, E. S. (1934). The use of confidence or fiducial limits illustrated in the case of the binomial. Biometrika, 26(4), 404–413. https://doi.org/10.1093/biomet/26.4.404
Cooke, R. M. (1991). Experts in Uncertainty: Opinion and Subjective Probability in Science. Oxford University Press. https://doi.org/10.1093/oso/9780195064650.001.0001
Cox, D. R. (1972). Regression models and life-tables. Journal of the Royal Statistical Society, Series B, 34(2), 187–202. https://doi.org/10.1111/j.2517-6161.1972.tb00899.x
Efron, B. (1979). Bootstrap methods: Another look at the jackknife. Annals of Statistics, 7(1), 1–26. https://doi.org/10.1214/aos/1176344552
Gneiting, T., & Raftery, A. E. (2007). Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association, 102(477), 359–378. https://doi.org/10.1198/016214506000001437
Hanley, J. A., & Lippman-Hand, A. (1983). If nothing goes wrong, is everything all right? Interpreting zero numerators. JAMA, 249(13), 1743–1745. https://doi.org/10.1001/jama.1983.03330370053031
Heckman, J. J. (1979). Sample selection bias as a specification error. Econometrica, 47(1), 153–161. https://doi.org/10.2307/1912352
Hoeting, J. A., Madigan, D., Raftery, A. E., & Volinsky, C. T. (1999). Bayesian model averaging: A tutorial. Statistical Science, 14(4), 382–417. https://doi.org/10.1214/ss/1009212519
Intergovernmental Panel on Climate Change. (2021). Climate Change 2021: The Physical Science Basis, Chapter 1: Framing, Context and Methods. https://www.ipcc.ch/report/ar6/wg1/chapter/chapter-1/
International Organization for Standardization. (2018). ISO 31000:2018: Risk Management — Guidelines. https://www.iso.org/standard/31000
Kaplan, E. L., & Meier, P. (1958). Nonparametric estimation from incomplete observations. Journal of the American Statistical Association, 53(282), 457–481. https://doi.org/10.1080/01621459.1958.10501452
Kaplan, S., & Garrick, B. J. (1981). On the quantitative definition of risk. Risk Analysis, 1(1), 11–27. https://doi.org/10.1111/j.1539-6924.1981.tb01350.x
Knight, F. H. (1921). Risk, Uncertainty and Profit. Houghton Mifflin. Library of Congress catalog: https://www.loc.gov/item/21012860/
Manski, C. F. (1990). Nonparametric bounds on treatment effects. American Economic Review, 80(2), 319–323. https://ideas.repec.org/a/aea/aecrev/v80y1990i2p319-23.html
National Institute of Standards and Technology. (n.d.). Exact Binomial Confidence Limits. NIST/SEMATECH Dataplot Reference Manual. https://itl.nist.gov/div898/software/dataplot/refman2/auxillar/exacbici.htm
O'Hagan, A., Buck, C. E., Daneshkhah, A., Eiser, J. R., Garthwaite, P. H., Jenkinson, D. J., Oakley, J. E., & Rakow, T. (2006). Uncertain Judgements: Eliciting Experts' Probabilities. Wiley. https://doi.org/10.1002/0470033312
Rubin, D. B. (1976). Inference and missing data. Biometrika, 63(3), 581–592. https://doi.org/10.1093/biomet/63.3.581