Trueness calibration method: enterprise programme waves

25 October 2026. Proposed methods for an anchor programme and two audits; not empirical validation of Trueness. The current per-cell design supersedes the earlier single-audience pilot.

1. What the enterprise pilot should establish

The pilot is one annual anchor programme and two separately scoped Enterprise Audits over six operating months. Everyday value is Panel, Verdict, Challenge, Talk, Ask the research and Export. Measurement establishes which specific quantitative claims may accompany those experiences; fluent characters and useful discussions are not evidence that twelve voices represent a population.

Foundation covers three audiences in two markets, with quarterly full human waves. Enterprise covers six audiences in four markets, with quarterly or semiannual full waves explicitly contracted and costed. Market-level monthly pulses between full waves are drift screens only. The approved annual contract, not a generic product name, defines cadence and scope.

The initial objective is a reproducible record per audience × market × language × task × configuration, with pass/fail/inconclusive endpoints and a useful decision trail. A six-month pilot measures two Foundation waves and commercial continuation; it cannot observe a full annual renewal or establish category-wide accuracy. All sample/cost figures are planning designs pending supplier feasibility, method review and client agreement.

2. Establish the unit of claim before building an audience

Complete a one-page claim card for each audience-market-task record:

There is no default commercial accuracy tolerance. Record the minimum useful difference and maximum tolerable error per decision, then power the actual design. If the sample cannot support that claim, increase it, narrow the claim or accept an inconclusive outcome.

3. Human evidence and rights audit

Inventory only customer-authorized evidence. Record research owner, licensing and permitted use, participant permissions where applicable, fieldwork dates, geography, language, sample source and base, quotas, weights, instrument, stimuli, exclusions, known bias, storage location and deletion date. Research that can be read for a client's current project is not automatically authorized for model training, cross-client pooling or future commercial use.

Keep three typed layers:

• Observed: identifiable source facts, measured estimates and genuine participant quotations, with original base/date and citation.
• Derived: researcher-created segmentation, weighted estimates, summaries and explicit assumptions, each traceable to inputs.
• Generated: simulated reactions, invented narrative detail and model interpretations. These never become observed evidence merely by being saved.

An audience should preserve competing explanations and within-segment variation. Do not turn an average trait into a universal personal characteristic. Avoid invented biographies that supply unsupported behavioral drivers. If illustrative personas help a researcher think, mark them as illustrations and leave them out of the quantitative weighting mechanism.

Human data remain a fallible reference. Review coverage, nonresponse, panel conditioning, bots, duplicate respondents, response quality, language access, acquiescence, social desirability and stimulus exposure. Ask the supplier for quality procedures and exclusion logs. Predefine exclusion rules; do not discard inconvenient human responses because they disagree with the simulation. See AAPOR best practices.

4. Programme waves, per-cell samples and Audit scope

A cell is one audience × market × language for a named task/instrument. The following implements the agreed scale-up, rather than dividing a fixed 1,000-person market wave across an unlimited number of audiences.

Foundation: three cells per market, two markets. First wave is 1,200 completes per market (2,400 programme total); later waves are 960 per market (1,920 total). The first service year has 8,160 quantitative completes, including 5,760 sealed observations across 24 separate cell-waves, plus 72 initial interviews. A later unchanged year has 7,680 completes. These annual totals are not the sample size of any individual claim.

Enterprise: six cells per market, four markets. First full wave is 2,400 per market (9,600 total); later waves are 1,920 per market (7,680 total). Semiannual full waves plus ten intervening monthly pulses of 150 people per market produce 23,280 first-year completes. Quarterly waves plus eight intervening pulse months produce 37,440. Initial interviews total 288. These are material fieldwork commitments; see the cadence costs. A cheaper cadence changes the evidence age and product claim, not just the bill.

Quota each audience separately using eligibility grounded in the client's segmentation. If memberships overlap, identify unique people, report overlap, and preserve their dependence in analysis. Never count one person's answer as two independent respondents. Sampling rare segments or subgroup precision requires boosts beyond the baseline. Market totals are recruitment allocations, not evidence of representativeness.

Each human may see all four concepts only if pretesting establishes acceptable burden, with randomised order and matched synthetic exposure. Four answers from one person remain a cluster. Monadic or incomplete-block designs reduce per-concept bases and require a new power calculation. Four new concepts are a bounded transfer test, not a reliable correlation across all future creative work.

Enterprise Audit

Freeze the incumbent configuration, outputs, source snapshot, prompts, population definition, mapping and available provider metadata. Black-box limitations must be recorded rather than concealed. Obtain permission to test and retain outputs. Each contracted cell receives 240 new sealed quantitative completes on up to four reserved concepts, plus four contextual interviews in the financial allowance; interviews do not tune or repair the system under test.

Audits do not receive a development quantitative split or fit a correction to their own holdout. Compare the incumbent with feasible frozen historical/researcher baselines. Missing access or an untestable population can yield a paid data-gap/inconclusive outcome. Own-build interests require disclosure, external review and separate custody; a vendor-funded audit is not automatically independent.

The pricing hypothesis is $15,000 per cell, $75,000 floor, with added scope quoted. Ten cells are $150,000; eighteen are $270,000 before special recruitment or added task families. The finance model's $110,000 average represents a mix of seven/eight-cell engagements. The pilot instead uses two exact seven-cell $105,000 audits: 1,680 quantitative completes and 28 contextual interviews each. Their cell maps must be named in the SOW, not inferred from a product label. The 6–8-week clock starts only after access, supplier capacity, stimuli and protocol readiness.

Across the pilot: two Foundation waves contribute 4,320 quantitative completes; the two audits contribute 3,360; total 7,680, of which 6,240 are sealed across separate engagements and waves. Initial/contextual interviews total 128. None of these totals can be pooled into a universal accuracy claim.

5. Precision and what the sample cannot prove

For rough planning only, the worst-case 95% halfwidth for a single unweighted binomial share is 1.96 × sqrt(0.25/n). This assumes independent probability-like observations and ignores coverage/measurement bias. Actual analysis must use appropriate intervals and survey design; NIST's Wilson-interval guidance is relevant for simple binomial data.

These are single-estimate illustrations, not the confidence width for every validation statistic. Simultaneous inference, subgrouping, weighting, clustering and model error can require more data. The familiar n≈384 is not enough to guarantee a model is within five points: it only gives an approximate five-point human-share halfwidth before other uncertainty, assuming worst-case proportion and the stated design.

For weighting diagnostics report n_eff = (sum(w))² / sum(w²), weight range and trimming decisions. This Kish calculation captures unequal-weight loss, not every source of design effect or nonprobability selection bias. Use survey-aware variance estimation where the design requires it. A quota or opt-in panel's intervals are conditional on modeling/sampling assumptions and do not establish population representativeness.

With a 240-person holdout and an effective base of 160 after weighting/design effects, even an apparently small miss may have an upper error bound too wide to pass the decision-specific tolerance. A subgroup of 120 already has about nine-point single-share uncertainty before those penalties. Plan for inconclusive subgroup findings. Do not “repair” an underpowered group by generating more responses.

To power a specific comparison, simulate the intended human design using plausible effect sizes, within-person correlations, weights and expected error; select sample size to meet a stated probability of passing the predeclared equivalence rule when performance is genuinely adequate. The table is not a power analysis.

6. Build and calibrate candidate methods

Start with transparent evidence conditioning and a small calibration layer. Fine-tuning is optional later; it creates extra costs, rights considerations and validation requirements. Use the strongest available frontier models requested by the venture owner, but choose the production configuration by task-specific evaluation, latency and cost rather than model prestige.

Evaluate a small preregistered set of baselines:

1. Researcher using the authorized evidence without simulated personas: record their forecast/shortlist before the holdout is opened.
2. Simple empirical baseline: development mean or a regularized segment-level estimator, using only variables available at prediction time.
3. The same model with generic audience information and the same response instrument.
4. The same model with permitted customer evidence and explicit uncertainty/abstention rules.

The comparison between 3 and 4 estimates the incremental value of customer grounding under that setup. The comparison with 1 and 2 asks whether the overall service adds value. Extra complexity earns its place only if it improves a predeclared outcome or produces documented workflow benefits without unacceptable error.

Fit only low-complexity corrections supportable by the first wave's 160 development humans per cell and four development concepts: for example a global intercept/shrinkage correction, with strong regularization and any segment correction justified in advance. Four concept means cannot support a flexible concept-level calibration model. Do not claim an elaborate mapping is identified because it contains thousands of generated rows. Any text-to-scale anchors must be fixed using development material before evaluation; the consumer SSR paper's tuning caveat makes this especially important (Maier et al.).

Use independent generation contexts for simulated observations. Do not let one generated respondent see another's answers unless group interaction is the explicitly validated construct. Match question wording, options, randomization, information available, stimulus order and exposure as closely as possible. A model with unlimited document inspection is not undergoing a human timed-ad exposure task.

Version and retain model/provider identifier, request date, system/developer prompts, temperature and other available decoding settings, reasoning settings, tool access, retrieval rules, source hashes, calibration transform, population weights, response parsing, refusal handling, invalid-output rules, retries and API usage. A reproducible manifest is more useful than an unanchored confidence score.

Limit development search, for example to two model configurations × three prompt/calibration variants, chosen before evaluation. Log all trials. Use development cross-validation where possible, but do not pretend four concepts supply rich cross-task validation. Choose one configuration and freeze it. Any change made after seeing the holdout consumes that holdout; the revised method needs new human evidence for confirmation.

Later-wave development adds 80 people per cell, with dated previous evidence usable only for development. Accumulation does not erase temporal change or increase the current sealed base. Limit model flexibility to what effective developmental data support.

7. Holdouts and leakage controls

Three distinct holdouts answer different questions:

Record who can access each split. Keep sealed target responses and any reports containing their outcomes outside prompts, retrieval, summaries, notebooks available to model developers and generated-persona memory. Separate a holdout custodian from the person tuning the system; one contractor can fill this role if necessary. Store hash/timestamp manifests before predictions are revealed.

Historical test results already in a client's deck are not an honest holdout if that deck or its summary is included in the grounding corpus. Public benchmark data may also have appeared in pretraining. Private, prospectively measured new concepts reduce this concern but do not prove the base model has never seen similar material.

Reserve a test set only once; do not inspect results after every prompt revision. An automatic evaluation pipeline does not make repeated tuning on a test set independent. The analytic flexibility evidence supports this separation (Cummins).

Rotation across quarters and languages

Assign respondents and stimuli before fielding. The custodian generates separate development and sealed IDs; supplier staff cannot expose sealed answers through live dashboards, transcript exports or agent tools. Freeze candidate outputs before collecting or opening final sealed outcomes. Development participants, initial interviewees and previously exposed validation respondents are excluded from the same-wave final test.

Prefer fresh people at each full wave; record prior-wave participation and panel tenure. Avoid a blanket exclusion of experienced panel members that could create new coverage bias. A designed recontact study is a separate diagnostic with clustered longitudinal analysis and cannot be relabelled fresh cross-sectional evidence. Supplier-to-supplier de-duplication uses privacy-preserving identifiers under the approved contract.

After the signed record is immutable, old holdout data may enter the next development snapshot if rights allow. They are permanently marked consumed; they never become an independent future holdout again. Validation stimuli are also retired from the future new-stimulus bank. Stable anchors are useful for drift but are reported separately from novel-stimulus transfer.

Each market/language uses its own recruitment frame, instrument adaptation, cognitive pretest and weights. Translation/back-translation and bilingual review establish a workable instrument, not measurement invariance. Pool only under a prespecified hierarchical/multigroup model with justified comparability; always display market-specific results. A global pass cannot conceal a failed market.

8. Evaluation: keep the targets and uncertainty separate

Calculate these before writing the persuasive narrative:

If a client needs correlations, multivariate relationships or price sensitivity, those become separate primary targets with appropriate human studies. Means or rank agreement cannot validate them. The distinction is supported by Bisbee et al..

For qualitative analysis, two humans independently code a prespecified set of human and synthetic responses using a development-derived codebook, blinded to origin where feasible. Report disagreements and adjudication. Keep new human themes visible instead of forcing them into synthetic categories. A generated quotation is labeled as generated everywhere it appears. A small qualitative sample does not establish the population prevalence of a theme.

Separate four kinds of uncertainty:

• Human-reference uncertainty: respondent sampling, weighting, missingness and measurement; estimate with methods suitable for the design.
• Finite-generation variability: repeat runs of the frozen model; useful for computational stability, not a population margin of error.
• Configuration sensitivity: model, prompt, ordering, paraphrase and context perturbations; diagnose as robustness, not independent human replication.
• Transfer uncertainty: new stimulus types, audiences, markets and time; cannot be removed by bootstrap resampling of the existing cases.

For the fixed four-concept test, bootstrap whole human respondents with all their answers together, respecting stratification/weights. If calibration is estimated from development people, propagate its uncertainty through a nested development-resample/refit when making an inference about the fitted method rather than conditioning on the frozen fit; distinguish those analyses in the report. Record simulation variability separately or propagate it as a computational component. Neither approach captures unknown systematic model bias.

For claims beyond the four concepts, new concept sets must be the independent replication units. Do not bootstrap thousands of synthetic records and call the resulting narrow interval evidence of generalization. Even 20 correct out of 25 independent binary decisions has a wide Wilson interval (approximately 61–91%), and related decisions are not independent. Three customer case studies support learning, not a universal performance percentage.

9. Decision-specific gates: pass, fail and inconclusive

Before fieldwork, register maximum tolerable error Δ, minimum actionable human difference δ, robustness escalation ρ, subgroup targets, answered-coverage floor and multiplicity procedure. These are chosen for the decision and powered against the actual design. There is no default ten-point product promise.

Use design-aware simultaneous intervals, such as a respondent-cluster bootstrap maximum statistic or an appropriate conservative multiplicity correction. Preregister whether the family is per record, task, market or whole programme; do not advertise a whole-programme pass by combining unadjusted per-cell passes. Secondary findings and post-hoc subgroups are exploratory. Repeated quarterly looks require a sequential rule or separately bounded claims; repeatedly checking until a pass is not validation.

Pass: the relevant uncertainty set is wholly inside tolerance. Fail: evidence shows unacceptable error/harm or a required integrity condition fails. Inconclusive: uncertainty spans both acceptable and unacceptable performance, reference quality/bases fail, or the human comparison cannot resolve the decision. A nonsignificant difference is not equivalence. Answer/abstain rules are frozen as well, with coverage and errors among answered cases reported.

These decisions govern quantitative claims, not the existence of a useful exploratory workspace. A failed claim can still leave lawful, clearly labelled Talk or Ask the research functionality available. Neither generated agreement nor commercial pressure changes the record.

10. Release, cadence and model-version changes

Every answer carries generated/observed/derived labels, audience and market, relevant record/version, allowed task, actual fresh-human base for this run (often zero), historical validation base, expiry and limitations. Detailed evidence sits behind a short readable status. Historical validation intervals are not new-concept population intervals.

Foundation full waves run quarterly. Enterprise's SOW chooses quarterly or semiannual full waves and specifies monthly market-level pulses between them. A 150-person pulse per market screens large changes in shared anchor measures; it neither supplies 150 observations per audience nor recertifies six audiences or small subgroups. Prespecify pulse thresholds and escalation. No significant movement does not prove no drift.

Run a stable computational sentinel suite continuously and on provider/prompt changes. It detects behaviour, formatting and latency changes; it is not a human study. Maintain immutable configuration manifests and a provider retirement calendar. Any material change creates a candidate version and an affected-record register. Freeze the old predictive scope or serve an already validated fallback; retain exploratory access only with clear status and permission.

The annual programme includes investigation, remediation and necessary model-change revalidation for its existing agreed scope, funded by a model-change reserve, with no automatic upsell. Extra audience/task/market scope may be a change order. A reserve is a cost allowance, not guaranteed adequate sample or permission to reopen a consumed holdout. If it is exhausted, management funds the existing obligation, agrees a contractual remedy, or pauses unsupported claims; it does not silently substitute a model-only check. Detailed steps are in the model-version SOP.

Reviewer approval attaches to the exact evidence/configuration. Expiry and events are enforced in web/API/MCP and exports. Full-wave review cannot indefinitely extend an unrelated task or every free-form conversation.

11. Two-quarter pilot and accountable responsibility

Audit categories may differ from the anchor. Keep a comparable bounded task family where practical but treat each as a separate case, not pooled proof. Clinical, financial-outcome or regulated advertising claims require specialist scope and review; enterprise price does not remove methodological risk.

The research lead owns design and interpretation; the custodian controls sealed targets; the external reviewer checks own-build claims; the venture and commercial leads own contracts. Neither Claude nor Codex is the legal research signatory.

12. Capacity, procurement and quote requirements

Foundation planning capacity is 0.5 research FTE per programme; the model hires a whole $180,000 loaded lead before activation for up to two programmes. Initial interviews, source review, six cell configurations, four annual full waves, analysis, agency enablement and executive reviews must all fit measured hours. Budget 10 external review days per Foundation year; 24 annual cell-wave records imply about 3.3 reviewer hours each before common-method review, so complexity may require more.

Enterprise cadence comparisons increase capacity: semiannual 1.25 research FTE/25 review days, quarterly 2.25 FTE/45 days, plus shared engineering and client-management allocations. This is a bottom-up planning proposal to be tested, not proof that these staff can support any corpus or stimulus. Reserve reviewer and custodian slots before contracting, including contingencies for failed waves and changes.

The base seven-cell audit budgets 180 research hours, 5.5 reviewer days, 1,680 quantitative completes and 28 contextual interviews. Audit delivery capacity is additional to programme-lead capacity. Never promise an 18-cell audit at the same human budget.

Obtain quotes for per-cell incidence, recruitment sources, market/language, eligibility screen, instrument length, incentives, translation, qualitative moderation, exclusion/replacement rules, duplicate detection, wave cadence and required lead times. Distinguish supplier analysis from separately paid research labour to avoid double counting. The financial case reports fieldwork and margin for both Enterprise cadences. No confidential historical quote establishes the current price.

The initial procurement assumption is four-to-six weeks for scope/translation/recruitment readiness and two-to-three weeks for fieldwork/QA per wave; these are supplier-confirmed targets before sale, not service guarantees. Use overlapping preparatory work, never overlap sealed answers with development access.

13. Reusable implementation lessons from the reference build

Read-only code inspection identifies design patterns, not transferable client assets or scientific proof. No underlying confidential corpus was opened or copied.

This pilot does not copy the existing code or client material. Any future reuse requires a separate ownership/licensing decision. The transferable insight is the workflow: evidence intake → explicit simulation → challenge → human validation → bounded use → monitoring.

14. Programme and pilot decision record

Deliver a scope register, evidence inventory, frozen protocols, private input/output manifests, human quality/exclusion logs, per-cell comparisons, external-review findings, Calibration Records, version history and a useful customer workflow. Each quarterly executive read covers adoption, decisions, human follow-ups, restrictions, costs and next-wave commitments separately from statistical performance.

At M6, report all endpoints—including failed/inconclusive ones—actual/effective bases, model change history, useful actions and repeat users, human hours, paid continuation and contribution versus budget. Do not use record completion, speed, a high panel-run count or an executive testimonial as proof of validity. Conversely, an inconclusive quantitative result does not erase measured retrieval or exploratory value.

Continue only within supported claims and contractually funded delivery. If broader claims need more evidence, fund and preregister it; if the commercial scope cannot support the necessary humans, reprice or narrow before renewal.