Experimental evidence from GeoLift

ammm can compare a time-series model with a GeoLift study and condition a new fit on an admissible experimental estimate. Experimental calibration adds a likelihood; it preserves the configured parameter priors.

In ammm 3.3.2, this is the interface for treating a GeoLift result as experimental evidence with uncertainty. GeoLift estimates an outcome contrast against a counterfactual. Its point estimate, uncertainty procedure and design assumptions remain part of the evidence. ammm does not promote that estimate to an error-free “ground truth” or infer causal validity from a successful import.

The calibration guide gives the exact equations for the existing add_lift_test_measurements method. That method calibrates a saturation-curve difference and has different sign, history and persistence semantics. Passing a GeoLift total to its delta_y column does not invoke the experimental-evidence model described here.

This initial surface supports one study, one changed channel and one treated geography matching the declared time-series model geography. It does not support FE, CRE, named RE, time-varying media/intercepts, custom additive effects or automatic local-to-national extrapolation. Use the Normal outcome likelihood, trailing NoAdstock, GeometricAdstock or DelayedAdstock, and NoSaturation or LogisticSaturation.

The MMM may contain other unchanged channels and supported controls or seasonality. The restriction is one changed channel in the experiment, not one channel in the whole MMM. There is no automatic pooling of studies or multivariate experimental covariance model.

Supported uses

What is measured

The study supplies complete treatment and counterfactual media histories. ammm evaluates both through its configured transforms with the same sampled parameters and training-derived scales, then sums their difference over the outcome window. The result is cumulative incremental outcome in original target units.

The intervention window and outcome window are separate. Both paths include the required prehistory. Include a carryover tail only when the study measures it. Counterfactual media need not be zero. A GeoLift synthetic-control outcome does not provide the counterfactual media path.

Unchanged baseline and control effects cancel in the admitted additive model. The study must isolate the represented intervention. Other media changes, promotions, spillovers, treatment assignment versus delivery and post-treatment controls require methodological assessment outside this calculation.

Model-implied experimental contrast

Let $x^{(1)}_{j,t}$ and $x^{(0)}_{j,t}$ be the fixed treatment and counterfactual input histories for the changed channel $j$ . Let $c_{x,j}>0$ and $c_y>0$ be its channel scale and the target scale from MMM training. Let $A_j(\,\cdot\,;\psi_j)$ denote the configured finite-lag adstock and $S_j(\,\cdot\,;\phi_j)$ the configured saturation, including its amplitude. For adstock_first=True, define

For adstock_first=False, the order is

For the study’s inclusive outcome window $W$ , ammm computes

This is a cumulative outcome contrast in the original target units. The two histories use the same current sampled parameters. ammm reuses the model’s existing parameter tensors and transform implementations; it does not create a second set of adstock or saturation priors. Supported fixed parameters remain fixed.

For example, normalised geometric adstock with l_max=L uses

The current period counts among the $L$ lags. The implementation respects the configured normalisation and active alpha or half-life parameterisation. It requires at least $L-1$ pre-intervention periods for finite-lag carry-in; NoAdstock requires no carry-in. Training and experimental histories must have the same regular date cadence. Both training and experimental histories require at least three observations, including with NoAdstock; the training dates must be timezone-naive. A one-period outcome window remains possible within a longer history. No resampling or lag-horizon adjustment is inferred from the study.

The two input histories must agree outside the declared intervention window. Their responses can still differ after media return to the same level because of adstock. An outcome window that includes those periods measures part or all of that remaining effect. ammm includes only the tail that the study actually measures.

In the admitted additive model, unchanged intercept, control, seasonality and other-channel contributions cancel in this paired contrast. That algebraic cancellation is conditional on the model. It does not prove that confounding, spillovers or simultaneous interventions are absent from a real experiment.

Import a study

from ammm.mmm.calibration import load_geolift_study

study = load_geolift_study("data/experiments/study.json")

The portable JSON export has three root fields:

The study mapping requires:

• study_id, channel, unit, model_unit, outcome, outcome_unit and input_unit;
• intervention_start, intervention_end, outcome_start and outcome_end;
• history: rows with date, treatment and counterfactual, or a mapping containing csv and its sha256;
• footprint: all outcome estimation periods, including donor and pre-treatment observations, represented by unit, inclusive start and end;
• uncertainty: the observation model and its method/approximation justification;
• discrepancy_sd and discrepancy_justification, including an explicit choice of zero;
• independence: conditional_independence, stable_response, intervention_isolated, rationale and limitations;
• provenance: software, version, exact source_commit, augsynth_version, source_hashes, inference_requested, inference_resolved, call and optional fallback_messages.

unit must equal the explicit model_unit declaration. ammm’s time-series inputs do not contain a geography or currency registry: these metadata are analyst assertions, not independently verified measurement or transport claims.

The result mapping requires incremental, n_treated: 1 and n_periods. The period count must equal the number of history rows in the outcome window. Optional average_att must reconcile with incremental. Optional cumulative interval fields are lower_bound, upper_bound and confidence_level. Only two-sided finite intervals are supported initially.

The explicit sign convention is treatment minus counterfactual, and the supported estimand is cumulative_incremental_outcome. Units must be consistent before import. ammm does not infer exchange rates, conversion value, media delivery from spend, or a conversion from percentage lift or incremental return on advertising spend (iROAS). An uncertain conversion or denominator requires its own uncertainty treatment; changing the unit label is insufficient.

The loader accepts either average_att_standard_error or cumulative_standard_error. An average ATT standard error is multiplied by the number of treated units and outcome periods. A cumulative error is used directly. Per-period ATT_se is rejected. Numeric uncertainty belongs in result, not duplicated in study.uncertainty.

For an equally weighted average treatment effect $\widehat a$ over $G$ treated units and $T$ measured periods, the matching cumulative quantity is

The second identity applies to the standard error of that aggregate average. It does not turn a vector of daily standard errors into an aggregate error. For estimated period effects $\widehat d_t$ ,

Ignoring those covariance terms is an additional assumption. The initial importer admits $G=1$ and checks $T$ against the declared outcome window. Original interval bounds supplied to result must already be cumulative; the loader does not rescale interval fields. GeoLift’s inference explanation illustrates the distinction between average ATT and cumulative output.

CSV paths resolve relative to the manifest. Their checksums are verified before use. All resolved histories and metadata are retained with the fitted model; reloading does not require the original JSON or CSV files.

Choose an uncertainty contract

Comparison with the original interval

Use kind: interval_only with a declared method and the original interval in result. The study can be compared with a fitted model. Calibration is rejected.

ammm does not invert confidence intervals into standard errors or interpret conformal p-value grids as probability densities. GeoLift’s inference method and any fallback must be recorded explicitly. See the GeoLift inference explanation.

Calibration using a justified Normal summary

Use kind: normal_summary, a declared method and a non-empty approximation justification, together with a positive aggregate standard error in result. Let $\widehat\Delta_E$ be the signed study estimate, $s_E>0$ its cumulative sampling standard error, and $\tau_E\geq0$ the declared cumulative discrepancy standard deviation. The observation model is

where the second Normal argument in this equation is a variance. The PyMC call receives sigma=sqrt(s_E**2 + tau_E**2), a standard deviation. All three effect/error quantities use the original target units.

One interpretation is $\widehat\Delta_E=\Delta_E(\theta)+B_E+\varepsilon_E$ , where $B_E\sim\mathcal N(0,\tau_E^2)$ and $\varepsilon_E\sim\mathcal N(0,s_E^2)$ are independent conditional on the model. Marginalising $B_E$ gives the stated likelihood. ammm does not estimate $\tau_E$ from a single study.

Under the declared conditional independence of experimental and MMM observations, the posterior update is

This term directly informs the response parameters that affect the complete experimental contrast. A single aggregate study generally constrains a combination of amplitude, curvature and carryover; it does not identify each parameter separately.

A standard error alone does not establish a Gaussian sampling distribution. The approximation must be justified for the study estimator. discrepancy_sd assumes an additional independent zero-mean discrepancy. It is not inferred from the study, and increasing it cannot repair an invalid intervention mapping. Assess sensitivity using separately configured fits.

For example, an estimate of 1,200 conversions with sampling SE 200 and discrepancy SD 150 contributes a Normal observation with SD 250 conversions. It favours parameter draws whose complete experimental contrast is compatible with 1,200 at that uncertainty. It does not set any coefficient to 1,200 or to an ROI derived from that number. This is an arithmetic illustration, not a reported study.

The Gaussian observation is an analyst-justified approximation to the study estimator’s sampling behaviour. A conformal confidence set is constructed by test inversion; its p-value curve is not a posterior density or a sampling likelihood. A jackknife+ interval is not automatically a Gaussian error model either. Retain the original inference method and any fallback, and use comparison-only evidence when a suitable sampling model is unavailable.

Signed and inconclusive estimates are retained. If treatment increases media in every changed period, a precise negative estimate can conflict with the default positive media-response priors. A media reduction can legitimately predict negative lift under those same priors. The observation likelihood preserves the response support implied by the configured structural priors; it does not impose positive response coefficients or expand that support.

If an interval includes zero, describe the estimate as inconclusive as to sign at that confidence level, unless it supports a narrower practical-effect claim. Do not discard the result or replace it by zero. A precise conflict with the model should prompt review of the intervention mapping, outcome definitions, response support and discrepancy assumptions.

Compare without changing a fit

comparison = fitted_mmm.compare_experiment(study)

The returned xarray dataset retains the paired model-implied draws and study metadata. Comparison does not change the fitted posterior. Overlapping or interval-only studies remain descriptive comparisons; they are not independent validation or a new hypothesis test.

Calibrate a new fit

The following example assumes X_train and y_train already describe the same channel, outcome, geography and units as the imported study. Choose the model specification using the MMM design; do not copy transform settings from an unrelated experiment.

from ammm.mmm import GeometricAdstock, LogisticSaturation
from ammm.mmm.panel import PanelMMM

mmm = PanelMMM(
    date_column="date",
    target_column="sales",
    channel_columns=["tv"],
    estimator={"type": "time_series"},
    adstock=GeometricAdstock(l_max=8),
    saturation=LogisticSaturation(),
)
mmm.build_model(X_train, y_train)
mmm.add_experiment_evidence(study)
idata = mmm.fit(X_train, y_train)
report = mmm.experiment_report()

Add evidence after building and before sampling. Attaching it to an already fitted model is rejected; create a new model instance. The first version accepts one study and does not combine it with older lift-test or cost-per-target calibration nodes.

Calibration requires explicit conditional-independence, stable-response and isolated-intervention declarations. It also checks for overlap between training outcome dates and the study’s same-unit estimation footprint. Include the full GeoLift pre-period and donor footprint: removing only treated post-period rows does not prevent double-counting. Non-overlapping rows do not prove independence; shared shocks, serial dependence and data-driven selection remain analyst responsibilities. ammm does not automatically remove training observations.

The model retains the response configuration used to build its graph. Changing transform settings, fixed response parameters, transform order or the likelihood after construction invalidates experimental attachment, comparison and saving. Build a new model for a changed specification.

The study used for calibration is part of the fit’s evidence. Agreement with it is a consistency check, not independent causal validation.

To quantify the change due to the external likelihood, retain a baseline model fitted with the same specification and priors but without the experiment. Call baseline.compare_experiment(study) and compare that report with the new fit’s experiment_report(). The calibrated report does not reconstruct a missing uncalibrated posterior.

Run the bundled workflow

From the repository root, the complete demo configuration can be loaded through the public YAML builder:

from ammm.mmm.builders.yaml import prepare_mmm_from_yaml
from ammm.mmm.panel import PanelMMM

prepared = prepare_mmm_from_yaml("data/demo/geolift/config.yml")
mmm = prepared.complete()  # builds the graph and applies its calibration step
idata = mmm.fit(prepared.X, prepared.y)  # uses the configured sampler defaults
report = mmm.experiment_report()

mmm.save("geolift_calibrated.nc")
restored = PanelMMM.load("geolift_calibrated.nc")
restored_report = restored.experiment_report()

Do not call add_experiment_evidence again after complete(): the YAML calibration step has already attached the study. Likewise, loading a retained experimental model restores the study’s graph contribution once.

For the full staged pipeline:

python3 runme.py --demo geolift

For a baseline/calibrated comparison that reads generator truth only after fitting, use the separate demo runner:

python3 scripts/geolift_run_demo.py --output-dir /tmp/ammm-geolift-comparison

That output directory must be new. The bundled sampling budget is intentionally short and does not establish convergence. The demo README documents generation, the external R commands and the special uncertainty law.

YAML and retained reports

calibration:
  - method: add_experiment_evidence
    params:
      study:
        geolift_export: data/experiments/study.json

The manifest path resolves relative to the model YAML. Pipeline diagnostics write experiment_report.json for a calibrated model, separate from generic predictive calibration diagnostics. The public report includes the declared study uncertainty, model contrast and applicability limitations.

The complete study contract is saved with inference data and rebuilt exactly once on load. Experimental log-likelihood is retained separately from the training outcome variable. ammm’s training LOO/WAIC diagnostics select the training outcome likelihood; an experimental aggregate is not another daily observation.

Holdout fits constructed from the same YAML also retain its experimental evidence. Their scores are conditional on that evidence. They do not establish an as-of-time backtest when the study would have been unavailable at the forecast date.

Read the report correctly

compare_experiment and experiment_report return xarray datasets. The experiment coordinate identifies the study. Each posterior draw evaluates both intervention histories with the same parameter draw before subtracting their responses.

Interval-only reports have no invented sampling SD, observation-predictive interval or predictive tail area. An original frequentist study interval and a Bayesian model-contrast interval have different interpretations; overlapping intervals are not a formal agreement test.

For posterior contrasts $\Delta_E^{(1)},\ldots,\Delta_E^{(M)}$ and combined observation SD $s_*=\sqrt{s_E^2+\tau_E^2}$ , the predictive mixture has CDF

The reported tail area is $2\min\{F_E(\widehat\Delta_E),1-F_E(\widehat\Delta_E)\}$ . It is neither the posterior probability of a positive effect nor an independent experiment p-value. When the study helped fit the model, this checks consistency with assimilated evidence. It does not provide a confirmatory test on new data.

Report attributes retain the resolved study and its limitations, overlap dates, and whether it was used for calibration. comparison_role is assimilation_consistency for the fitted study and descriptive_comparison otherwise. The software does not label these reports as independent validation. Examine convergence and model fit alongside the report; a narrow contrast interval from unreliable sampling is not a reliable uncertainty statement.

Analyst responsibilities and failure modes

Before calibration, establish:

1. The same causal contrast. Identify what was assigned, what media were delivered, which other interventions changed, and what the counterfactual media would have been. Synthetic-control outcome predictions alone do not answer the last question.
2. Compatible measurement. Align outcome definitions, input and outcome units, geography, period frequency, intervention dates and measured tail. Distinguish cumulative outcomes from averages, ratios and percentages.
3. Defensible uncertainty. Retain the study’s original inference method and fallback. Justify any sampling approximation and the externally chosen discrepancy scale. A standard error is not proof of unbiasedness.
4. Independent information. Include pre-period and donor outcomes in the study footprint, not just treated post-period outcomes. Check shared shocks, estimated preprocessing and selection even when dates do not overlap.
5. Stable applicability. Assess changes in campaign, population, response regime and measurement between the study and the MMM. The imported unit declarations are assertions, not a geographical transport model.
6. A sensitivity and review plan. Compare separately fitted baseline and calibrated specifications, inspect parameter/contrast conflicts and vary defensible discrepancy assumptions. Keep a genuinely independent study separate when evaluating transfer.

The initial software can reject structural incompatibilities and declared outcome overlap. It cannot verify the causal assumptions from a manifest. Do not bypass an overlap rejection by deleting estimation periods from the footprint, or weaken an uncertainty declaration simply to make a study import.

Demonstration and evidence boundary

The GeoLift demonstration under data/demo/geolift/ uses original synthetic data with separate training and experimental episodes. It runs GeoLift outside the Python runtime and imports the resulting export. Its uncertainty law is specific to the documented synthetic design. It is not a general method for converting arbitrary GeoLift intervals to likelihoods.

The standard GeoLift walkthrough outcomes alone lack the required media histories. Attempting a model-ready import without them fails explicitly.

The synthetic demo uses one treated unit with independent Gaussian outcome errors and 24 identical noiseless donors. It checks that the fitted synthetic trajectory equals the observed donor trajectory. Only under that artificial construction does its exporter use pre-period residual SD times the square root of the number of measured periods as cumulative sampling SE. It labels the plug-in Normal approximation explicitly and omits uncertainty in the estimated residual SD. This is a demonstration-specific approximation, not a general GeoLift uncertainty estimator.

The retained bounded demo completed both fits, but its short chains produced convergence warnings. Its generator truth is stored separately and is read only for scoring, after inference. A result becoming closer to that truth in one seed is not evidence of general bias reduction or nominal interval coverage.

Deterministic tests verify contracts, numerical contrasts and parameter dependence. Bounded sampling verifies execution and persistence. Neither a successful demo nor passing software tests establishes general effect recovery, interval coverage or causal identification. Those claims require separately specified statistical qualification.

Sources and implementation

• GeoLift walkthrough: study workflow and reported effect quantities.
• GeoLift inference explanation: conformal confidence sets, p-value grids and jackknife+ inference.
• Study contract and GeoLift importer: admitted fields, aggregation and validation.
• Experimental model: symbolic response contrasts, observation likelihood and reports.
• Calibration guide: the existing saturation-lift and cost-per-target interfaces and their different contracts.