Compare budget solvers

This P0 experiment compares single-start SLSQP, multi-start SLSQP, and constrained differential evolution followed by SLSQP refinement. It does not change the production optimizer or qualify a model for media decisions.

Run the experiment

Use the existing development environment with SciPy and pytest installed. From the repository root, run the reference problems first:

.venv/bin/python scripts/compare_budget_solvers.py \
  --output .planning/budget-solver-comparison/reference \
  --max-evaluations 300 --seconds 60

Add a trusted saved time-series model to compare the actual compiled posterior objective under historical channel bounds and a committed-channel constraint:

OPENBLAS_NUM_THREADS=1 OMP_NUM_THREADS=1 \
.venv/bin/python scripts/compare_budget_solvers.py \
  --model results/timeseries_20260916_181416/20_model_fit/model.nc \
  --output .planning/budget-solver-comparison/full-posterior \
  --max-evaluations 300 --seconds 60

The model path is an example of an existing local run, not a packaged fixture. The script never refits or changes the source file. Existing output directories are rejected. Defaults use all retained posterior draws, a 13-week horizon, eight local starts and three repeated seeds (17, 29, 43). A pilot can specify --draws-per-chain 100; this explicitly selects evenly spaced posterior indices and records them. It does not sample a fresh posterior for each solver. Prior groups with different draw counts remain unchanged.

The fixture must be time-series, with positive reference-period spending in every channel. Panel estimators and arbitrary custom constraints are rejected. Benchmark retained models only from trusted local sources, as with normal model loading.

Comparison contract

  • The methods use one fixed compiled objective and posterior. Repeated initial evaluations provide a smoke check for a stochastic objective or gradient. A wrapper may cache these evaluations, so this check does not prove determinism; the saved-posterior objective must itself satisfy that contract.
  • Bounds and total budget are identical. Historical mean weekly channel spend defines the initial point and budget; bounds are 50%–150% of that spend. The committed-channel case fixes the first modeled channel to its original spend.
  • All methods use normalized budget shares and an identical objective scale. One variable is eliminated to enforce the budget equality; a linear inequality preserves its bounds. Fixed channels are removed from the search coordinates. This compares solver strategies, not production’s unnormalized execution time.
  • Each method/seed receives the same objective/gradient-pair request cap and elapsed-time allowance. Single-start need not consume its whole allowance if it converges. Multi-start shares the cap across starts; differential evolution receives 70% of the work/time allowance, reserving the rest for local refinement.
  • Starts include the baseline and seeded feasible corner/interior points. They are not claimed to sample the feasible region uniformly. The evolutionary population is initialized from the same feasible-start procedure.
  • Gradients use the existing compiled PyTensor objective. Differential evolution ignores the returned gradient, so this implementation measures a shared objective/gradient oracle, not the fastest possible value-only DE implementation.
  • Compilation and warmup are reported separately from solve time. Time limits are checked between objective calls; they do not interrupt an already running compiled function. Solver iteration counts are also bounded.
  • The best visited feasible allocation is retained, even when a solver reaches a limit. Raw termination statuses remain visible. “Any local solve converged” does not certify that the selected incumbent is a global optimum.
  • Feasibility is checked independently in original spend units with a tolerance of 1e-9 times total budget. Output includes residuals and exact allocations. A feasible initial allocation provides the incumbent; a solver cannot replace it with a worse objective or an infeasible allocation.

The runner uses SciPy’s documented custom-method hook via AMMM3’s public optimize_budget method. It does not access private compiled-objective attributes or install a new production default. See the SciPy minimize interface and differential evolution interface.

Evidence and interpretation

comparison.json records runtime versions, source and benchmark-code hashes, posterior selection, bounds, starting allocation, every seed/method result, constraint residuals, work counts, and termination statuses. comparison.md provides a readable table. Lower objectives are better; the actual MMM objective is negative model-implied response. Read the JSON status: failed runs can retain completed reference evidence without completing the saved-posterior comparison.

The reference quadratic has a known solution. A deliberately non-convex two-peak example demonstrates why one start may miss a better allocation and why a finite multi-start or evolutionary search still has no global guarantee. These examples exercise computational behavior, not a calibrated marketing response surface.

Compare objective agreement, constraint residuals, repeatability, work and runtime before choosing a method. A production change needs improvement that is meaningful for the planning decision, not just a smaller last decimal place. A single demo cannot establish that one solver dominates across models or objectives.

This experiment uses the existing optimizer objective. It does not establish the proposed planner’s history-conditioned incremental estimand, objective/reporting window agreement, executable currency rounding, geo-panel support, or causal validity. Those remain separate P0/P2 gates. No statistical approval is inferred from the fixture’s existence or a successful numerical solve.

Focused verification

.venv/bin/python -m pytest tests/mmm/test_budget_solver_evidence.py --no-cov
ruff check src/ammm/mmm/evidence/budget_solvers.py \
  scripts/compare_budget_solvers.py tests/mmm/test_budget_solver_evidence.py

Tests cover known solutions, spend-unit invariance, locked and fully fixed budgets, infeasible inputs, non-convex local optima, deterministic seeds, work/time limits, non-finite/stochastic objectives, and the public custom-method integration.