Quickstart

This is the shortest supported path for using the repo.

Preconditions

• Run commands from the repository root.
• Have R available locally.
• Install the minimal repo dependency used by validation:

install.packages("yaml")

Create A New Market/KPI

Rscript scripts/scaffold_market.R IT store_visits

This creates:

markets/IT/store_visits/

Fill In The Market Folder

At minimum:

1. update config.yml
2. implement transform.R
3. implement decomp.R
4. implement build.R
5. update the market README.md
6. record exact validated runtime versions in versions.lock

Validate

Validate one market:

Rscript scripts/validate_market.R markets/IT/store_visits

Validate all migrated markets:

Rscript scripts/validate_all.R

Build

Rscript scripts/build_market.R markets/IT/store_visits

Submit

1. commit the market changes
2. open a pull request
3. product/engineering reviews for mergeability and standards

What This Does Not Yet Guarantee

The current scaffold enforces structure, config shape, and forbidden local-path patterns. Full runtime transform/predict/decomp/build verification is still a next-step enhancement once real market logic is migrated.

Getting Started

This repo is for market-specific implementations that sit on top of the shared product/package layer in forecast-mmmapi-r.

Run all commands from the repository root.

Typical Workflow

1. Create or update a market/KPI folder under markets/.
2. Keep market-specific logic in bounded files such as transform.R and decomp.R.
3. Run validation locally.
4. Open a pull request for product/engineering review.
5. Build the upload artefact from the repo version, not from a local copy.

Standard Commands

Validate all migrated markets:

Rscript scripts/validate_all.R

Validate one migrated market:

Rscript scripts/validate_market.R markets/CN/store_visits

Scaffold a new market/KPI:

Rscript scripts/scaffold_market.R IT store_visits

Build one market/KPI:

Rscript scripts/build_market.R markets/CN/store_visits

First Things To Read

• Repo Model
• Market Folder Contract
• Validation and Build

Current State

The repository structure and validation scaffold are in place. The next major step is to migrate real market logic, starting with China store visits.

Repo Model

This repository is intentionally separate from forecast-mmmapi-r.

Split Of Responsibility

forecast-mmmapi-r

• reusable package/product code
• API contract
• packaging logic
• shared templates and helpers

forecast-mmmapi-markets

• market/KPI implementations
• market configuration
• bounded transform/decomp logic
• validation entrypoints
• build manifests and governance

Why Split Repos

Without this split, the reusable product repo tends to become a hybrid of:

• library code
• one-off market scripts
• operational artefacts

That makes ownership weak and encourages drift. The market repo is meant to hold the operational market layer in one governed place.

Source Of Truth

For any migrated market/KPI:

• the folder in this repo is authoritative
• local SharePoint/Windows-drive copies are not authoritative
• upload artefacts should be built from the Git version

Intended Outcome

The goal is a workflow where market logic is:

• version-controlled
• reviewable
• reproducible
• validated in a clean environment
• traceable to a commit

Market Folder Contract

Each market/KPI implementation should live under:

markets/<MARKET>/<KPI>/

Required Files

• config.yml
• transform.R
• decomp.R
• build.R
• validate.R
• README.md
• versions.lock

Optional support folders:

• fixtures/
• manifests/

Expected Functions

transform.R

transform <- function(data, config) { ... }

decomp.R

decomp <- function(data, model, config) { ... }

build.R

build_api <- function(config) { ... }

Hard Rules

• No hard-coded local Windows, SharePoint, or OneDrive paths in committed code.
• No setwd().
• No hidden reads from analyst-local infrastructure inside transform() or decomp().
• Package calls should be namespaced explicitly.
• The same model representation must be used consistently across transform, predict, and decomp.

Config Expectations

At minimum, config.yml should describe:

• market code
• KPI name
• owner
• week start convention
• model source
• mapping source
• input locations
• decomp regex/configuration
• validation requirements

See Config Reference for a worked example and field notes.

Market README Requirements

The market README.md is required and should include:

• purpose
• owner
• expected input artefacts
• market-specific assumptions
• validation status
• transition notes from local scripts

See Market README Template.

Why This Contract Exists

The contract exists to stop each market drifting into a private implementation that is hard to review, hard to reproduce, and inconsistent with the product workflow.

Config Reference

Each market/KPI folder must define a config.yml.

Example

market_code: CN
kpi: store_visits
owner: Winnie Chen
week_start: 7

artifact_name: CN-store_visits

model:
  source: external_rds
  path: data/local/model/_sv_info.rds
  object_path: model
  api_model_policy: use_one_consistent_model_for_predict_and_decomp
  update_strategy: explicit

mapping:
  source: model_rds
  object_path: tbl_map

inputs:
  raw_data: data/local/input/cn_raw_import.csv
  fixed_forecast: data/local/input/cn_fixed_forecast.csv

decomp:
  synergy_vars_regex: "^(m_|family_|fammed_|pro_|orgsoc_|pr_)"
  adstock_var_regex: "..."

validation:
  require_clean_session: true
  require_predict: true
  require_decomp: true
  forbid_local_paths: true
  forbid_hidden_imports: true

Required Top-Level Keys

• market_code
• kpi
• owner
• week_start
• artifact_name
• model
• mapping
• inputs
• decomp
• validation

Field Notes

market_code

• short market identifier, for example CN, IT, UK

kpi

• KPI folder name and logical build target

owner

• named analyst or market owner

week_start

• 1 for Monday
• 7 for Sunday

artifact_name

• human-readable artefact stem used by build logic

model.source

• where the model object comes from
• current scaffold assumes an external RDS-style source, but this can evolve

model.path

• explicit path to the model artefact used by the market

model.object_path

• object name or object-path identifier within the artefact

model.api_model_policy

• expected consistency rule for predict/decomp

model.update_strategy

• how model updates are handled, for example explicit replacement or inherited

mapping

• where the mapping table comes from

inputs.raw_data

• explicit market input artefact used by transform

inputs.fixed_forecast

• fixed forecast input if applicable

decomp.*

• market-specific decomp regex or configuration

validation.*

• switches describing required validation behaviour

Current Limitation

The current repo validates config shape and week-start values. It does not yet validate a full schema of allowed values for every nested field.

Market README Template

Every market/KPI folder requires a README.md.

Minimum Required Content

Title

• market name
• KPI name

Purpose

• what this market folder implements

Owner

• named analyst owner
• named product/engineering reviewer if useful

Input Artefacts

• what model artefact is expected
• what raw data is expected
• whether fixed forecast input is required

Market-Specific Assumptions

• week start convention
• date/calendar assumptions
• important transform/decomp quirks

Validation Status

• scaffold only
• partially migrated
• fully migrated and validated

Notes

• known risks
• transition notes from local scripts

Why This Matters

The market README is meant to make the folder understandable without relying on oral context or analyst memory.

Roles and Ownership

This page defines who owns what in the operating model.

Product/Engineering

Owns:

• repository structure
• validation standards
• build standards
• mergeability decisions
• what counts as the authoritative repository version

Market Analyst

Owns:

• market logic
• business context
• KPI-specific transform and decomp intent
• keeping market assumptions current

Migration Authority

A market/KPI is considered migrated when product/engineering approves the implementation, validation passes for the agreed scope, and the team accepts the Git version as authoritative.

Disagreements

If analyst and product/engineering disagree:

• analyst owns the market/business intent
• product/engineering owns the repository standards and final merge decision

In practice, that means product/engineering decides whether an implementation is fit to become the authoritative build path in this repo.

Validation Contract

This page separates what is enforced now from what is intended later.

Enforced Now

Current validation checks:

• required market files exist
• config.yml loads
• required top-level config keys exist
• week_start is one of the allowed values
• required functions can be sourced
• forbidden local-environment patterns are blocked in committed scripts

Planned Next

Not fully enforced yet:

• transform runs against real market input
• transformed columns match the intended model contract
• predict succeeds
• decomp succeeds
• build produces a traceable artefact
• formula regression checks for cases like lead() and I(...)

Why This Distinction Matters

The repo should not overstate its current level of technical assurance. Today, validation proves that the market scaffold is structurally governed. It does not yet prove that every migrated market implementation runs end to end.

Validation and Build

Validation is not optional in this repository. The purpose of validation is to catch the kinds of failures that historically appeared only after packaging or deployment.

Run commands from the repository root.

Validation Entry Points

Validate all migrated markets:

Rscript scripts/validate_all.R

Validate one migrated market:

Rscript scripts/validate_market.R markets/CN/store_visits

Build Entry Point

Build one market/KPI:

Rscript scripts/build_market.R markets/CN/store_visits

Market-Specific Local Repros

Where a market investigation needs a tactical proof outside the main scaffold, the market folder may also carry a local repro bundle.

For China store visits, the current local repro can be run with:

./markets/CN/store_visits/local-repro/run_cn_repro.sh

This is not the long-term replacement for the repo-wide validation harness. It is a market-specific runner used to demonstrate that the local transform/refit/decomp path works from explicit artefacts.

What Validation Should Prove

Current enforced minimum:

1. required files exist
2. config loads
3. forbidden local-environment patterns are absent
4. required functions can be sourced
5. the market contract is satisfied

Planned next-stage validation:

1. transform runs in a clean session
2. transformed output matches model requirements
3. predict works
4. decomp works
5. build output is reproducible and traceable

Clean Environment Principle

If a market implementation only works on one analyst machine because of:

• attached packages
• hidden sourced files
• local paths
• interactive session state

then it is not ready.

See Validation Contract for the distinction between enforced and planned checks.

Build and Release

Builds should be reproducible from the repository state.

Current Build Entry Point

Rscript scripts/build_market.R markets/CN/store_visits

Current State

The build scaffold exists, but real market build logic still needs to be migrated into each market folder.

Intended Build Outcome

Each build should eventually produce:

• a market/KPI-specific upload artefact
• a traceable artefact name
• an association to the commit SHA that generated it

Intended Traceability

At a minimum, a future build path should capture:

• market
• KPI
• build timestamp
• commit SHA
• validated runtime versions

CI Versus Local Build

Current scaffold:

• local build entrypoint exists
• CI validates structure only

Target state:

• local build for iteration
• CI-backed validation before merge
• optional CI build publication once workflow stabilises

Troubleshooting

This page captures common failure modes that this repo is intended to prevent.

Transform Works Locally But Fails In API

Typical causes:

• hidden package dependencies
• un-namespaced function calls
• local helper scripts not committed
• local data reads embedded in transform logic

Model And Transform Are Out Of Sync

Typical symptom:

• transformed data does not contain the variables expected by the model object

Typical cause:

• the analyst changed the transform logic or variable representation without rebuilding or updating the model artefact consistently

Decomp Fails In Packaged Runtime

Typical causes:

• formula parsing edge cases such as lead() or I(...)
• mismatched mapping table
• model and decomp wrapper not using the same representation
• hidden reliance on attached packages or local session state

Week Start / Calendar Problems

Typical symptom:

• fixed-forecast or future data is shifted, duplicated, or partially NA

Typical cause:

• one part of the workflow assumes Monday weeks while the market uses Sunday weeks, or vice versa

Local Script Drift

Typical symptom:

• product team fixes do not match the analyst’s “real” script

Typical cause:

• local SharePoint or Windows-drive copies acting as private forks

First Questions To Ask

1. What is the authoritative Git version?
2. What exact model object is being used for predict?
3. Is the same model object used for decomp?
4. Does transform() produce the variables the model actually expects?
5. Are there any hard-coded local dependencies left?

Model Consistency

One of the main failure modes this repo is designed to prevent is inconsistency between:

• transform output
• model object used for predict
• model/decomp logic used for contribution outputs

Rule

For a given market/KPI build, transform, predict, and decomp must refer to the same intended model representation.

What This Means In Practice

If a model is rewritten for forecast/upload purposes, for example by replacing an inline combined term with an explicit single variable, then:

• transform must produce that rewritten variable
• the API model must be the updated model that expects it
• decomp must use that same updated model representation

Not Acceptable

Examples of inconsistency:

• transform produces updated variables but the API still uses the old stored model
• predict uses one model object while decomp closes over another
• decomp assumes a mapping or formula representation different from the build model

Why This Exists

This exact class of inconsistency was one of the initial problems surfaced by the China store visits investigation.

CN Store Visits

This is the first worked example and the initial driver for creating this repo.

Why It Matters

The China store visits case exposed several problems that were not just isolated coding mistakes:

• analyst logic was living outside a governed Git workflow
• local scripts had drifted away from the product-side expectations
• transform, model, and decomp logic were not clearly aligned
• date/week conventions introduced market-specific risk

Main Lessons

One Model Representation

If a market updates the variable representation for forecast/upload purposes, then:

• transform
• predict
• decomp
• build output

must all use that same updated representation.

No Hidden Local State

Market implementations cannot rely on:

• local SharePoint paths
• analyst-specific Windows paths
• attached packages being present by accident
• untracked helper scripts

Explicit Market Conventions

China highlighted that week-start conventions matter. These assumptions must be encoded in config and logic, not left implicit.

Repository Impact

The China case directly informed:

• the repo split between product/package code and market code
• the market folder contract
• forbidden-pattern validation
• the need for future full clean-session checks

Current Local Repro

A local repro bundle now exists under:

markets/CN/store_visits/local-repro/

This includes a repeatable runner:

./markets/CN/store_visits/local-repro/run_cn_repro.sh

That runner demonstrates the local:

• transform
• updated model refit
• decomp

path from the supplied China sample artefacts.

Next Documentation Step

Once the real China implementation is migrated into markets/CN/store_visits, this page should be expanded into a true before/after example with exact technical detail.

Glossary

Market

A country or region-specific implementation, for example CN, IT, or UK.

KPI

A market-specific model/output surface such as store_visits.

Migrated

A market/KPI is migrated when the Git folder is approved as the authoritative implementation and local copies are no longer treated as the source of truth.

Source Of Truth

The repository version that should be used for validation, build, and review.

Forecast/Upload

The downstream workflow this repo is preparing artefacts for. In practice this may involve scenario-style future input handling rather than formal out-of-sample forecasting.

Decomp

Model decomposition logic used to produce contribution outputs from the model.

Clean Session

An R runtime that does not rely on accidental attached packages, untracked helper objects, or analyst-local interactive state.