The data pipeline

How a dispatch interval gets from AEMO's public archive into a solved optimisation problem. This page is the reference for the two data sources, every table and field the package reads, and the extraction stages between them.

Overview

        NEMWeb (AEMO public archive)
                 |
        +--------+--------+
        |                 |
   MMS archive       NEMDE archive
   monthly ZIPs      daily ZIPs of
   of CSV tables     per-interval XML
        |                 |
        v                 v
   DBManager        XMLCacheManager
   SQLite mirror    file cache on disk
        |                 |
        +--------+--------+
                 |
          RawInputsLoader          <- positioned on ONE interval
                 |
   +-------------+-------------+-------------+
   |             |             |             |
 UnitData    DemandData  InterconnectorData ConstraintData
   |             |             |             |
   +-------------+------+------+-------------+
                        |
                   SpotMarket           <- the JuMP model
                        |
                   dispatch!
                        |
        dispatch quantities, prices, binding constraints

Two sources, because neither is sufficient alone.

The MMS data model is AEMO's relational publication of the market: regional demand, published prices, unit registration, interconnector definitions and loss models. It is authoritative for anything that describes the market's configuration.

The NEMDE case files are the dispatch engine's own inputs, one XML document per five-minute interval. They carry the ten-band offers, FCAS trapeziums, initial conditions, ramp rates and generic constraints that NEMDE actually solved with. They are what make an exact reconstruction possible rather than an approximation.

Source 1: the MMS data model

Where it comes from

https://www.nemweb.com.au/Data_Archive/Wholesale_Electricity/MMSDM/
    {year}/MMSDM_{year}_{month}/MMSDM_Historical_Data_SQLLoader/
    DATA/PUBLIC_DVD_{TABLE}_{year}{month}010000.zip

One ZIP per table per month, each containing a single CSV. DBManager fetches them, unpacks them, and loads the columns the model needs into SQLite.

Why only some columns

Each table is declared with an explicit column list, and only those columns are loaded. PUBLIC_DVD_DISPATCHLOAD alone is around 100 MB per month with dozens of columns the dispatch model never reads. Naming the columns keeps the SQLite mirror small enough to query quickly and makes the model's data dependencies legible: if a column is not in the list, nothing in the package uses it.

Row filtering

MMS tables are versioned in four different ways, and the loader knows which applies to each:

FilterRuleUsed by
:settlement_daterows where SETTLEMENTDATE equals the intervalper-interval outcome tables
:start_endrows valid for the interval: START_DATE <= t < END_DATEregistration data with validity windows
:effective_verthe latest EFFECTIVEDATE/VERSIONNO effective at or before the intervalversioned reference data
:no_filterall rows; the table is staticpure reference tables

Getting this wrong is a quiet failure rather than a loud one — a stale VERSIONNO gives a plausible loss model for the wrong week — so the rule is attached to the table definition rather than left to each caller.

The tables

DISPATCHREGIONSUM — regional demand (per interval)

ColumnMeaning
SETTLEMENTDATEInterval ending, the key for every per-interval table
REGIONIDNSW1, QLD1, SA1, TAS1, VIC1
TOTALDEMANDRegional demand as published
DEMANDFORECASTThe forecast change over the interval
INITIALSUPPLYSupply at the start of the interval

Operational demand is built from INITIALSUPPLY and DEMANDFORECAST rather than taken from TOTALDEMAND, because that is what the dispatch engine balances.

DISPATCHPRICE — published prices (per interval)

ColumnMeaning
SETTLEMENTDATE, REGIONIDKey
ROPRegional Original Price for energy, $/MWh
RAISE6SECROP … LOWERREGROPROP for each of the ten FCAS services, $/MW

ROP, not RRP. ROP is the price the dispatch produced, before the administered-price and scaling rules that are applied afterwards. A dispatch model compared against RRP is being scored on the model plus those rules. See Prices and duals.

DUDETAILSUMMARY — unit registration (validity window)

ColumnMeaning
DUIDDispatchable unit identifier
START_DATE, END_DATEValidity window for this row
DISPATCHTYPEGENERATOR or LOAD
CONNECTIONPOINTIDTransmission connection point
REGIONIDMarket region
TRANSMISSIONLOSSFACTORTLF
DISTRIBUTIONLOSSFACTORDLF
SCHEDULE_TYPESCHEDULED, SEMI-SCHEDULED, NON-SCHEDULED
SECONDARY_TLFThe TLF of a bidirectional unit's other side

The combined loss factor is TLF × DLF. SECONDARY_TLF is why local_prices returns one row per (unit, dispatch_type) and not per unit: a battery's generating and consuming sides sit at the same connection point but carry different loss factors.

DUDETAIL — registered capacity (versioned)

DUID, EFFECTIVEDATE, VERSIONNO, REGISTEREDCAPACITY.

INTERCONNECTOR — topology (static)

INTERCONNECTORID, REGIONFROM, REGIONTO. Fixes the sign convention: a positive flow runs from REGIONFROM to REGIONTO.

INTERCONNECTORCONSTRAINT — limits and loss parameters (versioned)

ColumnMeaning
FROMREGIONLOSSSHAREShare of losses attributed to the sending region
LOSSCONSTANT, LOSSFLOWCOEFFICIENTCoefficients of the loss equation
ICTYPEREGULATED or MNSP
IMPORTLIMIT, EXPORTLIMITStatic limits

ICTYPE matters more than it looks. An MNSP is modelled as two directional links with their own loss factors; a regulated interconnector is one flow variable. Basslink was converted from MNSP to regulated during 2025, and because the type is read from the data rather than hard-coded, the same model handles both sides of that change.

LOSSMODEL and LOSSFACTORMODEL — the loss curve (versioned)

LOSSMODEL gives the MW breakpoints of the piecewise-linear loss curve (LOSSSEGMENT, MWBREAKPOINT). LOSSFACTORMODEL gives the per-region demand coefficients (REGIONID, DEMANDCOEFFICIENT) that make the curve demand-dependent.

Reconstructing the curve from these is one of two options; the other is to read NEMDE's own per-interval loss model out of the case file. See LOSS_MODEL_FROM_XML in Behavioural flags.

MNSP_INTERCONNECTOR — market network service providers (versioned)

LINKID, FROMREGION, TOREGION, FROM_REGION_TLF, TO_REGION_TLF, LHSFACTOR, MAXCAPACITY. One row per directional link.

DISPATCHINTERCONNECTORRES — published flows (per interval)

INTERCONNECTORID, MWFLOW, MWLOSSES. Not an input; a cross-check. The loss model was validated by reproducing MWLOSSES at the published MWFLOW.

Optional: DISPATCHLOAD — published unit targets (per interval)

Large — roughly 100 MB per month — and not in REQUIRED_TABLES, because the model takes initial MW and SCADA ramp rates from the case file instead. Download it when you want to compare the model's dispatch against AEMO's, which the BESS event study does.

Key columns: TOTALCLEARED (the published target), INITIALMW, AVAILABILITY, RAMPUPRATE, RAMPDOWNRATE, SEMIDISPATCHCAP, the FCAS enablement limits, and the per-service actual availabilities.

Optional: DISPATCHCONSTRAINT — published constraint outcomes (per interval)

Also large. CONSTRAINTID, RHS, LHS, MARGINALVALUE, VIOLATIONDEGREE.

MARGINALVALUE is AEMO's own shadow price for each constraint, and it is the strongest available check on the model's duals — the multipliers that enter the local-price decomposition are exactly these. The BESS analysis gates on it.

What is deliberately NOT downloaded

The MMS bid tables. In the archive they are PUBLIC_DVD_BIDDAYOFFER (~160 MB/month) and PUBLIC_DVD_BIDPEROFFER1 / BIDPEROFFER2 (~3.5 GB each).

They are not needed. Every offer the model uses — the ten price and volume bands, FCAS trapeziums, availabilities and ramp rates — is in the NEMDE case file, at per-interval rather than per-day resolution and already in the form the engine used. Attempting to download the bid tables is what makes a naive input download appear to hang.

Source 2: the NEMDE case files

Where they come from

https://www.nemweb.com.au/Data_Archive/Wholesale_Electricity/NEMDE/
    {year}/NEMDE_{year}_{month}/NEMDE_Market_Data/NEMDE_Files/
    NemSpdOutputs_{year}{month}{day}_loaded.zip

One ZIP per calendar day, holding one XML document per five-minute interval, named NEMSPDOutputs_{YYYYMMDD}{NNN}00.loaded where NNN is the three-digit interval number within the market day.

XMLCacheManager downloads a day's ZIP, unpacks it into the cache directory, and thereafter reads files straight off disk. A day with at least 200 cached files is treated as complete and skipped, so re-running a download costs a directory listing.

The market day is not the calendar day

A NEM market day runs 04:05 to 04:00. Intervals between 00:00 and 04:00 of a calendar day live in the previous day's ZIP, and populate_by_day! starts one day early to account for it. If the loader cannot find an interval, check that its market day is in the cache rather than its calendar day.

Size

Roughly 1.5 GB per day unpacked. A month is 40–100 GB. Download the days you need, not the month.

Structure

The parts the package reads, as paths under the document root:

NEMSubmission/
└── NemSpdInputs/
    ├── PeriodCollection/Period/
    │   └── TraderPeriodCollection/
    │       └── TraderPeriod[@TraderID]/
    │           └── TradeCollection/
    │               └── Trade[@TradeType, @PriceBand1..10, @BandAvail1..10,
    │                         @MaxAvail, @EnablementMin, @EnablementMax,
    │                         @LowBreakpoint, @HighBreakpoint,
    │                         @RampUpRate, @RampDnRate]
    ├── TraderCollection/
    │   └── Trader/TraderInitialConditionCollection/
    │       └── TraderInitialCondition[@InitialConditionID]
    ├── GenericConstraintCollection/
    │   └── GenericConstraint[@ConstraintID, @Type, @ViolationPrice]
    │       ├── LHSFactorCollection/{TraderFactor, InterconnectorFactor, RegionFactor}
    │       └── ...
    └── InterconnectorCollection/Interconnector/LossModelCollection/
        └── LossModel/SegmentCollection/Segment[@Limit, @Factor]

Trade — one offer, for one unit, for one service

@TradeType selects the service:

CodeServiceCodeService
ENOFenergy, generatorR5REraise regulation
LDOFenergy, scheduled loadL5RElower regulation
BDOFenergy, bidirectional unitR1SEraise 1 s
DROFwholesale demand responseL1SElower 1 s
R6SE / L6SEraise / lower 6 sR60S / L60Sraise / lower 60 s
R5MI / L5MIraise / lower 5 min

Each Trade carries ten @PriceBand and ten @BandAvail attributes — the offer stack — plus, for an FCAS trade, the four trapezium parameters (@EnablementMin, @LowBreakpoint, @HighBreakpoint, @EnablementMax) and @MaxAvail.

TraderInitialCondition — the unit's state entering the interval

@InitialConditionID selects the quantity: INITIALMW, SCADARAMPUPRATE, SCADARAMPDNRATE, the fast-start mode and time-in-mode, and the AGC status.

GenericConstraint — the SPD constraint set

Each carries its @Type (LE, GE, EQ), its right-hand side, its @ViolationPrice, and a left-hand side built from three kinds of factor:

FactorAttaches toUsed for
TraderFactora unit and a serviceunit terms in network and FCAS constraints
InterconnectorFactoran interconnectorflow terms
RegionFactora region and a serviceregional FCAS requirements

RegionFactor is why regional FCAS prices are recovered as factor-weighted sums of constraint duals rather than as duals of dedicated requirement constraints: there are no dedicated requirement constraints. See Prices and duals.

Violation prices

Read per interval from the case file rather than assumed. The constraint violation price hierarchy decides which constraint the dispatch breaks first when it cannot satisfy them all, and it is a property of the interval. CVP_FACTORS in NEMX.ZBenchmark holds fallbacks used only when a case file is unavailable.

Extraction

Stage 1 — acquire

using NEMX, Dates
const ZB = NEMX.ZBenchmark

db    = ZB.DBManager("data/nemx_2025_09/historical_mms.db")
cache = ZB.XMLCacheManager("data/nemx_2025_09/xml_cache")

ZB.populate!(db; start_year = 2025, start_month = 9,
             end_year = 2025, end_month = 9,
             tables = vcat(ZB.REQUIRED_TABLES, "DISPATCHLOAD", "DISPATCHCONSTRAINT"))

ZB.populate_by_day!(cache; start_year = 2025, start_month = 9, start_day = 2,
                           end_year = 2025, end_month = 9, end_day = 3)

Both are idempotent. From a script, --download does the same.

Stage 2 — position on an interval

loader = ZB.RawInputsLoader(cache, db)
ZB.set_interval!(loader, DateTime(2025, 9, 2, 12, 5))

RawInputsLoader holds both sources and a cursor. set_interval! parses that interval's XML document and remembers the timestamp; the MMS side is queried lazily, with the right filter rule per table.

Stage 3 — build the input classes

ClassReadsProduces
UnitDatacase-file trades and initial conditions, DUDETAILSUMMARY, DUDETAILoffer stacks, availabilities, ramp rates, FCAS trapeziums, fast-start profiles, unit info
DemandDataDISPATCHREGIONSUMregional operational demand
InterconnectorDataINTERCONNECTOR, INTERCONNECTORCONSTRAINT, MNSP_INTERCONNECTOR, LOSSMODEL, LOSSFACTORMODEL, or the case filedefinitions and the loss model
ConstraintDatacase-file generic constraints and violation pricesthe constraint set, its LHS factors, and the CVPs

Three transformations inside UnitData are worth knowing about, because each would be invisible in the output if it were wrong:

Loss-factor scaling. Case-file energy prices are pre-referred to the regional reference node. Ingestion multiplies by the combined loss factor to restore connection-point prices; the objective divides by the same factor to refer them back. The round trip is exact and is what makes the objective a reference-node objective. XML_PRICES_PRESCALED governs the first half.

FCAS trapezium construction. add_fcas_trapezium_constraints! turns the four breakpoints into the enablement constraints, and applies the preconditions that decide whether a unit may be enabled for a service at all. It mutates UnitData, so every FCAS getter must be called after it — which is why build_spot_market exists rather than each script open-coding the order.

Ramp-rate reconciliation. The effective rate is the lesser of the bid rate and the SCADA rate, in MW/h, over a window of rate × 5/60. A bidirectional unit's composite rate spans both directions.

Stage 4 — assemble and solve

market, inputs = ZB.dispatch_interval!(loader)

See Zonal benchmarking for what that does, pass by pass.

Inspecting the extraction

julia --project=. scripts/zbenchmark/export_interval_csvs.jl 2025-09-02T13:25

writes every extracted input for one interval to CSV. This is the first thing to run when an interval behaves unexpectedly, and what to attach to a bug report.

Storage

ItemSizeCommitted?
MMS SQLite mirror~1.4 GB per monthno
NEMDE case-file cache10–105 GB per monthno
Model output CSVs~200 MB per studyno
Network cases (.m)1.4 MB eachyes

Nothing needs to live inside the package: every script takes --data-dir, and a copy held elsewhere can be symlinked. See data/README.md.

Provenance and reuse

All market data is AEMO's, published through NEMWeb. Cite AEMO as the source of any market data underlying a result, and state the period — the archive is revised, so a result is reproducible only against a stated vintage.

The synthetic 2000-bus network is not AEMO data and is not a model of the actual transmission system; see the warning in Market data.