ZBenchmark API
Full reference for NEMX.ZBenchmark, the zonal dispatch reconstruction. See Zonal benchmarking for the workflow and Prices and duals for what the outputs mean.
NEMX.ZBenchmark — Module
NEMX.ZBenchmarkZonal (copper-plate) reconstruction of the NEM dispatch engine.
This submodule reproduces AEMO's five-minute regional dispatch from published data: it downloads the MMS data model and the NEMDE case files, turns them into tidy per-interval market inputs, assembles a single-interval linear program in JuMP, and solves it. Regional energy prices come out as the duals of the demand balance constraints and regional FCAS prices as factor-weighted sums of the generic-constraint duals, exactly as they do in the market engine it replicates.
It is the reference against which the network-resolved models of NEMX.NBenchmark are measured, so its default behaviour is deliberately conservative: every behavioural switch defaults to the setting that reproduces published prices, and none of them is changed implicitly.
Layout
| Folder | Contents |
|---|---|
data/ | AEMO data acquisition — MMS SQLite mirror, NEMDE XML case cache, and the loader that joins them per interval |
core/ | Market-input classes (units, demand, interconnectors, constraints), the SpotMarket container and its set_*! API, and result extraction |
prob/ | The optimisation itself — constraint assembly, dispatch!, and post-solve pricing |
Typical workflow
using NEMX, Dates
const ZB = NEMX.ZBenchmark
db = ZB.DBManager("data/historical_mms.db")
cache = ZB.XMLCacheManager("data/xml_cache")
# One-off: download a month of MMS tables and a day of NEMDE case files.
ZB.populate!(db; start_year = 2025, start_month = 9, end_year = 2025, end_month = 9)
ZB.populate_by_day!(cache; start_year = 2025, start_month = 9, start_day = 2,
end_year = 2025, end_month = 9, end_day = 2)
loader = ZB.RawInputsLoader(cache, db)
ZB.set_interval!(loader, DateTime(2025, 9, 2, 12, 5))
market = ZB.build_spot_market(loader) # assemble every input class
ZB.dispatch!(market)
ZB.get_energy_prices(market) # regional reference pricesbuild_spot_market is the packaged form of the assembly sequence that the reference implementation open-coded in each of its driver scripts; see build_spot_market for the individual steps and the flags that govern them.
Behavioural flags
All of these are Refs set at run time, e.g. ZB.LOSS_MODEL_FROM_XML[] = true. Defaults reproduce the validated benchmark; see the manual's flag reference for the evidence behind each.
| Flag | Default | Effect |
|---|---|---|
LOSS_MODEL_FROM_XML | true (env NEMX_LOSS_XML) | Take interconnector loss curves from the NEMDE case file rather than re-deriving them from MMS demand coefficients |
BDU_CROSS_SIDE_REG_LOWER_SUBTRACT | true | Subtract (rather than add) the cross-side regulation term in the lower joint-capacity constraint of a bidirectional unit |
ZONAL_MLF_KEEP_SCALING | false | Leave the energy bid stack scaled by the marginal loss factor instead of referring it to the regional reference node |
LAZY_LOSS_TIGHTENING | true | Lazily enforce SOS2 adjacency on the loss interpolation after an LP solve |
LOSS_TIGHTEN_TIME_LIMIT | 30.0 | Seconds allowed per loss-adjacency re-solve; nothing for no limit |
FCAS_DUAL_CVP_PRIORITY | false | Opt-in refinement of degenerate FCAS requirement duals by CVP priority |
DIRECTED_UNIT_PRICE_RELAX | false | Experimental dual-degeneracy resolution for directed units |
XML_PRICES_PRESCALED | true | Treat NEMDE case-file energy prices as pre-referred to the regional node |
SOLVER_FACTORY | HiGHS.Optimizer | Optimizer used by dispatch! |
NEMX.ZBenchmark.ConstraintData — Type
ConstraintData(loader::RawInputsLoader)Reads constraint-related inputs for the current interval. Mirrors constraints.ConstraintData. The generic (network/security/FCAS-requirement) constraints invoked by NEMDE for the interval are parsed once at construction.
NEMX.ZBenchmark.DBManager — Type
DBManager(db_path::String)Create (or open) a SQLite database at db_path that will hold AEMO MMS tables. Equivalent to nempy's mms_db.DBManager(connection=...).
NEMX.ZBenchmark.DemandData — Type
DemandData(loader::RawInputsLoader)Reads regional demand for the current interval. Mirrors demand.DemandData.
NEMX.ZBenchmark.InterconnectorData — Type
InterconnectorData(loader::RawInputsLoader)Reads interconnector definitions and the loss model for the current interval. Mirrors interconnectors.InterconnectorData.
NEMX.ZBenchmark.MarketInputs — Type
MarketInputsThe four per-interval input classes, kept together so a caller that needs one of them after assembly (unit info for a region map, violation prices for a report) does not have to rebuild it.
Fields
units::UnitData— bids, availability, ramp rates and FCAS trapeziumsdemand::DemandData— regional operational demandinterconnectors::InterconnectorData— definitions and the loss modelconstraints::ConstraintData— violation prices and generic (SPD) constraintscvp::Dict{String,Float64}— constraint violation prices, by category
NEMX.ZBenchmark.RawInputsLoader — Type
RawInputsLoader(xml_cache_manager::XMLCacheManager, mms_db_manager::DBManager)Bundle the NEMDE XML cache and the MMS SQLite database behind one interface. Call set_interval! before reading any inputs.
NEMX.ZBenchmark.SpotMarket — Type
SpotMarket(; market_regions::Vector{String}, unit_info::DataFrame)Create an empty spot-market model for the given NEM regions and unit metadata. unit_info must have columns unit, dispatch_type, region (one row per unit-direction; BDUs appear twice). Mirrors markets.SpotMarket.
NEMX.ZBenchmark.UnitData — Type
UnitData(loader::RawInputsLoader)Parse all per-unit inputs for the loader's current interval. Construction does the XML parsing once; the get_* accessors then return tidy DataFrames. Mirrors nempy's units.UnitData.
NEMX.ZBenchmark.XMLCacheManager — Type
XMLCacheManager(cache_folder::String)Manage a local cache of NEMDE XML case files. Equivalent to nempy's xml_cache.XMLCacheManager. The folder is created if it does not exist.
NEMX.ZBenchmark.BDU_CROSS_SIDE_REG_LOWER_SUBTRACT — Constant
BDU_CROSS_SIDE_REG_LOWER_SUBTRACTSign of the cross-side regulation term in the contingency joint-capacity LOWER constraint of a bidirectional unit (BDU).
A BDU bids energy on both a generator and a load side, and may bid the regulation services on the side opposite to the contingency trapezium being constrained. nempy adds that cross-side term with +1, which RELAXES the lower constraint. NEMDE subtracts it: a lower service bid on either side consumes the same downward headroom between the net energy target and EnablementMin.
Validated against the NEMDE case files themselves (120 intervals, 1 532 BDU lower/raise trapezium checks). Evaluating both signs on NEMDE's own published TraderSolution targets:
| slope | sign | violations | exactly binding |
|---|---|---|---|
| lower | -1 (NEMDE) | 0 | 381 |
| lower | +1 (nempy) | 0 | 0 |
| raise | +1 (both) | 0 | 221 |
| raise | -1 | 0 | 0 |
A constraint that is exactly binding in NEMDE's own solution is one NEMDE enforced; the +1 form is strictly weaker and never binds, so it cannot be the constraint NEMDE solved. The raise slope is unaffected — our existing +1 there is already the binding form.
Worked example, WANDB1 at 2025-09-02 11:35 (net energy -25 MW, EnablementMin -75 MW): NEMDE sets lower_5min = 20 and lower_reg = 30 (load side), giving -25 - 20 - 30 = -75, exactly EnablementMin. With the +1 sign the model instead reaches lower_5min = 44 (the unit's full MaxAvail), 24 MW of too-cheap Queensland lower supply. Because Queensland's lower family is written on QNI flow (F_Q++BCDM_L5/_L6/_L60, see below), that surplus displaced 4.67 MW of QNI flow and moved the marginal lower-service provider, which is what depressed the published Queensland lower prices.
true (default) reproduces NEMDE. Set to false to restore the legacy nempy sign — required to keep the July-2024 benchmark bit-identical.
NEMX.ZBenchmark.CVP_FACTORS — Constant
CVP_FACTORSFallback constraint violation prices, as multiples of the market price cap.
The constraint violation price hierarchy decides which constraint the dispatch breaks first when it cannot satisfy them all: a higher price means a constraint is relaxed later. These are FALLBACKS only — the actual per-interval values come from the NEMDE case file (see xml_violation_prices) and override them, so this table matters only when a case file is unavailable.
NEMX.ZBenchmark.DEFAULT_REGIONS — Constant
DEFAULT_REGIONSThe five NEM market regions, in the order used throughout the package.
NEMX.ZBenchmark.DIRECTED_UNIT_PRICE_RELAX — Constant
Resolve the dual degeneracy created by DIRECTED-unit generic constraints in favour of NEMDE's pricing convention.
A GE generic constraint whose LHS is a single trader energy term (e.g. NSA_S_TB3_40: TORRB3 energy >= 40, an SA system-strength direction) forces a unit on. When the forced level coincides with a boundary of the unit's offer bands, the LP has multiple optimal dual vertices: the dual mass can sit on the direction constraint (making the directed unit's expensive band price-setting) or on the network constraint the direction supports. NEMDE selects the vertex in which the directed unit's forced energy is NOT price-setting — observed 2025-09-03 00:05, where NEMDE prices SA1 at $149.00 (rate-of-change-of- frequency constraint dual −83.32) while the alternative vertex prices it at $65.73. Relaxing each BINDING such constraint by an infinitesimal ε in the pricing re-solve places the directed unit strictly inside its lower band and uniquely selects NEMDE's vertex; the primal dispatch is unchanged (ε = 0.1 kW).
EXPERIMENTAL — disabled by default. Investigation of 2025-09-03 00:05 showed the SA multiplicity is multi-dimensional (the SVML_ZERO outage constraint, V_S_NIL_ROCOF and the direction constraint exchange dual mass in a 2-plus-dimensional optimal face), so this one-constraint relaxation does not reliably select NEMDE's vertex and is kept off pending a principled lexicographic pricing rule.
NEMX.ZBenchmark.FCAS_DUAL_CVP_PRIORITY — Constant
Resolve dual degeneracy among FCAS requirement constraints in AEMO's favour.
Where several FCAS requirement constraints share a term (typically the same interconnector flow) the LP pins only the SUM of their duals; the split between them is a free parameter and the solver lands on an arbitrary corner. Observed 2025-09-04 08:20 in Queensland, whose three lower-service constraints are structurally identical apart from the service they carry:
F_Q++BCDM_L6 : NSW1-QLD1 flow + QLD lower_6s >= RHS (CVP 162 400)
F_Q++BCDM_L60 : NSW1-QLD1 flow + QLD lower_60s >= RHS (CVP 121 800)Both bind, the primal is exact (400 MW each, matching NEMDE), and the duals satisfy mu6 + mu60 = 69.63 with mu6 in [19.95, 37.63], mu60 in [32.00, 49.68]. Our solver reports (19.95, 49.68); AEMO publishes (37.63, 32.00).
AEMO's values are reproduced by allocating dual mass in DESCENDING VIOLATION PRICE order — the constraint that is more expensive to violate takes precedence: the highest-CVP degenerate constraint receives its maximum feasible dual (the RIGHT derivative of the objective with respect to its right-hand side) and the remaining degenerate constraints receive their minimum (the LEFT derivative).
Degeneracy is detected per constraint as a KINK: right and left derivatives differ. Non-degenerate constraints keep the solver's dual untouched, so this pass is a no-op wherever the price was already unique — which is every FCAS constraint in the July-2024 benchmark.
This rule reproduced AEMO exactly on the interval it was derived from (2025-09-04 08:20) but was falsified by a 1 000-interval sweep: overall FCAS disparity INCREASED, Tasmania lower-regulation mismatches appeared where there had been none, and the Queensland lower-service mismatch persisted. It was over-fitted to a single interval. The kept code is a diagnostic for identifying degenerate FCAS requirement groups, not a pricing rule; it is no longer called from dispatch!.
The sweep also showed why it could not have worked in general: the residual Queensland mismatch is NOT a dual-degeneracy problem at all (see the interconnector-coupling note in _add_generic_constraints!).
Costs two extra LP re-solves per binding FCAS requirement.
NEMX.ZBenchmark.LAZY_LOSS_TIGHTENING — Constant
Diagnostic switch. true (default) reproduces AEMO/nempy behaviour: SOS2/SOS1 adjacency is lazily enforced on the interconnector loss interpolation. Set to false ONLY to isolate the pure-LP relaxation when debugging a slow or suspicious interval — the resulting prices are NOT benchmark-valid at negative regional prices (that is exactly the case the tightening exists to fix).
NEMX.ZBenchmark.LAZY_LOSS_VERBOSE — Constant
Print per-iteration diagnostics from the lazy loss tightening.
NEMX.ZBenchmark.LOSS_MODEL_FROM_XML — Constant
Source the interconnector loss curve from NEMDE's OWN per-interval loss model (the <LossModel>/<Segment> collections in the case file) instead of re-deriving it from the MMS demand coefficients.
true (DEFAULT since the September-2025 loss validation) reproduces NEMDE's published InterconnectorSolution/@Losses exactly. Set NEMX_LOSS_XML=0, or nemjl.LOSS_MODEL_FROM_XML[] = false, to fall back to the MMS derivation.
Validation that promoted this to the default (22 matched September-2025 intervals, 1 210 price comparisons, same intervals both ways):
| metric | MMS derivation | NEMDE XML curves |
|---|---|---|
energy MAE ($/MWh) | 1.7125 | < 0.0001 |
| energy max abs error | 15.84 | 0.00094 |
FCAS MAE ($/MW) | 0.0510 | 0.0295 |
| FCAS max abs error | 10.09 | 3.78 |
| QLD lower-service FCAS MAE | 0.2817 | 0.0990 |
July-2024 is unchanged to 7 decimal places: mean price error -0.0798602820 (MMS) vs -0.0798602591 (XML), a shift of 2e-8 $/MWh. Its binding flows sit where the two curves agree, so the change is confined to the intervals the MMS derivation actually got wrong. NOTE this is no longer BIT-identical to the reference implementation, which is expected: the model now matches AEMO's own loss model rather than nempy's re-derivation of it.
The MMS derivation's residual is a purely LINEAR error in the demand-dependent coefficient — measured at 0.014 MW per MW of flow on V-SA, i.e. up to 14.8 MW across its range — which the NEMDE segments make unnecessary because they already embed that interval's demand term.
NEMX.ZBenchmark.LOSS_TIGHTEN_TIME_LIMIT — Constant
Wall-clock seconds allowed for each loss-adjacency re-solve. AEMO loss curves carry 60-120 breakpoints per link; when several links need tightening at once (typical when regional prices go deeply negative) the resulting MIP can fail to close. Bounding it makes the run deterministic: if the limit is hit the model reverts to the LP relaxation for that interval and still returns valid duals, instead of returning NaN or pricing off a poor incumbent. Set to nothing for no limit.
NEMX.ZBenchmark.REQUIRED_TABLES — Constant
REQUIRED_TABLESThe MMS tables the dispatch model actually needs.
All of them are small — under 10 MB for a month — so a full-month download of this set completes in seconds. The large tables DISPATCHLOAD (~100 MB) and DISPATCHCONSTRAINT (~150 MB) are deliberately NOT here: the model takes initial MW and SCADA ramp rates from the NEMDE case file, so they are needed only when comparing the model's dispatch and duals against AEMO's published ones.
Pass tables = vcat(REQUIRED_TABLES, "DISPATCHLOAD", "DISPATCHCONSTRAINT") to populate! when you want them.
NEMX.ZBenchmark.SERVICE_ROP_COL — Constant
SERVICE_ROP_COLMap from this package's service names to the corresponding DISPATCHPRICE column holding AEMO's published ROP — the "original" regional price, before any post-dispatch scaling or capping. ROP is the correct comparison target for a dispatch reconstruction: RRP has had administered-price and scaling rules applied to it that are outside the dispatch problem.
NEMX.ZBenchmark.ZONAL_MLF_KEEP_SCALING — Constant
ZONAL_MLF_KEEP_SCALINGSuppress the marginal-loss-factor referral in the zonal energy objective.
get_processed_bids multiplies case-file energy prices by the unit's loss factor, restoring connection-point prices; this engine normally divides by the same factor, so the round trip is an exact no-op and the objective is referred to the regional reference node. That is the validated benchmark, and it is what false (the default) reproduces bit-for-bit.
Setting true keeps the scaled bid stack – prices stay at the connection point and are never referred back. This exists for ONE purpose: to generate the MLF-scaled reference dispatch that the network-formulation study is measured against (scripts/nbenchmark/run_network_day.jl). Comparing the nodal formulations against AEMO's published ROP conflates two different things, the error of the power-flow model and the difference between a zonal and a nodal market; comparing them against a zonal run built on the SAME bid stack isolates the first.
It must never be set when reproducing the benchmark itself. Nothing in script/run_historical_dispatch*.jl touches it, and the default leaves those results unchanged.
NEMX.ZBenchmark._parse_mms_csv — Method
_parse_mms_csv(raw::String, keep_columns) -> DataFrameDecode AEMO's report-format CSV. The header ("I") row gives the column names; the data ("D") rows hold the values. Only keep_columns are returned.
NEMX.ZBenchmark.add_fcas_trapezium_constraints! — Method
add_fcas_trapezium_constraints!(u::UnitData)Build the (scaled, filtered) FCAS trapezium table from the processed bids. Must be called AFTER get_processed_bids (mirrors nempy's call-order rule).
NEMX.ZBenchmark.as_float — Method
as_float(x) -> Float64Coerce a value read out of a CSV, SQLite column or DataFrame cell to Float64, returning NaN for missing and for anything unparseable.
MMS tables are typed inconsistently between vintages — the same column arrives as Int, Float64, String or missing depending on the month — so every numeric read from published data goes through this rather than float().
NEMX.ZBenchmark.build_spot_market — Method
build_spot_market(loader; regions = DEFAULT_REGIONS, fast_start = true)
-> (market::SpotMarket, inputs::MarketInputs)Assemble a SpotMarket for the interval loader is currently positioned on, without solving it.
loader must already have been moved onto the interval with set_interval!; this function does not move it, so that a caller sweeping a series controls the interval cursor itself.
Arguments
loader::RawInputsLoader: joined MMS + NEMDE source for the interval.
Keywords
regions::Vector{String} = DEFAULT_REGIONS: market regions to model. Passing a subset is supported but changes the problem — interconnectors to an omitted region are dropped, not fixed at their historical flow.fast_start::Bool = true: apply the fast-start inflexibility profiles on the first pass ("fast_start_first_run"). The second pass is applied bydispatch_interval!after the unconstrained solve, because it needs that solve's dispatch to classify each fast-start unit's end mode.
Returns
A tuple of the assembled market and the input classes used to build it.
Notes
The order of the set_*! calls below is the order the reference market engine applies them and is not arbitrary: add_fcas_trapezium_constraints! mutates units in place and must precede every FCAS getter, and the ramp constraints must be set before the joint-ramping constraints that reuse the same SCADA rates.
NEMX.ZBenchmark.column_or_nan — Method
column_or_nan(row, name::Symbol) -> Float64Read column name from a DataFrameRow, or NaN if the column is absent.
A DataFrameRow supports neither get(row, :col, default) nor indexing by a missing column, and MMS schemas vary between vintages, so every published-data read goes through this.
NEMX.ZBenchmark.dispatch! — Method
dispatch!(m::SpotMarket; allow_over_constrained_dispatch_re_run=false, kwargs...)Build the LP from all inputs set so far, solve with HiGHS, and store dispatch quantities and regional energy prices (the demand-constraint duals). With allow_over_constrained_dispatch_re_run=true the OCD rerun is replicated: if a recovered price violates the market ceiling/floor AND a generic constraint is violated, the violated constraints' RHS are relaxed by (violation + 0.01) and the LP is re-priced — exactly nempy's procedure (no price clamping).
NEMX.ZBenchmark.dispatch_interval! — Method
dispatch_interval!(loader; regions = DEFAULT_REGIONS, fast_start = true,
over_constrained_rerun = :auto,
energy_market_floor_price = -1000.0,
fcas_market_ceiling_price = 1000.0)
-> (market::SpotMarket, inputs::MarketInputs)Assemble and fully solve one dispatch interval, including both fast-start passes and the over-constrained-dispatch (OCD) re-run.
This is the function a caller wants unless it needs to intervene between the passes; build_spot_market plus manual dispatch! calls is the escape hatch for that case.
Arguments
loader::RawInputsLoader: positioned on the interval viaset_interval!.
Keywords
regions::Vector{String} = DEFAULT_REGIONS: market regions to model.fast_start::Bool = true: run the two-pass fast-start procedure. The first pass solves without inflexibility profiles applied to their end modes; the result classifies each fast-start unit, and the second pass re-imposes ramp and joint-ramping constraints under"fast_start_second_run". Setting this tofalsesolves a single pass and is not benchmark-valid for intervals with a starting fast-start unit — it exists for speed in tests.over_constrained_rerun::Union{Symbol,Bool} = :auto: whether to re-price under the OCD rule.:autofollows the case file's own flag viais_over_constrained_dispatch_rerun, which is what the market engine does;true/falseforce it either way.energy_market_floor_price::Real = -1000.0: market floor used by the OCD re-price.fcas_market_ceiling_price::Real = 1000.0: FCAS ceiling used by the OCD re-price. The energy ceiling is taken from the case file's own value of VoLL (cvp["voll"]) rather than hard-coded, so a market price cap that changes between financial years is picked up from the data.
Returns
The solved market and its inputs.
NEMX.ZBenchmark.get_bid_ramp_rates — Method
get_bid_ramp_rates(u::UnitData) -> DataFrameBid (offer) ramp rates in MW/h with the unit's initial output: unit, dispatch_type, ramp_up_rate, ramp_down_rate, initial_output.
NEMX.ZBenchmark.get_binding_generic_constraints — Method
get_binding_generic_constraints(m::SpotMarket) -> DataFrameDiagnostic: shadow price, solved LHS, RHS and binding status of every generic constraint in the last solved model. The services column lists the FCAS (or energy) services the constraint couples, so FCAS regional requirement prices can be filtered directly (e.g. filter(r -> !isempty(r.services), df) or by the F_/D_ set-name prefix); dual is the marginal price ($/MW) of the requirement.
NEMX.ZBenchmark.get_constraint_violation_prices — Method
get_constraint_violation_prices(c::ConstraintData) -> Dict{String,Float64}Return the violation-price for each soft-constraint category. Mirrors get_constraint_violation_prices. Each value is the $/MW penalty applied to that constraint's slack variable in the objective.
NEMX.ZBenchmark.get_contingency_services — Method
get_contingency_services(u::UnitData) -> DataFrameTrapeziums for the CONTINGENCY services (1s/6s/60s/5min raise & lower).
NEMX.ZBenchmark.get_energy_prices — Method
get_energy_prices(m::SpotMarket) -> DataFrameregion, price — the demand-constraint duals (regional reference prices).
NEMX.ZBenchmark.get_fast_start_profiles_for_dispatch — Method
get_fast_start_profiles_for_dispatch(u::UnitData; unconstrained_dispatch=nothing)First run (no dispatch given): returns unit, current_mode, min_loading. Second run (pass the first run's get_unit_dispatch result): evolves each fast-start unit's mode over the 5-minute interval (committing mode-0 units with non-zero unconstrained dispatch) and returns unit, end_mode, time_in_end_mode, mode_two_length, mode_four_length, min_loading, time_since_end_of_mode_two. Mirrors nempy's _update_modes.
NEMX.ZBenchmark.get_fcas_max_availability — Method
get_fcas_max_availability(u::UnitData) -> DataFrameunit, dispatch_type, service, max_availability (post-scaling).
NEMX.ZBenchmark.get_fcas_prices — Method
get_fcas_prices(m::SpotMarket) -> DataFramePlaceholder for FCAS service marginal prices.
NEMX.ZBenchmark.get_fcas_regulation_trapeziums — Method
get_fcas_regulation_trapeziums(u::UnitData) -> DataFrameTrapeziums for the REGULATION services (raise_reg, lower_reg).
NEMX.ZBenchmark.get_fcas_requirements — Method
get_fcas_requirements(c::ConstraintData) -> DataFrameDEPRECATED in favour of get_generic_constraints. Regional FCAS requirement constraints are now handled uniformly as generic constraints (their RegionFactor terms), so this returns an empty frame to avoid double-counting. Kept only for API parity with the earlier example.
NEMX.ZBenchmark.get_generic_constraints — Method
get_generic_constraints(c::ConstraintData) -> (rhs_and_type, unit_lhs,
interconnector_lhs, region_lhs)Return the invoked generic constraints and their LHS factor tables. These bundle nempy's get_rhs_and_type_excluding_regional_fcas_constraints, get_unit_lhs, get_interconnector_lhs and the regional-FCAS terms — here kept together because the NEMDE XML supplies them uniformly per constraint id.
NEMX.ZBenchmark.get_initial_unit_output — Method
get_initial_unit_output(u::UnitData) -> DataFrameunit, initial_output (MW) at the start of the interval.
NEMX.ZBenchmark.get_interconnector_definitions — Method
get_interconnector_definitions(d::InterconnectorData) -> DataFrameinterconnector, from_region, to_region, min, max, from_region_loss_share. Mirrors get_interconnector_definitions.
NEMX.ZBenchmark.get_interconnector_loss_model — Method
get_interconnector_loss_model(d::InterconnectorData) -> (loss_functions, break_points)Return the piecewise-linear loss support points. loss_functions and break_points together let the market model build a convex piecewise-linear loss curve via SOS2 weights. Mirrors get_interconnector_loss_model.
NEMX.ZBenchmark.get_operational_demand — Method
get_operational_demand(d::DemandData) -> DataFrameReturn region, demand (MW) for the current interval. Mirrors get_operational_demand.
NEMX.ZBenchmark.get_processed_bids — Method
get_processed_bids(u::UnitData) -> (volume_bids, price_bids)Return the TEN-band volume and price bids as two wide DataFrames with columns unit, dispatch_type, service, "1"…"10". Before formatting, FCAS bids are scaled for AGC enablement limits, AGC (SCADA) ramp rates and UIGF, and the AEMO FCAS-enablement preconditions are enforced — bids failing them are REMOVED, exactly as nempy's get_processed_bids does.
Note on loss factors: nempy multiplies the XML energy prices by the combined loss factor (undoing the XML's referral to the regional node) and the market model then divides by the same loss factor when building the objective — the two operations cancel, so the objective coefficients equal the raw XML prices. We use the XML prices directly.
NEMX.ZBenchmark.get_published_rops — Method
get_published_rops(db::DBManager, interval::DateTime)
-> Dict{Tuple{String,String},Float64}AEMO's published ROP for every (region, service) in one dispatch interval, read from the DISPATCHPRICE table of the MMS mirror.
Returns an empty dictionary if the table is absent or holds no row for the interval, so a caller sweeping a series can score the intervals it has without special-casing gaps.
NEMX.ZBenchmark.get_regional_fcas_prices — Method
get_regional_fcas_prices(market::SpotMarket)
-> Dict{Tuple{String,String},Tuple{Float64,Int}}Regional marginal FCAS prices ($/MW) recovered from the solved LP, keyed by (region, service).
Regional FCAS requirements are not separate objects in this model: they are SPD generic constraints whose left-hand side carries RegionFactor terms, one per (region, service). By LP duality the marginal value of one more MW of a region's requirement for a service is
price(region, service) = Σ over constraints factor × constraint dualwhich is the same rule that makes a regional energy price the dual of that region's demand constraint.
Returns
For each (region, service), a tuple of the price and the number of constraint terms that contributed to it. The count matters: it distinguishes a price of 0.0 because nothing bound from a price of 0.0 because the model carries no requirement for that pair at all.
NEMX.ZBenchmark.get_scada_ramp_rates — Method
get_scada_ramp_rates(u::UnitData; include_initial_output=false) -> DataFrameSCADA telemetered ramp rates (MW/h) and, optionally, INITIALMW. Rows where both rates are missing are dropped (mirrors nempy's get_scada_ramp_rates).
NEMX.ZBenchmark.get_table — Method
get_table(m::DBManager, name::String; interval=nothing) -> DataFrameReturn rows of an MMS table, filtered to interval (an AbstractString such as "2024/07/01 12:35:00", or a DateTime) according to that table's filter rule. With interval === nothing the whole table is returned.
NEMX.ZBenchmark.get_unit_bid_availability — Method
get_unit_bid_availability(u::UnitData) -> DataFrameEnergy MAXAVAIL cap per (unit, dispatchtype). Mirrors `getunitbidavailability` (post-2023 behaviour: semi-scheduled units keep their bid availability constraint alongside the UIGF constraint).
NEMX.ZBenchmark.get_unit_dispatch — Method
get_unit_dispatch(m::SpotMarket) -> DataFrameunit, dispatch_type, service, dispatch (MW) for the solved interval.
NEMX.ZBenchmark.get_unit_info — Method
get_unit_info(u::UnitData) -> DataFrameunit, dispatch_type, region, loss_factor for every unit-direction with bids.
NEMX.ZBenchmark.get_unit_uigf_limits — Method
get_unit_uigf_limits(u::UnitData) -> DataFrameUnconstrained Intermittent Generation Forecast caps for SEMI-SCHEDULED units: unit, capacity (from the TraderPeriod @UIGF).
NEMX.ZBenchmark.get_violation_costs — Method
get_violation_costs(c::ConstraintData) -> Float64Base violation cost for generic constraints. The per-constraint penalty actually used is each constraint's own violation_price from the XML; this value is a fallback for constraints with no published violation price.
NEMX.ZBenchmark.is_over_constrained_dispatch_rerun — Method
is_over_constrained_dispatch_rerun(c::ConstraintData) -> BoolTrue when AEMO actually used the over-constrained-dispatch (OCD) rerun for this interval. Mirrors nempy exactly: the flag is 'OCD' appearing in the loaded case FILE NAME (AEMO publishes a NEMSPDOutputs_..._OCD.loaded variant only when the rerun process ran). The previous heuristic — "any constraint carries a positive violation price" — was true for nearly every interval and wrongly routed all intervals through the OCD price-capping path.
NEMX.ZBenchmark.link_interconnectors_to_generic_constraints! — Method
Attach interconnector LHS terms (set, interconnector, factor).
NEMX.ZBenchmark.link_regions_to_generic_constraints! — Method
Attach regional (FCAS) LHS terms (set, region, service, factor).
NEMX.ZBenchmark.link_units_to_generic_constraints! — Method
Attach unit LHS terms (set, unit, service, factor) to the generic constraints.
NEMX.ZBenchmark.load_interval! — Method
load_interval!(m::XMLCacheManager, interval) -> XMLCacheManagerLoad the cached NEMDE XML for a dispatch interval (a DateTime or AEMO time string) into memory, ready for the unit/interconnector parsers. Searches the cache folder for the matching file by interval number, falling back to a glob on the interval timestamp.
NEMX.ZBenchmark.local_prices — Method
local_prices(market::SpotMarket, unit_info::DataFrame) -> DataFrameThe local (connection-point) energy price of every unit, with the shadow-price adjustment that produced it.
A participant is settled at its region's reference price but dispatched against its local price. Writing the Lagrangian of the dispatch LP, an interior offer band of unit u prices at
π_u = λ_r(u) + Σ_c f_uc μ_cwhere λ_r is the regional demand-constraint dual, f_uc the unit's factor in generic constraint c, and μ_c that constraint's dual. The identity holds for generation and load bands alike — the direction's sign is carried inside the LP, in the -1 with which load energy enters both the regional balance and the constraint left-hand sides, not in the price.
The engine's objective is written in reference-node dollars (it divides the connection-point bid stack by the unit's loss factor), so π_u as computed above is a reference-node price; multiplying by the loss factor puts it at the connection point, which is the basis on which offers are submitted.
Arguments
market: a solvedSpotMarket.unit_info: the frame returned byget_unit_info, which supplies each unit-direction's region and loss factor.
Returns
One row per (unit, dispatch_type) — not per unit. A bidirectional unit's two sides share a reference-node local price but carry different loss factors (the generator side takes SECONDARY_TLF), so collapsing the directions would quote a charging battery's price on its discharging loss factor.
Columns: unit, dispatch_type, region, loss_factor, rrp, adjustment, local_price_ref, local_price_cp, n_constraint_terms, n_binding, and the single largest contributing term as top_set, top_factor, top_dual, top_contribution.
NEMX.ZBenchmark.mnsp_bids_active — Method
True when MNSP link bids should be modelled for interval (see MNSPBIDLIMITS).
NEMX.ZBenchmark.populate! — Method
populate!(m::DBManager; start_year, start_month, end_year, end_month,
tables=REQUIRED_TABLES)Download the requested MMS tables for each month in the inclusive range and load them into SQLite. Defaults to the small REQUIRED_TABLES the model needs (a few MB total). Mirrors DBManager.populate. Settlement-date tables are accumulated across all months; reference/effective-version tables only need the final month.
NEMX.ZBenchmark.populate_by_day! — Method
populate_by_day!(m; start_year, start_month, start_day, end_year, end_month, end_day)Download and unzip every daily NEMDE bundle in the inclusive date range into the cache folder. Mirrors XMLCacheManager.populate_by_day. Each daily zip expands to ~288 XML files (one per 5-minute interval) and is large (~1.5 GB/day uncompressed), hence the data-volume warning in the example script.
NEMX.ZBenchmark.populate_from_local! — Method
populate_from_local!(m::DBManager, folder::String; tables=keys(MMS_TABLES))Load MMS tables from MANUALLY-downloaded files instead of the network. Point folder at a directory containing AEMO's PUBLIC_DVD_<TABLE>_YYYYMM010000.CSV files (unzipped) and/or the original .zip archives. Each recognised table is parsed and written to SQLite. Use this when automated download is blocked — see MANUAL_DOWNLOAD.md.
NEMX.ZBenchmark.set_demand_constraints! — Method
Set the regional operational demand the dispatch must balance (equality, soft).
NEMX.ZBenchmark.set_energy_and_regulation_capacity_constraints! — Method
Add the energy↔regulation joint-capacity (regulation trapezium) constraints.
NEMX.ZBenchmark.set_fast_start_constraints! — Method
set_fast_start_constraints!(m, profiles; violation_cost)Apply the fast-start dispatch inflexibility profiles (second dispatch run). profiles needs unit, end_mode, time_in_end_mode, mode_two_length, mode_four_length, min_loading. Mirrors set_fast_start_constraints.
NEMX.ZBenchmark.set_fcas_max_availability! — Method
Cap each FCAS service at its offered max availability (soft constraint).
NEMX.ZBenchmark.set_fcas_requirements_constraints! — Method
Add regional FCAS requirement constraints (service, region, volume, type).
NEMX.ZBenchmark.set_generic_constraints! — Method
Register the generic constraints' RHS and sense (set, rhs, type, violation_price).
NEMX.ZBenchmark.set_interconnector_losses! — Function
Register the piecewise-linear loss support points (per interconnector LINK).
NEMX.ZBenchmark.set_interconnectors! — Method
Register interconnector definitions (one row per LINK; MNSPs are split).
NEMX.ZBenchmark.set_interval! — Method
set_interval!(l::RawInputsLoader, interval)Point the loader (and its XML cache) at a particular 5-minute dispatch interval. interval may be a DateTime or an AEMO time string such as "2024/07/01 12:35:00". Mirrors RawInputsLoader.set_interval.
NEMX.ZBenchmark.set_joint_capacity_constraints! — Method
Add the contingency-FCAS joint-capacity (trapezium) constraints.
NEMX.ZBenchmark.set_joint_ramping_constraints_reg! — Method
set_joint_ramping_constraints_reg!(m, scada_ramp_rates; fast_start_profiles,
run_type, violation_cost)Couple regulation FCAS to the unit's SCADA ramp window (FCAS MODEL IN NEMDE section 6.1). Only units actually BIDDING the regulation service receive the constraints. scada_ramp_rates needs unit, scada_ramp_up_rate, scada_ramp_down_rate, initial_output.
NEMX.ZBenchmark.set_tie_break_constraints! — Function
Set the tiny tie-break penalty that equalises equally-priced dispatch.
NEMX.ZBenchmark.set_unconstrained_intermittent_generation_forecast_constraint! — Method
Cap semi-scheduled (wind/solar) units at their UIGF forecast (soft constraint).
NEMX.ZBenchmark.set_unit_bid_capacity_constraints! — Method
Cap each unit-direction's energy dispatch at its bid-in availability (soft).
NEMX.ZBenchmark.set_unit_price_bids! — Method
Set the 10-band price bids (unit, dispatch_type, service, "1"…"10").
NEMX.ZBenchmark.set_unit_ramp_rate_constraints! — Function
set_unit_ramp_rate_constraints!(m, ramp_rates, scada_ramp_rates;
fast_start_profiles=nothing,
run_type="no_fast_start_units", violation_cost)Limit energy dispatch movement from the unit's initial output over the 5-minute interval. ramp_rates are the BID ramp rates (unit, dispatch_type, ramp_up_rate, ramp_down_rate, initial_output, MW/h); the binding rate is the lesser of the bid and SCADA rates. Fast-start adjustments mirror nempy's ramprateprocessing: first run drops units starting in modes 0–2; second run drops units ending in modes 0–2 and re-rates units freshly out of mode two.
NEMX.ZBenchmark.set_unit_volume_bids! — Method
Set the 10-band volume bids (unit, dispatch_type, service, "1"…"10").
NEMX.ZBenchmark.unit_constraint_terms — Method
unit_constraint_terms(market::SpotMarket)
-> (contributions, binding)Per-unit shadow-price terms of the local price.
contributions[unit] is a vector of (set, factor, dual, factor * dual), one entry per generic constraint carrying an energy factor for that unit with a non-zero dual. Only energy factors enter: an FCAS factor prices reserve, not the energy a unit is dispatched to produce or consume. binding[set] records whether the constraint was actually binding, so a real contribution can be told apart from a residual dual on a slack constraint.
NEMX.ZBenchmark.xml_fast_start_parameters — Method
xml_fast_start_parameters(m) -> DataFramePer-unit fast-start dispatch-inflexibility-profile parameters, read from the Trader attributes @MinLoadingMW @CurrentMode @CurrentModeTime @T1..@T4. @WhatIfCurrentMode(Time) overrides apply when present (mirrors nempy's get_unit_fast_start_parameters). Only traders WITH a @CurrentMode attribute (i.e. fast-start units) are returned.
NEMX.ZBenchmark.xml_generic_constraints — Method
xml_generic_constraints(m) -> (constraints, trader, interconnector, region)Extract the GENERIC (network / security / FCAS-requirement) constraints that NEMDE actually invoked for this interval, straight from the case file. These are the constraints that set most NEM regional prices — omitting them is why an unconstrained merit-order model under-prices and shows no regional separation.
Real XML hierarchy (verified against nempy's xml_cache navigation):
NEMSPDCaseFile ├── NemSpdInputs │ └── GenericConstraintCollection │ └── GenericConstraint @ConstraintID @Type(LE/GE/EQ) @ViolationPrice │ └── LHSFactorCollection │ ├── TraderFactor @TraderID @TradeType @Factor │ ├── InterconnectorFactor @InterconnectorID @Factor │ └── RegionFactor @RegionID @TradeType @Factor └── NemSpdOutputs └── PeriodSolution └── ConstraintSolution @ConstraintID @RHS @Intervention
TWO things that previously broke this:
- RHS must come from ConstraintSolution (the solved RHS for the interval), filtered to @Intervention == '0' (the non-intervention pricing run that ROP is taken from) — NOT from the inputs GenericConstraint (which has no @RHS).
- The factor elements live INSIDE each GenericConstraint and do not carry their own @ConstraintID, so they must be read as descendants of their parent constraint, inheriting that constraint's id.
Returns four tidy DataFrames: constraints : set, rhs, type, violation_price (only invoked constraints) trader : set, unit, service, factor interconnector : set, interconnector, factor region : set, region, service, factor
NEMX.ZBenchmark.xml_initial_conditions — Method
xml_initial_conditions(m) -> DataFrameExtract per-unit NEMDE initial conditions. nempy reads these InitialConditionID values: INITIALMW (the unit's MW output at the start of the interval, used for ramp limits), RAMPUPRATE / RAMPDOWNRATE (the telemetered SCADA ramp rates, MW/min) and AGCSTATUS. Mirrors nempy's get_unit_initial_conditions.
NEMX.ZBenchmark.xml_is_intervention — Method
xml_is_intervention(m) -> BoolTrue when the interval was subject to an AEMO intervention (the case file then carries TWO PeriodSolution elements: intervention and what-if). Mirrors nempy's is_intervention_period. During interventions the pricing (what-if) run uses the WhatIf* initial conditions.
NEMX.ZBenchmark.xml_is_ocd — Method
xml_is_ocd(m) -> BoolTrue when the interval's case file is the over-constrained-dispatch (OCD) rerun variant. Mirrors nempy's is_over_constrained_dispatch_rerun, which checks for 'OCD' in the loaded FILE NAME (AEMO only publishes an _OCD case file when the rerun process was actually used).
NEMX.ZBenchmark.xml_loss_at — Method
xml_loss_at(model, flow) -> Float64Loss (MW) at flow from an xml_loss_model entry: the integral of the segment factors from zero flow to flow.
NEMX.ZBenchmark.xml_loss_model — Method
xml_loss_model(m::XMLCacheManager) -> Dict{String,NamedTuple}NEMDE's OWN per-interval interconnector loss model, keyed by interconnector: (lower_limit, share, segments) where segments is a vector of (limit_mw, factor) pairs.
The case file carries the loss model NEMDE actually solved with:
<Interconnector InterconnectorID="V-SA">
<LossModel LossLowerLimit="1051" LossShare="0.82" ...>
<SegmentCollection>
<Segment Limit="-1043" Factor="-0.1234"/> ...Factor is the MARGINAL loss (dLoss/dFlow) on the segment ending at Limit, and the segments already incorporate that interval's demand-dependent term — so the loss curve is the integral of the factors anchored at ZERO flow, with NO further demand adjustment. Reconstructing this way reproduces NEMDE's published InterconnectorSolution/@Losses EXACTLY (verified to 5 decimals on all six interconnectors across intervals in both July-2024 and September-2025), whereas re-deriving the linear term from the MMS demand coefficients leaves a purely linear error — measured at 0.014 MW per MW of flow on V-SA, i.e. up to 14.8 MW.
LossModelID is a version stamp shared by every interconnector in the file, so the models MUST be read as children of their parent Interconnector element, not looked up by that id.
NEMX.ZBenchmark.xml_mnsp_offers — Method
xml_mnsp_offers(m) -> Dict{Tuple{String,String},NamedTuple}Parse the MNSP (market network service provider) OFFERS for the interval. An MNSP interconnector is a MARKET PARTICIPANT: its owner bids the link's capacity in ten price/volume bands per region, and NEMDE clears those bids exactly like a generator's. Without them an MNSP flows freely to its registered MAXCAPACITY on inter-regional spread alone, which over-states flow and collapses the price separation the bids create.
XML layout: NemSpdInputs/PeriodCollection/Period/InterconnectorPeriodCollection/ InterconnectorPeriod[@InterconnectorID, @MNSP=1]/MNSPOfferCollection/ MNSPOffer[@RegionID, @MaxAvail, @BandAvail1..10, @RampUpRate, @RampDnRate] NemSpdInputs/InterconnectorCollection/Interconnector/ MNSPPriceStructureCollection/MNSPPriceStructure/ MNSPRegionPriceStructureCollection/ MNSPRegionPriceStructure[@RegionID, @PriceBand1..10]
Returns (interconnector, region) => (maxavail, bands, prices, rampup, ramp_down).
NEMX.ZBenchmark.xml_trade_prices — Method
xml_trade_prices(m) -> Dict{Tuple{String,String},Vector{Float64}}Extract the 10 PRICE bands per (TraderID, TradeType). In the NEMDE XML prices are NOT on the period Trade element (which only holds the BandAvail volumes) — they live under each trader's price structure:
TraderCollection/Trader[@TraderID]/TradePriceStructureCollection/
TradePriceStructure/TradeTypePriceStructureCollection/
TradeTypePriceStructure[@TradeType, @PriceBand1..@PriceBand10]Returns a lookup keyed by (unit, NEMDE TradeType code) → [price1 … price10].
NEMX.ZBenchmark.xml_trader_periods — Method
xml_trader_periods(m) -> Vector{EzXML.Node}Return all TraderPeriod nodes (one per unit with offers) for the loaded interval. These carry the TradeCollection of 10-band bids per service.
NEMX.ZBenchmark.xml_violation_prices — Method
xml_violation_prices(m) -> Dict{String,Float64}Read the constraint VIOLATION PRICES (in $/MW) that NEMDE used for this interval from the case-solution element of the XML. nempy reads these same attributes; they map to the soft-constraint slack penalties in the optimiser. Each must be well above the market price cap so constraints are respected. Returns whatever is present; callers merge over the CVP_FACTORS fallbacks.
Attribute map (NEMDE @attr -> NempyJL key): @EnergyDeficitPrice -> regionaldemand @InterconnectorPrice -> interconnector @GenericConstraintPrice -> genericconstraint @RampRatePrice -> ramprate @CapacityPrice -> unitcapacity @UIGFSurplusPrice -> uigf @ASMaxAvailPrice -> fcasmaxavail @ASProfilePrice -> fcasprofile @FastStartPrice -> faststart @VoLL -> voll @TieBreakPrice -> tiebreak