When gas left the map
This autumn, European gas climbed past anything Morrowatt's models had been trained on, and our forecasts started running low. Here is what JITM Evergreen, the system that watches these models, did about it. The short version is below; the full write-up is on JITM.ai.
What happened
Gas plants set much of the price of power in Europe. In the data our models learned from, the TTF gas price rarely went above €65 per MWh. By mid-September it was above €75, and by mid-October €87. Power prices followed, and the models could not keep up: they read the market through recent prices, and once those climbed past the top of what the models had seen, the forecasts were capped. You could see it on this site: in October, Germany's next-day forecast errors were close to double July's.
What Evergreen did
To see the whole loop clearly, we replayed 1 August to 19 October through JITM one delivery day at a time, starting from models trained on data up to 1 August. Each day: a forecast, then the real auction price sent back as the outcome. Every Monday: a drift check.
Evergreen flagged Germany on 31 August and Spain on 28 September. Each time it trained a challenger on everything that had actually happened since, tested it against the current model on recent days neither had trained on, and promoted it when it won. To measure the effect, we kept an identical copy of each original model that was never updated.
What it could not fix on its own
Retraining helps a model catch up with prices it has now seen. It cannot teach a model to reason about prices it has never seen: on 19 October, with gas at €87.50, both models still missed the day by close to €70 per MWh. When the alarms went off, gas was the input furthest outside its training range, which pointed us at the fix.
The fix is a redesign: forecasting the price relative to the cost of gas, so a gas shock no longer pushes the models into territory they have never seen. New models built that way are training now. They will only replace the current ones if they beat them on the autumn data, and we will publish the comparison here either way.
A note on the replay: each day used the final published wind, solar and demand forecasts, some of which are not yet available at 11:00 the day before. Live accuracy would be somewhat worse. Both models in every comparison saw identical inputs, so the comparisons are fair.
The full story
How the challengers were judged, the direct test of the gas effect, and what we changed in Evergreen along the way.