Reinforcement Learning for Power Generation

Genesis Energy is a power generation company in New Zealand that sells electricity generated by hydroelectric and hydrothermal generators to the domestic energy market. Currently, people control the decisions surrounding power generation and pricing. Genesis asked Winder.AI to help them develop a reinforcement learning-powered solution to automate generation and pricing.

Reinforcement Learning Problem

New Zealand has the enviable situation of possessing high-altitude lakes refilled with ice melt. Discharging the lake presents an ample kinetic energy store that can be utilised for power generation via a turbine. Hydroelectric power generation is therefore sustainable and low carbon.

But utilising the lake for hydroelectric power isn’t as simple as turning generators on or off. There are a range of environmental considerations that require management, such as maintaining an appropriate lake level for the local community and ecosystem, avoiding flooding of the natural river, and balancing water flow so the downstream assets don’t get destroyed. Furthermore, due to the nature of the energy market, the value of electricity generation varies significantly both over the period of a day and over the seasons of the year.

But if the sluice gates were opened fully and remained open, the lake would drain and flood the downstream rivers. Furthermore, periods of low electricity demand might yield periods where it is uneconomical to generate power.

This presents a situation where it is necessary to balance competing concerns: the sustainable management of lake levels compared to generating electricity when demands are high.

Reinforcement learning (RL) is a machine learning (ML) technology that make strategic decisions in dynamic environments. It is well suited to control problems like this because it can learn to optimise a non-trivial reward function by modelling the environment.

We wanted a peer review of the RL code we had already developed. The outcomes for us was that RL is appropriate for our problem however our coding did need refinement.

Michael Eschenbruch, Innovation and Analytics Manager at Genesis Energy

Reinforcement Learning Solution

Traditional control system problems can be solved through the use of reinforcement learning techniques by assuming that the feedback loop is a Markov decision process. Allowing an agent to make an optional decision within a feedback loop fulfils the interface of a Markovian decision process.

In this situation, the goal was to demonstrate that optimal power generation could be achieved by carefully managing the water levels considering the current market price for energy. This was complicated further by the fact that the generating scheme had multiple control gates and generating units.

I feel like it took a bit longer than anticipated to get into the technicalities of our coding, however once time was made available to get into the coding the work flowed pretty fast. We found the level of RL experience at Winder.AI to be most impressive.

– Michael Eschenbruch, Innovation and Analytics Manager at Genesis Energy

Results

In truth, reinforcement learning may have been overly-complex for a single power generation scheme. But Genesis required the ability to optimise over multiple power schemes working within a dynamic energy market. Our reinforcement learning solution allows them to plug in and learn the operating conditions of new schemes within the same constraints.

We collaborated with Genesis’ engineers to help develop a simulation of the environment and a framework to help them solve it. Our domain expertise, combined with their domain expertise, allows us to swiftly develop a proof of concept that can be adapted to their needs. We provided this reinforcement learning service on a flexible basis to fit within their budget.