Making Beam Solar faster and more efficient

The problem

Over the development of Beam Home, we ran into several blockers making a simpler mobile-first assessment flow for residential customers. Our base assessment calculations were taking around 30 seconds to calculate recommended solar system sizes and the time goes up to 2 minutes when we try to return calculations involving batteries. 

This meant that the users trying to make an assessment, would press the button to initiate the calculations but would see a spinning UI for a very long time. Leading to confusion and worrying if there's going to be something coming back. 

To make matters more difficult, our existing calculations code were tangled and brittle. Many aspects of our codebase were tied together and slowing our development cycle. To speed up the calculations code, was just not a sustainable approach.

Not only that, our existing infrastructure cost has been increasing, so we were shopping around for alternative technologies and finding suitable database replacements for our use case for less cost.

For 2023, we have plans to implement support for modelling Heat pumps, EV chargers as well as more complex battery models. On top of the new energy load endpoints, our interval data parsing currently only supports 30 minute interval data. Which means that if the interval data available is in a different interval timeframe, an additional step to convert to 30 minute intervals would be required. 

Having the existing calculations as a core part of our platform and being difficult to inspect and develop on, we arrived at the conclusion that a rewrite of the calculations code would set us up for future success in quickly developing new features without accruing bad technical debt.

The solution

Rewriting the calculations code, this allows us to easier extend our calculations for additional sources of energy endpoints. The improved design allows for more isolation of each of the calculations steps, untangling the codebase. This way the developer working on the new feature does not need to spend their time understanding where the new feature fits in the current system before they can begin implementing the feature. They can simply understand what shape their new feature should output as well as what inputs they have access to and create the feature standalone. Allowing for easier testing and only requires plugging into the new system.

To help us create persist the data and create the isolated environments, we've used a combination of attribute-based database and library, namely Datomic and Pathom. Traditional relational databases store data in terms of columns and rows, where each row represents an individual item and each column is data associated with the item. 

When adding new data associated to the item, we would need to alter the specification, saying that all items of that type now also has an additional column of data. Requiring a database migration for every change. This is where attribute-based data comes into play. The difference will be that the data stored will be a row contains the new data as well as data linking back to the item. This way, we don't require to migrate existing data just to play nice with the new data. 

There are more features we will not require for this phase but will have available to us through Datomic.

As for Pathom, we use it to manage our data navigation by querying the Pathom network for keys we require without needing to wire up the data to where it's needed. Not only that, we're able to reuse the code on both front and backend. Reducing the number of times we'd be spending recreating the same functions.

We've set ourselves up for ease of development while optimising the speed of our calculations. Furthermore, we're able to run Datomic in-memory, allowing for fast lookups to the database without requiring Redis to cache data ahead of time and reducing our expenses.

The result

Fast forward to current state, we've built the new calculations for our residential platform. Very happy to announce that our calculations now takes ~10s for an assessment and ~30s with battery calculations.

Check out our residential platform with the improved calculations below, designed for simplicity for home owners in ACT to assess Solar for their homes. 

The future

At the moment, our sped up calculations are for the residential platform, we're working on porting over the new changes to our commercial platform.

Other than the massive speed boost to our assessments, we have a system designed that will allow us to assess more battery options, as well as support more configurations (involving EV chargers, heat pumps, and so on). This allows us to release new features without having to rework the existing system.