Overhauling our simulation framework

Simulations are long-running virtual users that drive realistic business journeys (such as onboarding or payments) against the fully deployed production‑like stack. They surface regressions through metrics, logs, and alerts, rather than explicit test assertions. Unlike conventional automated tests, simulations behave like real traffic over time, covering end‑to‑end flows instead of isolated components.

Engine now runs continuous simulations that catch issues earlier across the deployed platform. Learn how we rebuilt our framework from scratch to achieve a configuration-driven framework that has had significant benefits for both our product team and engineers.

Our test suite, which ran as part of our continuous delivery pipeline, had extensive coverage and gave us confidence in the quality of our code. Although our existing simulations gave engineers an array of test coverage, they had a number of limitations:

• High developer friction: Writing simulations required navigating a steep learning curve, which often led to engineers copying and pasting boilerplate code instead of focusing on business logic.

• Slow feedback loops: Deploying directly to staging environments was easier than running locally, which dragged down iteration speed.

• Inaccurate test fidelity: The framework failed to faithfully mimic our public APIs, choosing instead to call internal endpoints that are not exposed outside of the network. We needed tenant-perspective tests with all traffic routed through the API federation layer to serve as living integration examples.

• Limited operational observability: We had to rely heavily on application-level logs and Sentry (a developer-focused application monitoring and error tracking platform) to catch simulation failures. Whilst both of these tools were helpful in flagging simulation errors, we lacked an overall birds-eye view of which simulations were being run, and when, in a given environment.

• Unwanted dependencies on staging environments: Some simulations were dependent on infrastructure that was only available in staging environments, and therefore could not be reliably simulated in our own development environment.

As our delivery speed increased, and with it our feature set, we were at risk of introducing regressions. Proactive analysis of our internal incidents highlighted key areas for improvement; it was clear that our existing testing strategies would not scale.

We’ve built a declarative, configuration-driven simulation framework from scratch. Schedules, load patterns and parameters are defined in config, enabling engineers to focus on scenario logic rather than orchestration boilerplate.

This has resulted in a much better experience for engineers, who can now run simulations locally instead of making canary deployments to a staging environment. The feedback loop is shorter, and out-of-the-box observability gives us automated alerts and dashboards that highlight any failures we need to address.

We’ve also added support for parameterised simulations, meaning we can manually invoke any simulation using different parameters. This is useful in many situations, from debugging to creating test accounts.

The simulation framework is integrated into our release and observability pipelines. A failing simulation in our demo environment now triggers an incident, ensuring that we find and prevent issues before they reach production. Simulations are now also part of our definition of done, which means they’re a prerequisite rather than an afterthought.

Finally, we’ve introduced gap analysis tooling to help expedite the migration of our existing features to the new simulation framework.

Implementing the new framework wasn’t without its difficulties. Existing simulations required fine grained tuning to ensure the appropriate cadence and representative data volumes. Connection pool pressure emerged as a result of many parallel simulations hitting a database, and required query optimisation and right‑sizing of resources to resolve.

We also found that product configuration meant a single environment couldn’t express every variant that we wanted to simulate at the same point in time, and we realised that broader variant coverage would need multiple simulations or multiple-environments.

Despite these complications, the benefits have been significant:

• Proactive defect interception: The system has already caught multiple critical defects before they could reach tenant environments, fully validating our investment of time and resources.

• Cultural shift toward quality: By elevating simulations and alerting to first-class status we have fostered an engineering culture that prioritises reliability long before code hits production.

• Accelerated engineering velocity: Automated simulations and a comprehensive regression suite have largely replaced most manual verification, shortening feedback cycles and enabling faster, safer releases.

We’re planning to evolve from scheduled executor patterns to more event-driven traffic modelling. This will better mimic real usage, expose race conditions, and enable credible load testing. We’ll also migrate all our legacy simulations onto the new framework, and explore running targeted simulations inside tenant environments to expand variant and code-path coverage.

Learn more about life at Engine by reading Andjela Labovic's conversation about being manager in a growing team , or find out how the team ensure our rollback process is super fast and ultra reliable.