Model-Based Stateful Testing with madhouse-rs
Some bugs only emerge from chaos.
Early 2024, Stacks faced a mysterious stall in production. Miners would occasionally fail to build on their own blocks after a network reorg, causing consensus issues that were tricky to reproduce with traditional scripted tests.
The chaotic, non-deterministic conditions that triggered the stall simply couldn’t be captured with carefully scripted test sequences.
This failure led to a radical idea: what if we could generate thousands of random test scenarios instead of writing them by hand? What if chaos, not careful scripting, could reveal the hidden corners of complex systems?
This series chronicles that journey—from a mainnet stall that couldn’t be caught, to the creation of madhouse-rs, a framework designed to embrace chaos in testing complex distributed systems.
At its core, this is a story about the expression problem—a fundamental design choice that determines whether your testing framework can grow from dozens of test scenarios to thousands. The difference between these approaches isn’t academic; it’s the difference between catching production bugs and missing them entirely.
Why This Matters
-
Production Bugs Hide in Chaos. The most critical bugs—race conditions, timing issues, complex state interactions—emerge from non-deterministic scenarios that manual tests rarely explore. Traditional testing assumes you can predict where bugs live. Reality is messier.
-
Scale Reveals Design Flaws. A testing approach that works for 10 scenarios may collapse at 100. At 1,000+ scenarios, only certain designs survive. The expression problem determines which side of that divide your framework lands on.
-
Framework Choice Shapes Your Tests. Model-based testing frameworks like proptest-state-machine and madhouse-rs make fundamentally different trade-offs. Each focuses in different scenarios—understanding these trade-offs helps you choose the right tool for your specific testing needs.
Articles
This series examines the expression problem in Rust, from theory through practical implementation in two MBT frameworks:
- Part 1 - The Expression Problem, with Counters
- Part 2 - Model-Based Stateful Testing with proptest-state-machine
- Part 3 - Scaling Model-Based Stateful Testing with madhouse-rs
- Part 4 - Chaos Testing stacks-node with Model-Based Stateful Testing
- Part 5 - The Expression Problem in Practice: A Trait-Based Testing Harness
Prerequisites
A Rust development environment is recommended for following along with the code examples. Basic familiarity with Rust’s enum and trait system will be helpful, though the concepts are explained from first principles.
About the approach
“The expression problem is a new name for an old problem. The goal is to define a datatype by cases, where one can add new cases to the datatype and new functions over the datatype, without recompiling existing code, and while retaining static type safety.”
–Philip Wadler
This series takes a practical approach, grounding the theory in real-world code. We start with a simple Counter program to build intuition, then apply the concepts to two testing frameworks, and finally demonstrate the approach on a production system—stacks-core.