Experimentation is a word we've all heard before.
For many of us, our first introduction came long before our professional lives. I still vividly remember learning the scientific method in a seventh-grade science class and the colorful poster that walked us through each step: make an observation, ask a question, conduct background research, develop a hypothesis, perform a test, analyze the results, and refine your thinking. Perhaps it was due to the context that this process was introduced to me, but at the time, it felt like something reserved for laboratories and textbooks. However, over time, I noticed use of the same process expressed with different language, showing up in business and design thinking principles (e.g., A/B tests and iterative design cycles). While the vocabulary changes, the underlying idea remains the same: we test assumptions to generate learning.
Because of this familiarity, experimentation can seem straightforward. I’ll encourage you to not let that comfort deceive you. Experimentation is an incredibly powerful tool, but like any powerful tool, it needs to be used thoughtfully. If mishandled, it's entirely possible to learn very little, or perhaps worse, walk away with conclusions that reinforce false assumptions.
If I'm honest, in some contexts this may not be particularly consequential. Reality has a way of correcting us over time, and the primary consequence may simply be that it takes longer to get where you were trying to go. But operating within complex systems, like organizations and communities, are different. These are environments where variables cannot be easily isolated, outcomes are difficult to predict, uncertainty is abundant, and decisions often have meaningful consequences for the people involved.
In these contexts, experimentation becomes more than a useful technique; it becomes perhaps the only disciplined way of learning to enable better decision-making and build confidence in a particular course of action. It's for this reason that experimentation is a core value of Roots + Rivers Collective.
Since experimentation is more nuanced than it first appears, I want to highlight some of the most common missteps that I’ve seen when working in complex systems, and how to avoid them.
Every experiment is shaped by a question or set of questions. In complex systems, however, I've often observed that teams move directly into testing without first articulating what they are trying to understand. Even when a question exists implicitly, it is rarely made explicit enough to guide the work.
This is particularly important because the question itself is not always fixed. In Human-Centered Design, for example, much of the Understand phase is devoted to uncovering, reframing, and refining the problem as new insights emerge. As understanding evolves, so too should the question(s) being asked. At any given point, however, the current question should be clear enough to define the assumptions being tested and inform the design of the experiment.
Without that clarity, there is no stable definition of what "learning" would look like, nor a sound basis for interpreting the results. What counts as success becomes fluid, and outcomes can be explained in multiple, often contradictory ways after the fact.
When in doubt, ask yourself: What are we actually trying to understand, and what assumptions are we testing in doing so?
Learning depends on being able to connect cause and effect. When multiple changes within a solution or experiment occur simultaneously, that connection quickly disappears. New tools, new processes, new incentives, and new structures may all contribute to a positive outcome, but by changing everything, we've learned very little about any one thing. Complexity already obscures attribution; we shouldn't make it harder ourselves!
Perhaps the most common misconception is expecting proportional responses. We commonly assume larger interventions, investments, etc produce larger results, yet complex systems are governed by thresholds, feedback loops, adaptation, and emergence. Given this, experimentation becomes valuable precisely because it helps us discover these relationships instead of assuming they exist.
Complex systems are more often defined by relationships instead of discrete components. Changing one part inevitably influences others, and often in unexpected ways. Therefore, hyperfixating on improving a local metric may simply move the constraint elsewhere or create entirely new problems. Experiments should therefore ask not only What changed? but What else might have changed because of it?
Complex systems, like organizations, naturally gravitate toward measuring outcomes such as revenue, throughput, defects, costs, and customer growth. These metrics are tangible, familiar, easy to communicate, and often serve as proxies for organizational health. Yet internal system dynamics frequently begin shifting long before these outcomes reflect any change.
Leading indicators of progress, such as changes in behavior, decision-making patterns, communication flows, coordination, and learning, often shift first. When we measure only final outcomes, we risk discarding promising experiments simply because we were looking in the wrong place for evidence that progress was already underway.
Good experimentation isn't about achieving the impossible and controlling a complex system. It's about creating enough structure to separate signal from noise. In doing so, every experiment becomes another opportunity to update our understanding of how the system actually behaves rather than how we assumed it would. And over time, these small increments of learning become the foundation for better decisions and stronger solutions.