From Meetings to Prototypes: The Importance of Being Experimental

Educators are problem-solvers in a profession riddled with instructional and non-instructional problems. In any one school, countless problems need to be solved concurrently; Maybe one grade level needs help with improving students’ decoding skills, while the specialist teachers are curious how adjusting class periods would affect learning, while at the same time, despite heavy investment by the school, the virtual learning platform is not being adopted. Once one problem is addressed, another inevitably rises to take its place at the top of the queue.

Something I read lately that is especially relevant to problem-solving and decision-making in education is “The Controlled Experiment” by Timo Hannay. It’s a two page essay from “This Will Make You Smarter,” edited by John Brockman. Hannay makes the argument that professionals in all fields, including education, would be wise to take a more scientific approach to solving problems.

The Controlled Experiment

While most of us understand the basic principles underlying experimentation as they relate to the lab, few educational leaders are used to the idea of conducting controlled experiments of their own in order to generate knowledge about how to solve problems. Most school leaders’ default mode, in my experience, is to call meetings to facilitate collaborative brainstorming in a small group. This introspective approach to problem-solving follows a predictable sequence that I’m sure all teachers will be familiar with:

  1. Principal assumes there is a problem.
  2. Principal calls a meeting between a small group of relevant leaders and they think, think, think.
  3. Then he/she calls a larger meeting between relevant teachers and stakeholders and they think, think some more.
  4. A solution to the problem is selected, sometimes by consensus, sometimes loudest voice wins, or sometimes it’s clear that all of this was for show because the solution had already been determined by the principal long before the meetings took place.
  5. The solution is implemented.

The main disadvantage to this non-scientific, consensus-based method is that the solution that ends up being chosen is limited by the beliefs, biases, intuitions, and imaginations of the team members that were present at the meetings. Of course, this issue could be mitigated if the meetings consisted of participants conducting thorough reviews of research related to the problem and choosing between others’ ready-made solutions. But, as we know, research requires time and specialized knowledge and skills, both of which can be lacking, so decisions are usually made hastily, based entirely on the available knowledge and skills of those present at the meetings; Not ideal. 

Rather than relying on what Hannay described as, “instinct or partially informed debate” (p. 25) to solve problems in our schools, we should consider conducting controlled experiments in which only one variable is manipulated at a time. At my school we call this prototyping, and I’ve written about it before. Hannay described how such experimentation is already happening in other fields:

Online companies, such as Amazon and Google, don’t anguish over how to design their Web sites. Instead, they conduct controlled experiments by showing different versions to different groups of users until they have iterated to an optimal solution (p. 26).

Deploying a prototype or conducting an experiment can look different depending on the nature of the problem and the context, but the part that tends to be forgotten is the controlled part. Whenever we test something in our schools, we should do our best to organize a business-as-usual group, and we should change only one variable at a time. If we roll out everything for everyone all at once and keep changing it, how can we expect to learn from the results?

Maybe schools should reconsider the effectiveness of “deep thinking around tables” for solving educational problems. Instead of looking no further than that which we already know, schools should consider expanding our knowledge by conducting controlled experiments (prototyping) in tandem with group literature review (research). It may not be how schools have traditionally done things, and it may be frustrating in instances where results come back inconclusive, but, as Hannay explains, “None of this takes away from the fact that the controlled experiment is the best method yet devised to reveal truths about the world, and we should use them wherever they can be sensibly applied (p. 27).

– Zach Groshell, tweetz @mrzachg


Brockman, John. This Will Make You Smarter. Transworld, 2013.