Scaling Research into Practice

Research and implementation of evidence-based practices is undeniably valuable, but difficult to successfully do. Why? We don’t have a great understanding of the science of scaling yet. Scaling is taking a successful strategy from a small system to a much larger system. The study of how well we do this is called implementation science. It relies largely on trying to translate evidence from small-scale academic studies to large-scale social systems like education. This is a complex and difficult task, that often results in a significant decrease in effectiveness called a voltage drop. Some reasons for the voltage drop:

1. Fidelity. Does the scaled-up program match the integrity of the original smaller program? Dana Suskind, Professor at the University of Chicago,  explains that a small-scale tutoring program with 10 exceptional tutors with PhD’s in math, cannot easily be scaled to a program with 10, 000 math tutors. The average skill-level with 10 tutors is very different than the average skill level in the 10,000 tutors.
2. System Complication. Multiple systems are often required to get a project off the ground, but each system has different goals. Patricia Chamberlain from the Oregon Social Learning Center provides the example that three organizations might fund a project, but each organization has different goals, since the funded project cannot meet all the goals for all three organizations the project loses funding and fails.
3. Institutional Organization. Institutions can be built for efficiency or for effectiveness. A provincial education authority and school division board office are organized to be efficient. There are limits on the amount of funding that can go into administration within education, so these organizations are designed to be lean. Schools are inherently built for effectiveness. When systems designed to be efficient, interact with systems designed to be effective, the rules governing the two are so inherently different that ideas can have difficulty transcending from one system to another.

John List, an Economist from the University of Chicago, notes that complex service-delivery systems, such as education, are among the hardest to scale. The academic studies that show promise are effective at telling us what was effective in that specific context for that specific group, but since all contexts and groups are different in schools it can be difficult to make the intervention that worked in one place work in another. A good analogy is scaling a successful restaurant into a chain of similar restaurants. If the original restaurant is great because of the combinations of foods, this can be taught to other chefs and successfully scaled into a chain of restaurants. If the restaurant is great because the chef is uniquely talented, this is harder to scale, the likelihood of finding the same greatness as the original chef x 1000 new stores in the chain is unlikely.

So what do we do? Academic studies need repeatable results across varied contexts prior to scaling. Projects in consideration for scaling should hinge on replicable factors (ingredients, not chef skill). A new set of goals and objectives that meet the needs of all invested groups need to be established and clearly confirmed. Uptake needs to be feasible in the scaled industry.

Thoughts on Freakonomics Radio – “Policymaking is Not a Science (Yet) – Ep. 405 Rebroadcast March 24 2021 (By Steven Dubner)