One Step Forward

Lessons Learned from a Randomized Study of Multisystemic Therapy (MST) in Canada

Alison Cunnningham (2002)

Funded by the National Crime Prevention Centre in Ottawa, this 32-page report presents interim results of a four-year randomized study of multisystemic therapy (MST) in four areas of Ontario. The lessons learned include how the wrong conclusion would have been made about MST if a less rigorous methodology had been used. These points are illustrated using the follow-up data available in January, 2002. This document will be helpful for researchers conducting randomized field studies. Policy makers will benefit from understand how research can be used to improve programming. The final results of the three-year follow-up were released in 2006.

Download the Report

One Step Forward: Lessons Learned from a Randomized Study of Multisystemic Therapy in Canada (file size 458KB)

Table of Contents

• The MST Project

• Rationale for the Project

• Why MST?

• What is MST?

• Partners in the Project

• The Evaluation Strategy

• Participating Youth and Families

• Anticipated Project Outcomes

• Interim Results

• 10 Ways to Make the Wrong Conclusion

  1. Assume the U.S. Results would Replicate in Canada

  2. Assume improvements in pre/post testing are because of the treatment

  3. Assume improvements in pre/post testing will translate to reduced recidivism

  4. Use a one-group design

  5. Compare program completers with drop outs

  6. Use one indicator of outcome

  7. Use a small sample

  8. Use a short follow-up

  9. Study the program in only one place

  10. Use only one study to indicate a program's effectiveness

• Lessons Learned

  1. A randomized study was appropriate for MST...

  2. ... but not popular

  3. Sound research methods are crucial

  4. Random assignment did not control for all threats to internal validity

  5. Role clarification

  6. Securing appropriate referrals: The important of buy-in

  7. Good research takes time

  8. Balancing scope of project with resources

  9. Treatment fidelity

  10. Therapist attrition and burnout

  11. Use description of data

      • Standardized mean difference

      • Risk ratio

      • Percentage difference

      • Risk difference

      • Number needed to treat

  12. Cost comparison

• Factors that May Limit Generalization

• What would we do Differently?

• Conclusions

The Full Report

More detail on the MST technique, the broader context, and the implementation of MST can be found in Seeking Effective Interventions for Serious Young Offenders: Interim Results of a Four-Year Randomized Study of Multisystemic Therapy in Ontario, Canada (2002).

Final Results

The final results of the three-year follow-up are now available.