ICRE recently spoke with Dr. Linda Probyn, an ICRE 2011 What Works finalist and Radiology program director at the University of Toronto, on the pilot project, “Auditing a CanMEDS tool: Closing the teaching and learning loop”. Dr. Probyn developed the audit with Dr. Karen Finlay from McMaster University.
What was the impetus behind this pilot?
The audit was implemented to assess residents’ understanding of how the CanMEDS Roles were being evaluated in a Radiology OSCE performed in June 2011. The OSCE was intended to evaluate the following roles:
- Health Advocate (through a radiation safety station);
- Communicator (through a dictation station);
- Manager (through an on-call scenario regarding chief resident duties)
- Manager (through a case triaging and protocoling station); and
Given the increasingly requirements around incorporating CanMEDS Roles into resident teaching and evaluation, we wanted to see if our residents understood how these roles are integrated into our specialty-specific requirements and how they are subsequently evaluated.
What exactly did you do?
We developed a simple one-page audit tool questionnaire that each resident completed at the end each station. We intentionally asked very simple questions after stage of the OSCE, with all residents participating. After each station, the residents would identify through a multiple choice question which role was identified predominantly, list reasons why they thought this was the case and then list any secondary roles that were also being evaluated.
Importantly, various program directors throughout the country and some of our own program committee members were solicited to determine their opinion as to what roles were being evaluated at each OSCE station. The resident results were then compiled and compared to the expert opinions.
What did you find in terms of the resident awareness?
For the most part, residents’ responses on the CanMEDS roles were concordant with the opinions provided by the panel of experts — dictation station (communicator role) and case triaging/protocolling station (manager role). There were two role-playing stations — radiation safety station (health advocate role) and on-call scenario (manager role) — and residents perceived the presence of a mock patient or resident in the room as an indication that the communicator role was primarily being evaluated. Faculty thought that the communicator role was being secondarily evaluated for these stations.
In both cases, residents identified the correct role as being secondarily evaluated. The audit helped to confirm that there is overlap of the CanMEDS Roles even though one role is predominately being evaluated. For the most part, radiology residents have an understanding of how the CanMEDS roles we evaluated are integrated in our specialty but we need to continue this education and be more explicit. We had an immediate debrief session with the residents which allowed for immediate teaching and feedback about the stations being evaluated.
What was the resident feedback on this audit?
We did not formally evaluate this, but through verbal feedback in de-brief sessions, residents were quite positive about the experience. From a teaching perspective, the audit also provided a new opportunity to have practical discussions about where CanMEDS roles such as the Health Advocate fit into the Radiology curriculum.
Is this applicable to other programs?
Absolutely. Once an OSCE is developed (the previous iteration of the OSCE itself was presented at ICRE 2010), the audit tool can easily be applied to other specialities and programs. The one-page audit that we developed could easily be adapted and would be beneficial for any speciality to enhance not only resident awareness but faculty development as well. Also, faculty demands for developing and conducting the audit were minimal outside of devising the baseline consensus needed to evaluate resident feedback.
What are your next steps?
We are currently running this OSCE biannually, and we will run this audit again at the next exam to see if there has been any change I n results. The stations may be modified although will have enough consistency to compare the data and extract valid conclusions.
Leave a Reply