Monday, February 23, 2015

To Conform or Not to Conform?

Conformity is all around us. While it sounds negative to label someone a "conformist", saying that the person is a "team player" is appealing. Isn't this the same thing framed in a more palatable way? We celebrate historical figures for non-conformity, especially those political figures that proved to be on the right side of things in retrospect, but these non-conformists often faced opposition and strife. This is true of Lincoln, Rosa Parks, Martin Luther King Jr., ... to name a few.

Conformity is a reality of the world in which we live. Social scientists describe stages of development of the self that surround the notion of how others perceive us (e.g., Looking Glass Self).

We, as human beings, are highly influenced by those around us. A classic psychology experiment by Solomon Asch (1956) with American college students over 50 years ago demonstrated how much we are influenced to conform. Its surprising! He conducted a social experiment in which he asked a very simple question to a group of students. One of the students was set up. All the other students were told to pick the same wrong answer. Asch found that when presented with a very easy problem, one that had an answer that a child could figure out, if everyone gives the same false answer, 75% of students gave the same incorrect answer that the class chose 75% of the time. The questions were asked more than once and when all answers were added up an average of 35% of all responses (the guy who didn't know) conformed to the wrong answer.

So this means there are situations when in groups that we can feel pressure to knowingly choose the wrong answer. This experiment and others like it have been repeated many times. In fact this study was repeated with participants that were monitored with fMRI technology. What was found?

Basically, it is a painful experience to go against the group's position (Burns and colleagues, 2005). The activity in the area of the brain that shows pain and discomfort (e.g., amygdala) was very busy for the students that went against the the group's wrong answer. The brain image of the guy that went with the group opinion wasn't nearly as disturbed.

What could we do?

I suggest  know what you think and believe in advance or the group can easily influence you.

IME 2015 Conference Keck School of Medine at USC Oral Presenation,

The following is the longer version of the paper submitted to IME 2015 (conference syllabus).  Clicker here to view the oral presentation (googledocs) or here for a youtube version and clicker here to view the Q&A.

The Effect of Audience Response Systems on Metacognition in Graduate Students: A Two-Year Mixed Methods Study   

Melanie Brady, Ed.D., University of Southern California, Rossier School of Education
Jane Rosenthal, Ed.D., Assistant Dean, School of Applied Life Sciences, Keck Graduate Institute
Christopher P Forest, MSHS, DFAAPA, PA-C University of Southern California, Keck School of Medicine, Division of Physician Assistant Studies

Problem Statement
Use of educational technology to engage learners continues to grow at a rapid pace. Studies of effectiveness of clicker use find that when clickers are utilized with research-based instructional strategies the learning experience in large lectures is enhanced2. In a study with undergraduates (n=198) metacognitive self-regulation seems to improve when clickers were utilized in this manner. Comparison (low technology) and experimental (clickers) methods each demonstrated significance influence on learner metacognition, clickers with the summer cohort and the comparison method with the fall. However, when performance outcomes and qualitative data were factored in, clickers demonstrated a high degree of significance (p> .01). This current mixed methods study of audience response systems and metacognition investigates whether the experience for graduate health science candidates (e.g., 1st year Physician Assistant candidates 2013 and 2014) is consistent with the undergraduate experience and to what degree between graduate cohorts (2013-2014). 

The importance of these investigations lies in the growing body of research that self-regulated and metacognitively aware learners tend to have improved outcomes and that metacognition and self-regulation are teachable. Research suggests when clickers are utilized with instructional strategies (e.g., questioning and peer instruction), performance outcomes increase and metacognition may be affected. Metacognition, the regulation of cognition and self-knowledge, is an essential component in the learning process in order to become a self-regulated learner. This mixed methods comparative study examines the extent to which high-tech devices (clickers) and low-tech devices (paddles) affect learner metacognition. This study extends our 2013 mixed methods examination of clickers and metacognition conducted with 1st year Physician Assistant candidates and further comparison between clickers and paddles.,1,2  If the data generated by the two years is not robust enough, a third cohort is proposed for fall 2015 to increase the strength of results and potential for generalization. 

The response device that more effectively influenced metacognition would be  associated with higher performance outcomes. Based on the results of the undergraduate study we predicted that use of clickers would lead to less social comparison which could enable more productive learning; use of paddles would lead to more social comparisons that could interfere with the learning process.

Data were collected from 54 graduate candidates in 2013 and 51graduate students in 2014 during a behavioral sciences course.  Clickers were used during weeks 1-5 of the course and a low technology response system (paddles) during weeks 8-12. Paddles are handmade signs are held up to indicate preferred answers (A-E); this method was selected for comparison as an analogous system to clickers in that it provides a quick visual check of student responses that allows participants to be polled once as opposed to raising hands several times for a multiple choice question. This comparative, mixed-methods study employs several measurement instruments and a pre- and post-test design to compare the two response systems. The components of metacognition of interest in this study are Metacognitive Judgments and Monitoring and Metacognitive control and self-regulation. 

Quantitative instrument. In the first week of the course, pre-test data and demographic information were collected. Questions from the Motivated Strategies for Learning Questionnaire (MSLQ)3 served as the pre-post-test instrument. Two instruments that measure feedback systems and metacognition1 were administered at week 5 (experimental/clickers) and at week 10 (comparison/paddles) Metacognition in Lecture Survey2 (2013-clickers, α = .910; 2013-paddles α = .935; 2014-clickers α =  .806; 2014-paddles α =  .888) measures metacognitive self-regulation experienced by learners in lecture through changes in learning behavior inside or outside of lecture and Metacognition Attribution to Response Device Scale (2013-clickers, α = .723; 2013-paddles α = .704; 2014-clickers α =  .681; 2014-paddles α =  .827) measures the level of metacognitive influence that learners believe to experience as a direct result of the use of the polling method. Mean quiz scores from the first 5-week session served as the measure of performance outcomes for clicker use, and the mean participation scores for weeks 6-10, for the comparison treatment (paddles). 

Qualitative instrumentation. Participants completed an on-line qualitative survey using Qualtrics© that consisted of open-ended questions to elicit reflections about response device use.   Interviews were conducted using purposeful sampling using the following criteria: 1) low mean scores indicating little metacognitive influence attributed to clicker/paddle use; 2) mean scores in the median range indicating a moderate-to-neutral influence; and 3) high mean scores indicating a strong influence. 

High comfort level and prior use of audience response systems were reported by 60% of participants from the 2013 cohort and 100% of participants from the fall 2014 cohort on the initial survey.  Two tailed t-test for dependent means were conducted to examine between groups differences in metacognitive self-regulation, the pre-post-post –MSLQ), and performance outcomes (e.g., in lecture clicker quizzes. Significance was not found between the Graduate Health Science 2013 and 2014 cohorts on the pre-post-post-test administration, but significance was found with metacognition instrumentation. Lack of significance between pre-post-post-test between groups indicates group similarities which increases the potential strength of results and ability to generalize. The first post-test administration followed use of clickers, the second post-MSLQ following the comparison method (low-technology). This indicates that learners in both cohorts gauged individual metacognitive self-regulation similarly at the start of the course, following the treatment method (clickers) and following the comparison (low-technology polling). 

Differences were found on formative performance assessments between the 2013 cohort (M = 87.38, SD = 5.86 and the 2014 cohort: (M=75.41, SD=6.27) demonstrating differences in metacognition during lecture of the two groups (t(52) = 10.263, p = .001). Significance was demonstrated between groups for both instruments measuring influence of metacognition during lecture, and the attribution of metacognition to the response device (t(52) = 4.84, p = .001; t(48) = 5.83, p = .001). Qualitative analysis results were similar between groups. Clickers were perceived as a more effective way to monitor learning and the low technology method resulted in conformity and reduced pressure to prepare for lectures. Differences occurred in that a small portion of the 2014 cohort suggested that the low technology system created opportunities for discussion and learning and was enjoyable while the majority of peers in the same cohort did not share this opinion, nor did the 2013 cohort. Reports of positive learning experiences with paddle use tended to accompany indications of relief at the ability to rely on group lieu of individual preparation when schedules were busy

Lessons Learned
Quantitative results indicate that clickers influence learner metacognition more so than low technology response devices, suggesting that conceptual understanding may be clarified through use of clicker items and interactive teaching strategies (e.g., questioning, and peer instruction) leading to improved formative feedback for enhanced learning. Both cohorts indicated that clickers strongly influenced peer comparisons, consistently positively influencing the learning process. Clickers can improve accuracy of metacognitive judgments and influence strategies utilized for learning outside of lecture. Qualitative results suggest that graduate learners are more confident in strategies utilized for note-taking in lecture and for preparation for lecture. Several learners reported changing answers based on peer responses, and feeling less pressure to prepare for lecture when the low technology system was utilized. Focus of clickers on independent learning improved ability to monitor learning and results indicate that learners are more apt to prepare for lecture with individual response component. 

Selected References
1. Brady M, Rosenthal J, Forest C. Metacognition and Performance Outcomes: An Examination of the Influence of Clickers with 1st Year Graduate; in progress.
2. Brady ML, Seli H, Rosenthal J. Metacognition and the influence of polling
systems: How do clickers compare with low technology systems? Educ Technol Res Dev. 2013;61:885-902.
3. Pintrich, PR, Smith DAR, Garcia T, McKeachie WJ. Reliability and predictive validity of the motivated strategies for learning questionnarie (MSLQ). Educ Psychol Meas. 1993;53(3):801-813.