Amongst a cacophony of noise in the educational community from the competing and adversarial self-interested cries of practitioners, policy makers, education providers and teaching unions, there is a deafening silence in one crucial area in which there appears to be a broad consensus of agreement. More than anything else in a school, teaching quality matters as more effective teachers are an essential prerequisite in the elusive quest to produce high performing students equipped with the skills to succeed in the 21st Century (Ripley, p1, 2012; Jensen, p3, 2011). A similar agreement surrounding what exactly constitutes effective teaching amongst the fractional parties remains elusive in spite of an increase in research, however ‘appraising and providing feedback to improve teachers once they enter the profession and are working in schools’ is one of the five mechanisms through which Jensen (p7, 2011) argues teacher effectiveness can be improved and will provide the focus for this study. Whilst traditional instruments such as quantitative analysis of pupils achievement gains and observations from senior leaders will be considered, one of the most significant developments for education reform over the past decade has been the advent of student feedback in teacher evaluations (Ripley, p5, 2012). The aim of this study is, therefore, to develop an effective student survey to be used alongside traditional instruments of appraisal and feedback as part of a ‘balanced scorecard’ approach in order to improve teacher effectiveness (Jensen, p10, 2011).
Dr Goldacre (2013) extols the virtues of the transition to evidence based practice in the medical community in the face of vociferous inertia and now advocates the proliferation of evidence based practice within the arena of education. In particular, he makes the following recommendations:
research on what works best should be a routine part of life in education
teachers should be driving the research agenda, by identifying questions that need to be answered.
teachers should be empowered to participate in research
myths about randomised trials in education should be addressed, removing barriers to research
the results of research should be disseminated more efficiently
resources on research should be available to teachers, enabling them to be critical and thoughtful consumers of evidence
barriers between teachers and researchers should be removed
It is the first two of these recommendations which this study encapsulates most clearly. The question that has been identified in the introduction is how can student surveys be used alongside traditional instruments of teacher appraisal and feedback to improve effectiveness? In order to develop this survey, it is important to draw on existing studies.
1. Capturing the Dimensions of Effective Teaching
As part of the Bill & Melinda Gates Foundation’s Measures of Effective Teaching 7,500 lessons from 1,333 teachers in six American districts were recorded and compared using five classroom observation instruments and compared with measures of achievement gain for 44,500 students on state tests and student evaluations of teachers (Kane, p35, 2012). Analysis of student achievement gains revealed that teachers with a track record of producing high gains are likely to achieve similar gains with another group and, in maths, this correlation was 0.48 (Kane, p36, 2012). The key value, therefore, in this quantitative approach is its ability to foresee the achievement gains of future students which, moreover, are associated with higher earnings and greater participation at higher education (Kane, p36, 2012; Chetty et al, 2012). If this measure alone is so powerful at predicting future success, why bother with other instruments for appraisal? In spite of its high predictive power, focusing on quantitative measures of growth has a relatively low explanatory power, that is to say that it can reveal large levels of progress (or not!) but it cannot shed any light on what a teacher can do to increase this growth further through detailed and effective feedback on an individual teachers strengths and weaknesses (Kane, p36, 2012). Given that the greatest impact on student learning comes from meaningful feedback (Hattie, 2009), if we re-contextualise student learning as teacher learning, we can see that relying on such measures of achievement gain alone is unlikely to result in learning or, more pertinently, any increase in teacher effectiveness.
The second tenet of teacher appraisal is classroom observation, but of the three in the study this appeared to be the most problematic. 900 observers received between 17 and 25 hours of training on one of five observation criteria and graded 7 500 lessons; each lesson was graded 3 times by three different observation criteria (Kane, p37, 2012). The results from each observer were then compared to an agreed master observation and any that fell outside set parameters from the master were disqualified – in all 23% of observations (Kane, p37, 2012)! Allied to such inconsistencies in observations, concerns about observations stifling innovative teaching styles and the opportunity cost of senior leaders time in completing such observations raise concerns over the suitability of relying too much on classroom observations (Kane, p38, 2012). Moreover, eschewing these concerns, although such observations do, in theory, enable teachers to receive specific feedback about how to improve their practice which, if implemented, should manifest itself in improved teacher effectiveness, there is little evidence to suggest that the feedback does lead to improved student outcomes (Kane, p37, 2012).
The third tenet of teacher appraisal to be explored is the new component of student surveys. Student surveys have been a component (often the only component) used to feed back on instruction in Higher Education but it is only recently that this practice has been implemented in Secondary Education (Kane, p38, 2012). Two articles relating to this use of student surveys in Higher and Primary Education led to me to first consider using student surveys to improve my own teacher effectiveness last year and are explored in detail in ‘Pupil generated feedback: the wisdom of crowds?’ (Davies, 2013) In the first, a university Professor, used to teaching final year students, struggled to engage Freshers until he enlisted the support of one of his former students (Brighouse, 2013) and in the second, a Primary Teacher in Florida who periodically asks her class: “What are ways that I teach you that you like or that are really working for you? What could be changed to help you learn even more?”
In the Measures of Effective Teaching report the most powerful finding is the reliability of student responses to the survey. Student responses had a greater degree of correlation with student achievement gains in Maths and English than did classroom observations and, moreover, not only were the responses consistent across classrooms but they were also predictive of student achievement gains across classrooms (Kane, p39, 2012). In response to the question: “our class stays busy and does not waste time” less than 36% of pupils agreed in one classroom whilst more than 69% agreed in another (Kane, p38, 2012). Also, it was shown that feedback for teachers tended to be consistent across multiple classes with a correlation factor between different classes of 0.66 which was greater even than the original achievement gains measure (Kane, p38, 2012). It can be argued that even without the 17-25 hours of training that classroom observers were provided with students themselves were better at evaluating teachers because they have had months to form an opinion rather than the 30 minutes which senior leaders are typically present for (Ripley, p4, 2012). Whilst there will always be some “knuckleheads” who just mess the survey up and do not take it seriously, these may constitute as little as one-half of one percent which means the wisdom of crowds should prevail to allow teachers an accurate reflection of pupils experiences (Ripley, p4, 2012). Taking into account the effectiveness of student surveys and the fact that they are a relatively inexpensive way to add predictive power and reliability to evaluatory systems, they would seem particularly well suited to augmenting classroom observations in grades and subjects where student achievement gains are not available (Kane, p40, 2012).
2. Better Teacher Appraisal and Feedback: Improving Performance
Systems of teacher appraisal and feedback which are directly concerned with improving student performance have the potential to increase teacher effectiveness by as much as 30% (Jensen, p3, 2011). With a view to improving Australia’s current broken system of teacher appraisal and feedback the report encourages schools to employ at least four methods of teachers’ performance from the following list (Jensen, p9, 2011):
Student performance and assessments;
Peer observation and collaboration;
Direct observation of classroom teaching and learning;
Student surveys and feedback;
360-degree assessment and feedback;
Parent surveys and feedback; and
Although schools are free to employ the methods most appropriate for their context, there is manifest support for the inclusion of student surveys and feedback on the basis that students are able to report on teachers with a high degree of reliability, indeed their ratings of teachers have been found to be better predictors of student achievements than self-assessment and principal measures of effectiveness (Jensen, p16, 2011).
3. A Balanced score-card approach
To address the individual weaknesses inherent in any system of evaluation that relies on any one instrument the Measures of Effective Teaching Study explored a combined approach. Even with achievement gains, classroom observations and student surveys equally weighted the explanatory power was increased (Figure 1). However, by more accurately weighing each instrument to 0.758, 0.200, and 0.042 respectively on the basis of their effectiveness the resulting criterion-weighted or balanced score-card approach yields more of the two desirable properties – predicative power and reliability- than any of the other measures alone (Kane, p39, 2012).
Having explored the rationale for embracing a balanced scorecard approach to teacher appraisal and feedback, this section focuses on how best to design a student survey. In order to elicit student’s perception of teacher effectiveness – the questions within the survey are paramount: if you ask pupils the right questions, they can identify with uncanny accuracy, their most – and least- effective teachers (Ripley, p8, 2012). Whilst students are able to report on teachers with a high degree of reliability, the validity of the survey results depends on the instruments used (Jensen, p16, 2011). Of the 36 items included in the Measures of Effective Teaching study studied earlier, the following five correlated most with student learning and are surprisingly effective (Ripley, p4, 2012):
Students in the class treat the teacher with respect.
My classmates behave the way my teacher wants them to.
Our class stays busy and doesn’t waste time.
In this class, we learn a lot almost every day.
In this class, we learn to correct our mistakes.
Using this architecture as a starting point and augmenting it with college specific questions relating to values and learning principles, the following survey was designed:
The survey is to be introduced by the regular classroom teacher at the beginning of the lesson with an accompanying explanation that its purpose is to help the teacher develop their professional skills. This also helps to prevent any recency effects whereby a the particular lesson in which a survey is conducted skews responses in either direction.
In initial versions of the survey students were encouraged to leave their name in order to enable teachers to continue the discussion of any specific issues. However, in order to reduce the risk of pressure from teachers or peer pressure from fellow students and prevent pupils masquerading as others, it was considered important that the school takes steps to ensure anonymity of individual student responses.
The results remain solely with the individual teacher who themselves sends the link to the google survey and the results which are automatically populated on a colour-coded spreadsheet to facilitate analysis. Teachers are encouraged to share their results but there is no obligation to do so. Instead, there is, however, an expectation that the individual teacher will include reflections on the outcome from the surveys in their individual Professional Portfolios.
In order to prevent saturation, each department will have a designated month when it alone completes the surveys. The Head of Department has discretion as to which year group or ability range to target and even if some individual teachers decline to formally share their findings it would be expected that, at a departmental level, key findings and themes are at least discussed.
“No information is perfect. But better information on teaching effectiveness should allow for improved personnel decisions and faster professional growth” (Kane, p41, 2012)
Whilst measures of achievement gain, classroom observations and student surveys all have their relative strengths and weaknesses and no mechanism of teacher appraisal will ever be completely accurate, analysis of the literature reveals that by incorporating student surveys into traditional measures of teacher evaluation, teacher effectiveness can be improved and is more likely to result in higher performing students equipped with the skills equipped to succeed in the 21st Century (Ripley, p1, 2012). By building on existing research relating to what works with regards to student surveys and augmenting it with College specific questions relating to learning principles, a student survey and framework has been designed to be trialled across all departments in the High School through which, alongside traditional methods of teacher appraisal, it is hoped to elucidate a range of information through which teachers can better inform their practice. Although the student survey will not form part of official appraisals, it can be used by teachers alongside walk-ins by senior leadership, peer observations and achievement gains predicated on examination data to provide additional information upon which they can improve their effectiveness and therefore exemplifies the following four methods which Jensen (2011, p9) advocates:
Student performance and assessments [Exam results where available]
Peer observation and collaboration [Peer observation]
Direct observation of teaching and learning [Formal observation / SLT drop ins]
Student Surveys and feedback [To be introduced]
Whilst the introduction of student surveys is currently being restricted to the High School, there is no reason why it could not be extended to the Middle and Junior schools as even young children can evaluate their teachers relatively accurately with students in the same kindergarten class agreeing with each other across thousands of surveys (Ripley, p7, 2012).
Incorporating student surveys in to teacher appraisal (albeit it informally) as an additional instrument with such a high level of reliability that is independent of the race or income of pupils is truly, therefore, one of the most significant developments for education reform over the past decade however, the success will depend on the extent to which teachers themselves act on the results in order to ensure the feedback they receive is translated into higher performing students.
Brighouse, H. (2013) ‘Employing a student to criticize my teaching’. Accessed 04/01/14
Chetty, R., Friedman, J. & Rockoff, J. (2012) ‘Great Teaching’ Research, Summer 2012
Jensen, B. (2011) ‘Better Teacher Appraisal and Feedback: Improving Performance’. The Grattan Institute, Report No. 2011-3 APR 2011. Accessed 04/01/14 http://grattan.edu.au/static/files/assets/cdbcfbfe/081_report_teacher_appraisal.pdf
Davies, R. (2013) ‘Pupil generated feedback: the wisdom of crowds?’. Accessed 04/01/14 https://rccdavies.wordpress.com/2013/06/04/pupil-generated-feedback-the-wisdom-of-crowds/
Goldacre, B, (2013). Teachers! What would evidence based practice look like? Bad Science, March 15th 2013. Accessed 04/01/14 http://www.badscience.net/2013/03/heres-my-paper-on-evidence-and-teaching-for-the-education-minister/
Hattie, J. (2009) Visible Learning: A synthesis of over 800 Meta-Analyses Relating to Achievement. London: Routledge
Kane, T. (2012) ‘Capturing the Dimensions of Effective Teaching’. Education Next, Fall 2012. Accessed 04/01/12 http://educationnext.org/capturing-the-dimensions-of-effective-teaching/
Marshall Memo No. 453 http://www.marshallmemo.com/headlines.php
Marshall Memo No. 461 http://www.marshallmemo.com/headlines.php
Ripley, A. (2012) ‘Why Kids Should Grade Teachers’. The Atlantic, October, 2012. Accessed 04/01/14 http://www.theatlantic.com/magazine/archive/2012/10/why-kids-should-grade-teachers/309088/
Sandford, H. (2013) ‘Want to Improve Teaching? Listen to students’. Accessed 04/01/14
PICTURE CREDIT: Alamy in Tait, P. (2014) “Forget ‘Asian Tigers’ we need to focus on learning smarter”. The Daily Telegraph, 08 January 2014.