Sometimes when I’m riding my bike I have imaginary conversations with real people. This morning I spoke with the Seattle Times editor who wrote this piece. Here’s a transcript of our discussion:
Me: So I’m still waiting to take my test.
Editor: Which test is that?
Me: Well, I’ve been reading your paper for over 35 years, assuming that at some point you’re going to give me an assessment on my comprehension of local, national and world affairs so that you’ll know how well you’re doing.
Editor: We don’t do that.
Me: Seriously? Isn’t it your job to inform your readers? How do you know how well you’re doing if you don’t assess them on the extent to which they’re informed?
Editor: First of all, writing a newspaper is a very complex undertaking. We do a lot more than report on current affairs.
Me: I guess that’s true.
Editor: And besides, not everyone reads the articles about local, national and world affairs. Sometimes they just read the sports page. Or the funnies.
Me: Imagine that. But can’t you do something to make us pay attention to the important stories of the day?
Editor: Sure! Did you see this headline? And did you read the story?
Me: Of course. Everyone did.
Editor: Unfortunately, however, we don’t get a story that juicy and interesting every day.
Me: That’s actually a good thing.
Editor: Yeah…that’s what I meant. The point is; informing the public is an interactive process. We can only do our part. At some point our readers have to actually read our articles. And it doesn’t matter how well we write something if no one reads it.
Me: So you think you should only have to be accountable for performing your job, using best professional practices, and not by the degree to which your readers actually understand the news, since not all of them are even reading what you write, and even if they are, they might not be paying attention to it?
Editor: Exactly.
Me: Let me ask you this: if such data did exist; if you could learn which parts of your articles the readers understood and which parts left them confused, would that data be useful?
Editor: Absolutely. We would use it to improve our performance. We would know what we’re doing well and what we need to focus on improving. We would love to have that information!
Me: But wouldn’t you worry that people would use it to evaluate the job you’re doing?
Editor: Are you serious? Who in their right mind would advocate taking a narrow assessment of our readers’ understanding of local, national and world affairs and use it to evaluate how well we perform something as complex as publishing a newspaper?
Me: I can’t imagine.
Jason, I completely agree with this:
“There are many strong protocols for classroom observation and strong methods for using summative test results to distinguish between great, mediocre, and crap teachers. Cincinnati has a great observation protocol that uses master teachers who are removed for a year or two to do teacher effectiveness rating for the district alongside administrators. These should certainly be taken into account as part of a teacher effectiveness program. But teacher effectiveness needs to be assessed to make better choices for kids, and effectiveness is not looking good, it’s achieving results.”
I want my effectiveness to be assessed by the combination of my students’ achievement and my effectiveness in the classroom. Two of my classes next year will be special education inclusion classes. I’ve taught inclusion for years, and I know that for many (not all) of those kids, the degree of proficiency with which they enter my classroom determines just how far I can take them in our time together. I have data (not state test score data, because 9th grade is not a state-tested grade level) which shows that my I’ve been able to raise students’ reading GL by four, five, or even six years…they enter demonstrating reading at a third grade comprehension and leave reading at a eighth or ninth grade level. To me, that is student achievement. However, they might still fail the tenth grade assessment…and I might therefore be judged a “failure” as a teacher, despite the measured and demonstrated gains the student made in their time with me.
It is complicated. I am not opposed to testing. I am not opposed to testing being part of the assessment of a teacher’s performance. However, unless districts/states are also willing to in vest in the observation protocols that you mention (and which I’ve advocated for often on this site) it will continue to be an unfair and ineffective means of assessing both teacher quality AND student growth and achievement.
Using one test on one kid to make a high stakes decisions about teachers is almost never a good idea.
Having results come out a year later is totally unacceptable and useless.
Ignoring half the subjects schools are charged with delivering is a terrible idea.
None of these things undermine the idea of using student academic success to judge teachers. At worst they’re excuses, but most of the time they’re just technical problems that have technical solutions. Even student motivation has a technical solution (though Kane has a great paper that shows sorting doesn’t have nearly the magnitude effect that teachers assert in a sample with data from several different cities– if sorting doesn’t have this massive effect to take account for, and different motivation is a part of this sorting, then motivations not a problem. Additionally, if you use multiple years of data or multiple classes for high school teachers, the effects of sorting are basically smoothed right out).
There are many strong protocols for classroom observation and strong methods for using summative test results to distinguish between great, mediocre, and crap teachers. Cincinnati has a great observation protocol that uses master teachers who are removed for a year or two to do teacher effectiveness rating for the district alongside administrators. These should certainly be taken into account as part of a teacher effectiveness program. But teacher effectiveness needs to be assessed to make better choices for kids, and effectiveness is not looking good, it’s achieving results.
There’s a middle ground. There always is. I don’t mind being accountable for my students’ performance. But I do mind when that performance is ascertained using a single, high-stakes test. Jason, in Washington we have a test that only measures reading and math at my grade level. The results don’t come out until the following year. And only the 10th grade version is actually used to determine graduation. For many students the tests essentially don’t matter. They really only matter when the students and/or their parents decide they matter. It’s against this backdrop that Mark voices our frustration at the notion of making a high-stakes test by students the source of evaluation of teachers.
What I like about Brad Jupp’s ideas are that the data used in teacher evaluations are decided upon by the teacher and the administrator. When I meet with my principal to go over the data from my students, we both know the kids who are represented by the numbers, and we view them in that context. That makes a world of difference.
Teachers should be evaluated based on how well they do their jobs. Period. Not by how motivated their students are to perform well on a standardized test.
Jason, you write:
“On top of that, it’s abdicating a core responsibility of teaching if you claim you’re not responsible for making children want to learn, just going through motions daily that would lead to learning so long as the students decide to come along for the ride. If that’s all a teacher does, then we may as well replace teachers with computers right now, because a computer with video and software can just as easily deliver information that can be learned so long as a student ‘wants’.”
I don’t think that anyone is saying teachers shouldn’t try to help students want to learn. However, ignoring that students have choice ignores that no matter how hard a teacher tries, some students will simply choose not to “ride along.” It’s a bit absolutist of you, Jason, to assert that acknowledging student willpower means that “we may as well replace teachers with computers right now.” As you’ve tried to describe, it is much more complex than that.
And I’m curious, Jason: what consequences do you think these tests can have that will be meaningful to all students?
If every one of them cared about grades, scores, or a diploma, then we wouldn’t be having this conversation. What rewards or consequences do you propose in order to motivate all kids?
“Eliminating” and “reducing’ are not the same thing.
I grew up in New York pre-NCLB but after 2000 or so. In the period between 2000 and 2005 we had what were considered tough state exams in every high school subject. You had to pass 8 of these by the time I graduated to get a high school diploma. Not only that, but your score on these exams was worth 20% of your grade for the year. Everyone had to pass one English exam, two social studies exams, two math exams, two science exams, and a foreign language. You had as many chances as needed to take these exams which were offered three times a year.
You know what? It didn’t stifle creativity, it ensured the full curriculum was delivered each year, and non-compliance by a few kids had far greater risk for the kids than anyone else.
Is that the solution? Eh, I had many problems with the regents. But non-compliance on a test meant non-compliance with the entire idea of getting a high school diploma.
I’m sorry, but this is exactly a scenario where “obstructionism” would be the correct response. I can’t speak for when the tests have no stakes for students, like the NECAP in RI, where I expect non-compliance to be widespread and disproportionately found in urban communities. But if the tests have consequences that are meaningful for kids, then non-compliance is referring to a very small number (statistically) of kids. What we’re left with will be a problem not large enough to have a big effect on any of the analysis that you’re critiquing.
On top of that, it’s abdicating a core responsibility of teaching if you claim you’re not responsible for making children want to learn, just going through motions daily that would lead to learning so long as the students decide to come along for the ride. If that’s all a teacher does, then we may as well replace teachers with computers right now, because a computer with video and software can just as easily deliver information that can be learned so long as a student “wants”.
Jason: what about the kid who got a B in Geometry a couple of years ago but who failed the state test because because his parents told him it was stupid? What about the kid of mine who got a L4 on the state writing test but who never turned in a single assignment and earned a 17% in my class?
You say that if we make the stakes right, the number of “non-compliers” drops down quite a bit. However, that’s not good enough. If, as you state, we as teachers are expected to bring ALL kids to a certain level AND inspire them to demonstrate that level of competence on demand, we are not allowed to have “non-compliers.” But, the clear problem is that there will ALWAYS be non-compliers. We’re talking about kids here… some would argue that teenagers are hard wired for non-compliance. Heck, even in a group of adults, there will always be non-compliers.
Hey Tom,
Thanks for the response. I have a couple of quick things I’d add.
First, the whole “high stakes” thing is a direct response to the first issue– kids won’t try if the test has no effect on their lives, but make the consequences and benefits real and the number of “non-compliers” drops down quite a bit. The right balance of stakes to assure that everyone views the learning process as high stakes without creating false pressure on just a few moments in a kids schooling is the challenge, but I wouldn’t throw the baby out with the bathwater because some kids can’t be bothered when they have a lot to gain and a lot to lose based on their effort.
Of course, in some sense, a teachers job is to bring every kid to our expectations, regardless of whether those students are inclined to “care” or not. Teachers are meant to be inspiring, and short of that, simply successful at getting through to a certain minimum level for all kids. The reality that some kids are tough to teach is really besides the point. The way some of your kids come in with no motivation, other kids would have hardly required more than a warm body in front of the class to ace the test.
As for the second point, that motivation is not randomly distributed, you’re quite right. However, in any growth metric or value added metric I’ve ever seen, the kids in remedial math in a high-needs area would be treated quite different from the affluent neighborhood art teacher. Students are almost always compared to those who scored similar to them on previous exams (which can effectively act as a poverty control, sadly), and for things like teacher effectiveness, FRPL eligibility and other special classifications are pretty much always included in the model to reduce biasness. In fact, we often use controls at multiple levels– the teacher-level, school-level, and sometimes district-level to estimate the effect of any one of those layers.
As for non-random assignment from within a school across unobserved characteristics, Tom Kane has done significant work to show the effects of non-random assignment in schools and has shown several great experimental controls that account for these effects and the robustness of a good model even without these controls at assigning student learning to individual teachers.
So I guess what it comes down to is the complexities are worth pointing out, but they are also well accounted for by many of the models that are out there in the literature. It’s only obstructionist if you think that complex means that it’s not worth doing or that the obstacles are insurmountable.
First of all, Jason, you can only push an analogy so far before it breaks, usually at the weakest point; which I think you found!
But to add on to Mark’s comment, it will always be unfair to hold teachers accountable for student achievement for at least two reasons:
1. Education is an interactive process. Students will only learn from ANY teacher if they decide to do their part. Teachers can influence that decision, of course, and we all do, but all of us also know some students that just don’t do their part.
2. And those students are not evenly distributed. A high school math teacher, teaching remedial math in a high-needs area will probably have a lot more “non-participants” than a high school art teacher working in an affluent area.
That said, student achievement is the bottom line, and we all need to focus on it; but we also need to be mindful of context, and we need to be very careful about using test data to evaluate teachers and programs, which is something that the newspaper editors don’t always appreciate. And then when teachers, through their unions, try to explain the complexities involved, they get labeled “obstructionists.”
As Jason points out, the power is in the consumer’s choice, not in the talent of the writer.
So why are we obsessing about the talents of teachers when ultimately, it is the student’s choice whether to learn? For any given newspaper article, there will be some readers who pick it up and some who put it down. For any given lesson, there will be some students who choose to listen and learn, others do not. The problem is this: teachers are being expected (mandates followed with threats of punishment) to make 100% of the students learn…despite the fact that students have free will just as readers do. If a newspaper were to be mandated that every adult read and understand every issue, then the absurdity of the mandate is clear: readers have a choice to read or not (even if they don’t purchase the newspaper and keep the beautiful free-market rolling). Students have a choice to learn or not–to do assignments or not. We can tear our hair out trying to “inspire” and “motivate” them to learn, but ultimately, it is their choice.
“At some point our readers have to actually read our articles. And it doesn’t matter how well we write something if no one reads it.”
After this line, you totally miss the mark. The obvious, major difference between a newspaper and teaching is right there.
If no one reads it (either because it’s not entertaining or informative), the newspaper eventually goes out of business. They are being assessed every day on how well they are doing their job because they have to respond to market pressures. You produce the content people want at the quality and quantity levels they want for the price you’re charging or you go out of business.
Of course, what this means is that newspaper readers have considerable influence of what is considered to be the job of the newspaper…
Reminiscent of John S. Taylor’s “Absolutely the Best Dentist” — I like your analogy and would love to hear some real responses from editors and other critics.
I also have imaginary conversations, but I am impressed with your civility. When I imagine myself talking to the Times I sound more like Richard Nixon!
Perfect!
I love it.