Teacher ratings can’t tell good teachers from bad ones – back to the drawing board?

March 4, 2012

Corporate and business people who have lived through serious quality improvement programs, especially those based on hard statistical analysis of procedures and products in a manufacturing plant, know the great truths drilled by such high-quality statistical gurus as W. Edwards DemingThe fault, dear Brutus, is not in the teacher, but in the processes generally beyond the teacher’s control.

Here’s the shortest video I could find on Deming’s 14 Points for Management — see especially point #14, about eliminating annual “performance reviews,” because as Dr. Deming frequently demonstrated, the problems that prevent outstanding success are problems of the system, and are beyond the control of the frontline employees (teachers, in this case).  I offer this here only for the record, since it’s a rather dull presentation.  I find, however, especially among education administrators, that these well-established methods for creating champion performance in an organization are foreign to most Americans.  Santayana’s Ghost is constatly amazed at what we refuse to learn.

Wise words from the saviors of business did not give even a moment’s pause to those who think that we can improve education if we could only get out those conniving, bad teachers, who block our children’s learning.  Since the early Bush administration and the passage of the nefarious, so-called No Child Left Behind Act, politicians pushed for new measures to catch teachers “failing,” and so to thin the ranks of teachers.  Bill Gates, the great philanthropist, put millions of dollars in to projects in Washington, D.C., Dallas, and other districts, to come up with a way to statistically measure who are the good teachers, the ones who “add value” to a kid’s education year over year.

It was a massive experiment, running in fits and spurts for more than a decade. We have the details from two of America’s most vaunted and haunted school districts, Washington, D.C., and New York City, plus Los Angeles and other sites, in projects funded by Bill Gates and others, and we can pass judgment on the value of the idea of identifying the bad apple teachers to get rid of them to improve education.

As an experiment, It failed.  After measuring teachers eight ways from Sunday for more than a decade, W. Edwards Deming was proved correct:  Management cannot identify the bad actors from the good ones.

Most of the time the bad teachers this year were good teachers last year, and vice versa, according to the measures used.

Firing the bad ones from this years only means next year’s good teachers are gone from the scene.

Data have been published in a few places, generally over complaints of teachers who don’t want to get labeled as “failures” when they know better.  Curiously, some of the promoters of the scheme also came out against publication.

A statistician could tell why.  When graphed, the points of data do not reveal good teachers who constantly add value to their students year after year, nor do the data put the limelight on bad teachers who fail to achieve goals year after year.  Instead, they reveal that what we think is a good teacher this year on the basis of test scores, may well have been a bad teacher on the same measures last year.  Worse, many of the “bad teachers” from previous had scores that rocketed up.  But the data don’t show any great consistency beyond chance.

So the post over at the blog of G. F. Brandenburg really caught my eye.  His calculations, graphed, show that these performance evaluations systems themselves do not perform as expected:  Here it is, “Now I understand why Bill Gates didn’t want the value-added data made public“:

It all makes sense now.

At first I was a bit surprised that Bill Gates and Michelle Rhee were opposed to publicizing the value-added data from New York City, Los Angeles, and other cities.

Could they be experiencing twinges of a bad conscience?

No way.

That’s not it. Nor do these educational Deformers think that value-added mysticism is nonsense. They think it’s wonderful and that teachers’ ability to retain their jobs and earn bonuses or warnings should largely depend on it.

The problem, for them, is that they don’t want the public to see for themselves that it’s a complete and utter crock. Nor to see the little man behind the curtain.

I present evidence of the fallacy of depending on “value-added” measurements in yet another graph — this time using what NYCPS says is the actual value-added scores of all of the many thousands of elementary school teachers for whom they have such value-added scores in the school years that ended in 2006 and in 2007.

I was afraid that by using the percentile ranks as I did in my previous post, I might have exaggerated or distorted how bad “value added” really was.

No worries, mate – it’s even more embarrassing for the educational deformers this way.

In any introductory statistics course, you learn that a graph like the one below is a textbook case of “no correlation”. I had Excel draw a line of best fit anyway, and calculate an r-squared correlation coefficient. Its value? 0.057 — once again, just about as close to zero correlation as you are ever going to find in the real world.

In plain English, what that means is that there is essentially no such thing as a teacher who is consistently wonderful (or awful) on this extremely complicated measurement scheme. How teacher X does one year in “value-added” in no way allows anybody to predict how teacher X will do the next year. They could do much worse, they could do much better, they could do about the same.

Even I find this to be an amazing revelation. What about you?

And to think that I’m not making any of this up. (unlike Michelle Rhee, who loves to invent statistics and “facts”.)

You should also see his earlier posts, “Gary Rubenstein is right, no correlation on value-added scores in New York city,” and “Gary Rubenstein demonstrates that the NYC ‘value-added’ measurements are insane.”

In summary, many of our largest school systems have spent millions of dollars for a tool to help them find the “bad teachers” to fire, and the tools not only do not work, but may lead to the firing of good teachers, cutting off the legs of the campaign to get better education.

It’s a scandal, really, or an unrolling series of scandals.  Just try to find someone reporting it that way.  Is anyone?

More, Resources:


Rick Perry is the new Corrupt Bargain

November 1, 2010

The fiercely independent Democratic Blog of Collin County compiled a series of Burnt Orange Report posts that make the case that Rick Perry should be retired from the governorship, at a bare minimum.

Will voters wake up before Tuesday, and do the right thing?

Rick Perry’s Cover-Up and Corruption

From the BOR:

Rick Perry’s Cover-Up and Corruption: Texas’ Dropout Crisis

Rick Perry’s Cover-Up and Corruption: Texas Forensic Science Commission

Rick Perry’s Cover-Up and Corruption: Ethics Complaints

Rick Perry’s Cover-Up and Corruption: Emerging Technology Fund

Rick Perry’s Cover-Up and Corruption: Political Appointees

Rick Perry’s Cover-Up and Corruption: Secret Schedules

Rick Perry’s Cover-Up and Corruption: The $500,000 Land Deal

Rick Perry’s Cover-Up and Corruption: Texas Youth Commission

Rick Perry’s Cover-Up and Corruption: Teacher Retirement System

Rick Perry to Launch National Book Tour, Won’t Commit to Full Term as Governor

Bonus points if you know off the top of your head where “corrupt bargain” plays in U.S. political history.


Wegman Scandal: Attack on climate scientists based on shoddy scholarship

October 4, 2010

John Mashey assembled a massive document that nails down the case that bad science and politics make the complaints against scientists and the science that indicates global warming occurs, and can be attributed to greenhouse gases.  It is a scandal, though it’s unlikely to be reported that way.

Mashey’s entire paper — and it’s very, very large — is published at Deep Climate.

Mashey’s paper indicts staff work done for Rep. Joe Barton, R-Texas — not that any of the voters in Barton’s district will let this major breach of ethics sway their votes, but those who want to vote against him can be gratified that they are on the moral side of the ballot.

Mashey wrote:

This report offers a detailed study of the “Wegman Report”: Edward J. Wegman, David W. Scott, Yasmin H. Said, “AD HOC COMMITTEE REPORT ON THE ‘HOCKEY STICK’ GLOBAL CLIMATE RECONSTRUCTION”(2006).

It has been key prop of climate anti-science ever since. It was promoted to Congress by Representatives Joe Barton and Ed Whitfield as “independent, impartial, expert” work by a team of “eminent statisticians.” It was none of those.

A Barton staffer provided much of the source material to the Wegman team. The report itself contains numerous cases of obvious bias, as do process, testimony and follow-on actions. Of 91 pages, 35 are mostly plagiarized text, but often injected with errors, bias and changes of meaning. Its Bibliography is mostly padding, 50% of the references uncited in the text.  Many references are irrelevant or dubious.  The team relied heavily on a long-obsolete sketch and very likely on various uncredited sources. Much of the work was done by Said (then less than 1 year post-PhD) and by students several years pre-PhD. The (distinguished) 2nd author Scott wrote only a 3-page standard mathematical Appendix.  Some commenters were surprised to be later named as serious “reviewers.”  Comments were often ignored anyway.  People were misused.

The Wegman Report claimed two missions: #1 evaluate statistical issues of the “hockey stick” temperature graph,  and #2 assess potential peer review issues in climate science.  For #1, the team might have been able to do a peer-review-grade statistical analysis, but in 91 pages managed not to do so.  For  #2, a credible assessment needed a senior, multidisciplinary panel, not a statistics professor and his students, demonstrably unfamiliar with the science and as a team, unqualified for that task.   Instead, they made an odd excursion into “social network analysis,” a discipline  in which they lacked experience, but used poorly to make baseless claims of potential wrongdoing.

In retrospect, the real missions were: #1 claim the “hockey stick” broken and #2 discredit climate science as a whole. All this was a facade for a PR campaign well-honed by Washington, DC “think tanks” and allies, underway for years.

Now, if only Mashey had some e-mails stolen from Joe Barton, we could get some traction on the issue, eh?  ::wink-wink, nudge-nudge::

One may wonder what it will take to rehabilitate the skeptical side of the debate, to the point that they contribute more than mau-mauing.

Mashey’s paper makes that case that Joe Barton worked hard to pull off a great, hoaxed political smear, with a high degree of success.  Who will have the backbone to do anything about it?  Global cooling will proceed to the next ice age before any Republican shows backbone, I predict.

But, how long before the Fort Worth Star-Telegram or the Dallas Morning News picks up the story?

Other Texas bloggers?  Anyone?

It’s not an air-tight legal brief (I could quibble with some of the legal material), but in a better world, a world where politicians actually do good politics and public servants do public service, the House Rules Committee and Ethics Committee would be reading Mashey’s piece, and asking pointed questions.  U.S. attorneys in Washington, D.C., and the Northern District of Texas, would also be downloading Mashey’s piece, and puzzling it out.  Journalists in Dallas, Fort Worth, Austin, Galveston and Houston in Texas, and Washington, D.C., and New York, would also be poring over the piece.  Ken Cuccinelli in Virginia would also be paying attention to it, if he were concerned about justice.

More (watch for updates):


Fishy education software bill out of Utah

October 28, 2008

Remember about a year ago when Utah was all atwitter over a voucher proposal that was on a ballot?  Remember all the talk about saving money in education?

Utah Education Issues explains odd features in an omnibus funding bill recently passed by the Utah Legislature (The Economist praised Utah’s efficiency*).  Among other things, it gives away $1 million to an educational software company that will provide families with reading software — at a fantastic pricetag of $3,400 per installation (computer included, but still . . .).

Describing the smell of this bill doesn’t come close to the total repugnance — go read the report.  Fewer than 300 families can be served at that price, statewide.  One might suspect the true beneficiaries of this bill are not Utah voters, not Utah educators, nor even the Utah families who get the freebies.  Did I mention this involves a major publisher of public school textbooks?

It’s a commendable job of reporting for a blog, no?

Footnote:

*   The “cultural thing”, as businessmen from out of state delicately refer to Mormonism, helps in other ways. Utah’s almost universal conservatism makes for stable, consensual politics. It took the state legislature just two days last month to plug a $272m hole in the budget. By contrast, California’s budget was 85 days late. Nevada’s politicians are preparing for a nasty fiscal fight next year.


Oh, that explains it

May 16, 2008

Adnan Oktar ‘s conviction on charges of profiting from what amounts to a sex slave operation was a set up, he said.

Who would do such a dastardly thing?  The communists and the Freemasons!

This interview is from last September, but so far it’s perfectly in line with what his PR flacks are saying since the sentencing:  Video, selected transcripts.

I’m sure that’s what he tells his wife, Morgan Fairchild.

How can you keep from thinking this stuff is parody?  It looks and sounds like slightly amateurish “David Letterman” or “Saturday Night Live” routines.


Texas creationism scandal only one of many

December 6, 2007

McBlogger has an interesting, Texas-based take on the scandals at the Texas Education Agency: It’s a hallmark of Republicans in Texas government.

In other words, other agencies are similarly screwed up, and the common thread is Republican appointees out of their depth and unaware of it.

(Do short posts make this place start to look like Instapundit? Looks only — check the substance.)

Tip of the old scrub brush to Bluedaze.


Follow

Get every new post delivered to your Inbox.

Join 2,332 other followers

%d bloggers like this: