The Dim-Post

June 25, 2012

First do no harm

Filed under: economics,education — danylmc @ 8:33 am

Maui Street has a piece by Kelvin Davis summarising John Hattie’s research on 138 possible measures for improving education, which is the work that National and Treasury – supposedly – based their education reforms on:charter schools, league tables, performance pay, larger class sizes, improving teacher performance.

Charter Schools have an effect size of 0.20, or the 107th out of the 133 strategies that have some positive effect. Charter Schools are therefore an extremely pointless and expensive strategy.

There are still 40 strategies that are deemed pointless, but, are still more effective than Charter Schools.

Quality of teaching is 56/138. Professional development for teachers was 19/138. Teacher League tables and performance pay simply don’t register in Hattie’s research.

(I should also point out that very few of the studies in Hattie’s meta-analysis were conducted in New Zealand. And we have a very unusual education system: it performs better than most OECD countries, even though we spend less money on it. So things that might be ‘low hanging fruit’ that yield great gains in other education systems might have no effect here, because we’re already performing at a very high level.)

I was also interested to see that Hattie’s measurement of class size – the much vaunted claim that ‘it makes no difference’ (106/138) was in regards to classes that reduced in size from 25 to 15. National planned to change the range from 23-29 students/teacher to a set ratio of 27.5 students/teacher. They still think it was the correct thing to do, even though the very research that they cite fails to endorse this view.

But maybe they’re right! Who knows? Well, we have this amazing way of figuring out if things are ‘right’ or ‘wrong’ – it’s called the scientific method. You come up with a theory, test it and see what happens. If I was the Treasury Secretary, or the Minister of Education and I wanted to find out if these things did make a difference I’d run right out and set up some small scale trials in some classrooms, and measure the results, comparing them to normal classrooms in the same school. Seems to work? Let’s implement it across a bunch of different schools across the country and monitor them, and so on.

That’s how every other scientific discipline works, but for some reason (most) economists don’t roll like that – especially not the ones we have running our Treasury. When it comes to economics and public policy, the procedure is that economists come up with a theory, assume that they’re correct, even though they’ve made some really, really stupid mistakes in the very recent past and try and impose their theory on vital social infrastructure like the national education system without any empirical evidence whatsoever.

Unless we see a lot more humility, and a lot more actual science – in terms of empiricism and falsification – we should simply stop listening to those people.

About these ads

22 Comments »

  1. I had a meeting with two managers from Treasury’s National Infrastructure Unit last year, and raised a similar point – do they (or could they) use pilot programmes to test out new policies ahead of national roll-out? I actually used education as an example. (I think the specific example was around changing school opening hours to improve traffic around peak times, but that’s not important here.)

    They mentioned class sizes in response. Apparently, in Christchurch, the damage to infrastructure meant that many classes dramatically increased in size. The managers said these larger classes were being monitored to see whether the larger class sizes made a difference – and that preliminary results suggested minimal difference.

    I didn’t get the impression that there was any science behind this – there was no mention of a control group or how these kids were being assessed. More importantly, a forced, ad hoc response to a major natural disaster (with countless other factors playing in the background) is not exactly an ideal testing ground.

    Essentially, the managers acknowledged the benefits in pilot programmes, but their actual adoption of this approach appeared sub-optimal.

    Comment by J — June 25, 2012 @ 9:06 am

  2. If this were about education, you’d be right. But it’s not – it’s about politics and the snobbery of middle class voters.

    Comment by Me Too — June 25, 2012 @ 10:20 am

  3. I wonder what would happen if Labour/Greens campaigned on a policy of dis-establishing Treasury, on the basis that they are a parasitic menace ? I reckon you could crowd-source their policy advice and the resultant advice would be vastly superior – primarily because reality-based, testable ideas will typically always beat closed-minded, untested ideology.

    Of course, they would have to couch their terms a bit, possibly talking about “performance-based review” and “sensible, practical reforms”. Picking an outright fight with Treasury wouldn’t be smart move, but when they win in 2014 they could say they have a mandate and “we need to act quickly” and just get on with the job. Seriously, would anyone be exercised if we got rid of Treasury ?

    Comment by mikaerecurtis — June 25, 2012 @ 10:47 am

  4. Seriously, would anyone be exercised if we got rid of Treasury?

    You’re missing the obvious.

    Without a Treasury, where would we keep all our treasure?
    Those doubloons don’t bury themselves.

    Comment by Gregor W — June 25, 2012 @ 11:00 am

  5. I reckon you could crowd-source their policy advice and the resultant advice would be vastly superior – primarily because reality-based, testable ideas will typically always beat closed-minded, untested ideology.

    That’s not an argument for getting rid of Treasury – that’s an argument for getting rid of Politicians.

    Comment by Phil — June 25, 2012 @ 11:41 am

  6. “Well, we have this amazing way of figuring out if things are ‘right’ or ‘wrong’ – it’s called the scientific method. You come up with a theory, test it and see what happens.”

    Kind of difficult to do in the soft sciences. Would many parents be happy with their kids taking part in these small-scale trials?

    Comment by Hugh — June 25, 2012 @ 1:44 pm

  7. “Kind of difficult to do in the soft sciences”
    I can see it now: engineer a derpession, half the coutries of the world should borrow and spend to see if that gets them out, the other half should deregulate and balance the budget, and the remaining half should do nothing. What?

    Comment by Clunking Fist — June 25, 2012 @ 1:59 pm

  8. Most of NZ innovation has come from within the Schools. New ideas have been trialled perhaps in a single classroom then in a school and if it is good out into the wider school communities. This was often fostered by School Inspectors but they were abolished in 1989. Teachers and schools had been encouraged to run new ideas but the effect of National Standards/League Tables etc is to play it safe and narrow. I am certain that had NS been trialled and shown to do more than what is already known, Schools would readily have accepted the idea.
    Maybe the plan was not to convince but to agitate and demoralise the workforce.

    Comment by xianmac — June 25, 2012 @ 2:02 pm

  9. engineer a derpession, half the coutries of the world should borrow and spend to see if that gets them out, the other half should deregulate and balance the budget, and the remaining half should do nothing. What?

    I’m waiting for it to be revealed that the financial crisis was a randomised experiment by the Powers That Be to prove that in the next crisis (due to begin as soon as the German football team turns the last PIIGS standing into schweinebraten), we should do what Singapore and Australia did.

    Comment by bradluen — June 25, 2012 @ 2:38 pm

  10. “Maybe the plan was not to convince but to agitate and demoralise the workforce.”

    The sad thing is that if it wasn’t the plan the only other explanation is that the government are stupid and incompetent (and this does not preclude evil)

    Comment by nommopilot — June 25, 2012 @ 3:11 pm

  11. “soft sciences”

    Economics could be a soft science one day if economists had any intellectual honesty, but it is practised as a pseudoscience. The soft sciences and even the hard sciences can’t tick all the boxes – one can’t imagine astronomy as an experimental science, but its observations are treated rigorously and sociology also has its own disciplines. Economics alas lacks the discipline of even sociology in practice (largely discounting psychological and cultural factors in decision-making) while pretending to be as hard as classical physics. It’s that pretence that it’s a hard science and willful disregard and compensation for real, present complicating elements that makes it a pseudoscience.

    I’m reminded of the old joke about the physicist who tries to give a seminar to some dairy farmers. He draws a circle on the whiteboard and begins, “Consider a spherical cow in a vacuum…”

    Comment by Rhinocrates — June 25, 2012 @ 3:57 pm

  12. Economics could be a soft science one day if economists had any intellectual honesty, but it is practised as a pseudoscience.

    Technically, I understand that The Dismal Science is a semi-recognised quasi-academic proto-discipline of pseudo-science, available to post-graduate level in most erstaz-Universities and kindergartens.

    Comment by Gregor W — June 25, 2012 @ 4:50 pm

  13. was also interested to see that Hattie’s measurement of class size – the much vaunted claim that ‘it makes no difference’ (106/138) was in regards to classes that reduced in size from 25 to 15.
    Interesting. I saw Hattie interviewed on TVNZ7 and throughout the entire discussion his every reference to class sizes was in the context of reducing them having no discernible effect on outcomes. It’s a big stretch to then say that increasing class sizes doesn’t really matter. Maybe he doesn’t realise that this was in fact the the proposal?

    Comment by Neil — June 25, 2012 @ 6:14 pm

  14. @12 Thomas Carlyle coined the term “dismal science” because economics to him seemed to be depressing in its reductionism and materialism. Today most scientists – real ones that is – say that the term is half correct – it’s dismal, but it’s not science. Yes, it uses measurements, but so do astrology and phrenology. Where its useful, it’s simply accountancy, but in terms of making actual plans, policy-makers would be better off channeling the spirit of Jacques Derrida than they would be relying on these charlatans.

    Comment by Rhinocrates — June 25, 2012 @ 6:23 pm

  15. Yes, economists bad, physicists good!

    Seems to me that the “derpession” has already begun.

    Comment by Hugh — June 25, 2012 @ 6:30 pm

  16. No, pseudoscience embraced by people whose responsibilities require them to know better bad, empirical rigour good. Admittedly that doesn’t make as good a soundbite.

    Comment by Rhinocrates — June 25, 2012 @ 6:42 pm

  17. … sorry if that seemed snarky.

    Comment by Rhinocrates — June 25, 2012 @ 6:43 pm

  18. “Well, we have this amazing way of figuring out if things are ‘right’ or ‘wrong’ – it’s called the scientific method. You come up with a theory, test it and see what happens.”

    The UK govt has done some thinking on this: http://www.cabinetoffice.gov.uk/resource-library/test-learn-adapt-developing-public-policy-randomised-controlled-trials

    Comment by Scott — June 25, 2012 @ 10:48 pm

  19. Sigh!

    Lies, damn lies, statistics.

    Economics, economists, treasury.

    QED

    Comment by peterlepaysan — June 25, 2012 @ 11:12 pm

  20. Gregor W wrote: “Without a Treasury, where would we keep all our treasure?”

    we could just appoint a lone treasurer. How many people does it take to polish gold and cackle?

    Comment by kahikatea — June 26, 2012 @ 11:25 am

  21. Thomas Carlyle coined the term “dismal science” because economics to him seemed to be depressing in its reductionism and materialism.

    Thomas Carlyle first used the term when he was decrying liberal economists opposition to slavery. Here’s the quote:

    “Truly, my philanthropic friends, Exeter Hall Philanthropy is wonderful; and the Social Science—not a “gay science,” but a rueful [one]—which finds the secret of this universe in “supply-and-demand,” and reduces the duty of human governors to that of letting men alone, is also wonderful. Not a “gay science,” I should say, like some we have heard of; no, a dreary, desolate, and indeed quite abject and distressing one; what we might call, by way of eminence, the dismal science. These two, Exeter Hall Philanthropy and the Dismal Science, led by any sacred cause of Black Emancipation, or the like, to fall in love and make a wedding of it,—will give birth to progenies and prodigies; dark extensive moon-calves, unnameable abortions, wide-coiled monstrosities, such as the world has not seen hitherto!”

    The quote comes from Occasional Discourse on the Negro Question in which Carlyle was arguing for the reintroduction of slavery in the West Indies in order to better ‘regulate society.’

    See: http://www.econlib.org/library/Columns/LevyPeartdismal.html

    Comment by Quoth the Raven — June 26, 2012 @ 8:27 pm

  22. i fear you are shadow boxing. and so i look forward to the DimPost Economic Consultancy opening for business. Bastiat writes:

    In the department of economy, an act, a habit, an institution, a law, gives birth not only to an effect, but to a series of effects. Of these effects, the first only is immediate; it manifests itself simultaneously with its cause – it is seen. The others unfold in succession – they are not seen: it is well for us, if they are foreseen. Between a good and a bad economist this constitutes the whole difference – the one takes account of the visible effect; the other takes account both of the effects which are seen, and also of those which it is necessary to foresee. Now this difference is enormous, for it almost always happens that when the immediate consequence is favourable, the ultimate consequences are fatal, and the converse. Hence it follows that the bad economist pursues a small present good, which will be followed by a great evil to come, while the true economist pursues a great good to come, – at the risk of a small present evil.

    Comment by Ben — July 12, 2012 @ 4:47 pm


RSS feed for comments on this post. TrackBack URI

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

The Rubric Theme. Create a free website or blog at WordPress.com.

Follow

Get every new post delivered to your Inbox.

Join 413 other followers

%d bloggers like this: