The Dim-Post

April 1, 2014

Why are MSM poll stories bivariate?

Filed under: Politics — danylmc @ 8:57 am

Lots of discussion around the blogosophere about polling yesterday. One thing that struck me is that political bloggers like to present polls as time-series data going back to before the last election while the mainstream media outlets that pay for all this data present them as cross-sectional, and generally only compare them to the previous poll that they conducted. Why is that? In my experience people like to be able to look at ‘the big picture’ and see the trends. TVNZ (for example) has conducted fourteen polls since the 2011 election. They’ve spent a load of money on them: why not throw that all up on the screen instead of just comparing the new poll to the one before it?

About these ads

32 Comments »

  1. Torben Akel did a good job of that here. I jumped out of my seat and clapped when I saw him use a time series. http://www.3news.co.nz/Tricky-road-ahead-to-election-for-National/tabid/1348/articleID/335161/Default.aspx

    Comment by pete — April 1, 2014 @ 9:09 am

  2. Less long term data leads to a perception of bigger shifts, which make for more striking headlines.

    Comment by kalvarnsen — April 1, 2014 @ 9:14 am

  3. Do any of the journalists involved actually have any statistics-based qualifications? You’d think with resources MSM likes to put into polls, they’d have the odd job opening for front people who are qualified to actually understand and explain them to people.

    The cynical part of me, though, says that the handling of this type of thing is mostly about entertainment and retaining ratings.

    Comment by izogi — April 1, 2014 @ 9:17 am

  4. “Torben Akel did a good job of that here. I jumped out of my seat and clapped when I saw him use a time series.

    Comment by pete — April 1, 2014 @ 9:09 am”

    O-kaaaaaay. Too much screen time not enough big bright shiny thing in sky time?

    Comment by TransportationDevice A7-98.1 — April 1, 2014 @ 9:24 am

  5. @TransportationDevice A7-98.1: “not enough big bright shiny thing in sky time?”

    I’m in Dunedin, we don’t have one of those.

    Comment by pete — April 1, 2014 @ 9:34 am

  6. In my experience people like to be able to look at ‘the big picture’ and see the trends.

    Maybe you don’t know the right kind of people … .

    Comment by Andrew Geddis — April 1, 2014 @ 9:45 am

  7. Wrong. He knows the right kind of people. There just aren’t enough of them.

    Also, I suspect most journalists, as discussed in a thread elsewhere, are not numerate! Most show themselves to be barely literate.

    Comment by David in Christchurch — April 1, 2014 @ 10:03 am

  8. If only everybody was as intellectually discerning as commenters here at the Dim Po…

    Damnit, I almost managed to type that with a straight face!

    Comment by kalvarnsen — April 1, 2014 @ 10:13 am

  9. Damnit, I almost managed to type that with a straight face!

    Given the date, you might have gotten away with it, too.

    Comment by Andrew Geddis — April 1, 2014 @ 10:14 am

  10. It is the same for the vast majority of statistics the MSM presents that are part of a long data series. Eg house prices, exchange rates, share prices, inflation. They might make a comparison to a month ago, or a year ago. It’s easier to write about probable causes in a deterministic style, rather than try to write nuanced articles about potential emerging trends

    Comment by Swan — April 1, 2014 @ 10:16 am

  11. Guessing the overwhelming constraint in primetime news hour is avoiding losing viewers by providing too much pesky ‘pointy-headed’ information. Or detailed analysis of context.

    Comment by Ethan Tucker — April 1, 2014 @ 10:19 am

  12. You may find this aspect of the just released NZ Political Polling Code relevant.

    “Stories should focus on significant trends, which may not be just between the current and last poll, but over a number of polls”

    Comment by dpf — April 1, 2014 @ 11:09 am

  13. Comparing this poll to the last lets the journalists attribute any apparent movement (or lack thereof) to recent news stories/scandals. Such as: “Labour’s political fortunes have plunged/skyrocketed/stagnated following video emerging last week of Shane Jones eating a live frog”.

    Comment by RJL — April 1, 2014 @ 11:14 am

  14. AND do the ‘reporters’ have any understanding of error .
    Most changes from poll to poll are within sample error, (+- 3%) so don’t mean a lot.
    AND the sample error is not usually reported up front either.

    Comment by DV — April 1, 2014 @ 11:24 am

  15. @DV: margin of error for changes is more like +/-4.5%. There’s error in both the before and after measurement, so you get a larger error when you combine them.

    Comment by pete — April 1, 2014 @ 11:38 am

  16. Why is television news?

    Why is political blogging?

    Why?

    Comment by George — April 1, 2014 @ 11:45 am

  17. So, in summary:

    1. Journalist don’t know stats.
    2. Time series are too hard for audiences
    3. News will be less dramatic

    Any others? I’ll add:
    4. Any plot that requires close study cuts down on the reporter’s face-time.

    I basically rely on Danyl’s series since he first began doing them, to do any thinking about polls. Anything less is just some of those dots in one of his mighty mega plots. It’s really good stuff. It’s like a one picture summary of a thousand bullshit news stories, and what it all really means. I’d say the real reason that it doesn’t get onto the news is because what it really means, most of the time, is nothing. Which is unpalatable news to many.

    The data analysis I’d really like to see is not time series oriented, but based around probability. Like what iPredict does, but with stats rather than greedy punters as the underlying basis. I’d like to know the relative probabilities, based on the data collected, that certain outcomes will happen. Sampling doesn’t really directly answer this question, and I think that’s a major source of the confusion that arises in using it as the primary analysis technique. We do have the power to Bayes’ box and Monte Carlo this stuff now. I think there’s big opening in news reporting in NZ for this, that it would be really quite popular, since it directly answers the question of greatest interest, without all the caveats and confusions that come into all of this estimation of mean and variance stuff.

    Comment by Ben Wilson — April 1, 2014 @ 12:04 pm

  18. Ben Wilson: The sounds very nice but it sounds to me like one could easily fall into the the trap of overanalysing data to produce conclusions that are still meaningless because the data itself is not that reliable.

    Comment by wtl — April 1, 2014 @ 12:51 pm

  19. One could. On the other hand, one could make considerably more interesting discoveries and statements about the discoveries. I’d find it a lot more insightful that “here’s a probable mean. Now speculate wildly about why, what it means, where it’s going”. But one would need to be a statistician to do it, and others would have to turn their brain on to understand it. So maybe it’s not fit for mass media after all.

    Comment by Ben Wilson — April 1, 2014 @ 1:03 pm

  20. You’re basically asking for an NZ version of Nate Silver.

    Damn, it looks like onetwenty.com and onehundredtwenty.com are already taken.

    Comment by Phil — April 1, 2014 @ 1:36 pm

  21. heh, but the .co.nz ones are still around.

    Comment by Ben Wilson — April 1, 2014 @ 3:34 pm

  22. @Phil: I don’t know if you remember but Danyl did once state that it was his goal to become New Zealand’s Nate Silver.

    Comment by kalvarnsen — April 1, 2014 @ 5:37 pm

  23. Look at the way they report share market price movements and other financial “news”. This stock is up 5c (never a percentage) on (some random news item which has no clear causal link) with turnover of (absolute number which is not compared to average). No different. The demand for information outstrips the supply.

    Comment by Stephen J — April 1, 2014 @ 7:02 pm

  24. It’s so sweet to see the faith put in TV journalism, it’s a beauty contest and tv journos should be seen as miss Venezuela with an earnest expression. They are quite amusingly stupid and ill informed and it’s a delight to watch night after night for amusement and merriment.

    Comment by David — April 1, 2014 @ 8:39 pm

  25. Media outlets provide data and information when what people really crave are knowledge and wisdom. But we get what we tolerate.

    Comment by Sacha — April 1, 2014 @ 9:40 pm

  26. Perhaps the reason MSM don’t do time series of political or business stats is lack of time to explain it. Blogs aren’t limited by column inches or screen seconds. And Torben Akel’s piece was a whopping ten minutes – practically a feature documentary by modern standards.

    Re iPredict beware it’s pseudo science. The thing is easily rigged and has no value whatsoever in predicting results.

    Comment by nigelsagentinthefield — April 1, 2014 @ 9:51 pm

  27. >The thing is easily rigged and has no value whatsoever in predicting results.

    That’s not true. It’s just not rock solid science. But it’s was a lot more accurate at the last election. There are some convincing reasons why that might be likely, too. Essentially, there’s no limit of any kind to the kind of information that can be used to inform the betting there. They can use any stats, any news, any analysis. Attempting to push the market can lead to just losing money to market takers. It’s not like the gamblers in there don’t read the polls, or apply Danylesque fudge factors of their own.

    Seriously, if you really think it has no predictive power then you’re missing out on a golden opportunity to make money out of it. If you think you can make better predictions using science, have at it. There should be undervalued contracts on there, big time. But be warned, you wouldn’t be the first person to think of that. If you’d been using the scientific power of opinion polls last election and putting big money against your confidence, you’d have lost your shirt.

    Comment by Ben Wilson — April 2, 2014 @ 12:05 am

  28. I’m struggling to see Patrick Gower as Miss Venezuela

    Comment by Adrian — April 2, 2014 @ 3:35 am

  29. >Media outlets provide data and information when what people really crave are knowledge and wisdom.

    I wish. They often just provide opinion and conjecture without any data available to view. As mentioned rather than a journalist ratting on about how “this poll is terribly for party y and is a result of x.” I’d prefer a series of graphs showing trends with significant events marked – but of course that takes more work than just saying that x happened because of y even if it’s just opinion.

    Comment by bmk12bmk — April 2, 2014 @ 5:22 pm

  30. >That’s not true. It’s just not rock solid science. But it’s was a lot more accurate at the last election.

    Betting markets can be horribly wrong too – especially when it comes to over predicting a favourite. A recent example of this was the South Australian state election where the Liberals were $1.01 favourite – Labour just got re-elected. The funny thing about this was the polls pretty accurately predicted the final margin and if you put this margin into a seat calculator there it predicted the result. So clearly betting markets can incorrectly predict outcomes even based on correct data.

    Comment by bmk12bmk — April 2, 2014 @ 5:26 pm

  31. Ben. Trust me , IPredict is an easy rig. Several times someone has bank rolled a certain prediction by constantly buying an outcome at a certain value. Len Browns post-election stock to quit was a prime example. Someone kept booking hundreds of trades and it seemed crazy that they would but when you add it up, they spent about $300 all up. Not much for someone with deep pockets who’s looking for news stories promoting a certain outcome. It also happened with Bob PArker and now the Conservative party election result seems over inflated beyond even Colin Craig’s hope or reason. It’s a bit too common to be inside knowledge.

    And yes I do play the market and make a bit from the strange right wingers deliberately throwing money away.

    Comment by nigelsagentinthefield — April 3, 2014 @ 8:07 pm

  32. >Someone kept booking hundreds of trades and it seemed crazy that they would but when you add it up, they spent about $300 all up.

    Yup, you’re talking a low traded, short term stock. Yes, you don’t put a lot of trust into those values. Also, it’s possible that the person WAS trading on inside information, and was just unlucky. I don’t see you having refuted the odds that were generated just because a highly volatile outcome didn’t happen. Maybe it was Whale chucking bets on before releasing his next round of smear.

    >Not much for someone with deep pockets who’s looking for news stories promoting a certain outcome.

    No, in fact it’s peanuts even for the ALCP. Which is why I don’t think it’s significant. It’s well within the power of the political parties to push against each other. And don’t forget that the deepest pockets of all are those of people who like making money.

    Comment by Ben Wilson — April 4, 2014 @ 5:33 pm


RSS feed for comments on this post. TrackBack URI

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

The Rubric Theme. Create a free website or blog at WordPress.com.

Follow

Get every new post delivered to your Inbox.

Join 413 other followers

%d bloggers like this: