Basic statistical literacy or lack thereof

In the prior thread, there was discussion about how 7 Days disclosed all its polling and lack of participation, which is technically accurate, but missing a bit: it’s fairly easy to find a 7 Days article that references the poll results without talking about poor participation, such as:

http://www.7dvt.com/2010legislative-survey-results.  On that piece there is no mention of the poor results.  You have to go to the piece which does “analysis” of the findings for that.  It indicates a low response rate but doesn’t acknowledge that that low response rate is relatively poor:

Of the 400 surveys sent out, only 30 came back with legitimate answers – a response rate of 7.5 percent. That’s better than direct mail – for which a 2 percent return rate is considered successful – and not too much worse than the turnout for a Burlington election, which was 23 percent on Town Meeting Day. Honestly, we hoped for better.

Just to clarify here: direct mail has an entirely different goal than surveys.  Direct mail isn’t intended to assess opinion.  It’s intended to sell things.

Surveys which attempt to assess opinion generally use sampling, so having a small sample size is acceptable, but it depends on the size of the sample group and how much stock you want to take in the results.  A sample of about 100 people out of 400 would give you a fairly solid confidence interval.  A sample of less than 1/3rd of that gives you very little confidence at all in the results.

Honestly, the comparison of the results to direct mail response results isn’t just poor judgment: it represents a complete and total lack of understanding of what statistics are and how they work.  

That piece also makes an odd inference.  When referencing the “most ethically challenged” item, it doesn’t just report on the results, but actually tries to connect them to a specific event:

During last month’s Mardi Gras parade in Burlington, Sen. Peter Shumlin climbed aboard an anti-Vermont Yankee float  sponsored by the Vermont Public Interest Research Group. No doubt he saw it as an opportunity in the wake of the Senate vote against the nuke plant. Turns out he may have used VPIRG polling information to his political advantage, too. Smart, affable and possessing sales skills some would describe as “slick,” the senator from Putney never misses an opportunity to advance his ambitious agenda…

This isn’t part of the survey.  It’s not part of anything involving the survey at all.  It’s just 7D waxing about Shumlin.  That’s fine, but it’s not analysis of the results.  It’s just opinion under the guise of information.

Which brings me to the last point about this survey’s “analysis:”  when you send out surveys like this to individuals with political ambition and goals, it’s very easy for a small group of people to game the survey to serve their own agendas.  

And to be fair: I expect this sort of nonsense from political organizations.  The Heritage Foundation routinely uses real data to create an “analysis” of the information designed and targeted to suit their political agenda.  It’s slimy, but I expect it from that sort of organization.

Media, on the other hand, has an ethical obligation to see to it that those doing “analysis” of their information actually have the basic competencies to do the job and to do so in a fashion which doesn’t serve a political agenda.  By performing this sort of “analysis” under the guise of a legitimate news organization, they are basically doing the work of the Dubie campaign by allowing them to take fake facts and present them as real facts.

There’s a lot I like about 7 Days, but the way they handled this?  I don’t know if it’s malice or incompetence at play here, but if it’s neither of those, it’s just reckless disregard for the truth.  

23 thoughts on “Basic statistical literacy or lack thereof

  1. and as it was presented as nothing but a survey applying statistical analysis makes no sense.

    You’re flogging that proverbial dead horse … the live one is the fact that Dubie and “Friends” are basing an outright lie on an acknowledged non-representative survey.

  2. Sort of what I was getting at in a comment yesterday about the confidence interval on the # being +/-18%.

  3. Polls are designed to provide statistically relevant numbers (grain of salt here) …

    such as x% of voters across the nation agree with position Y. Don’t mistake this for any claim to legitimacy for any given poll – there are ways to encourage a certain set of answers. The point being a poll puts forth an opinion as a representative view of a total population based upon the opinions of a sub-group of that total population.

    A survey, on the other hand, is intended to represent the views of those who responded. (Yes – there are lots of blurry lines, and we often run into mis-labeling.)

    Why is it germane to the 7 Days/Dubie and “Friends” lying topic? Simple: although it was crappy journalism, the 7 Days reporting on their survey results was nothing more than that – reporting on their survey results.

    Simple fact: the 7 Days survey and following story is not some aberration in being crappy reporting. We see this in most, if not all, of todays news reporting media at varying levels of appearance.

    Take, for example, the horrendous coverage that the phony ACORN videos garnered around the nation. The weasel welch couldn’t wait to join his reactionary, radical right wing allies in congress to attack and destroy ACORN and all the good they were doing in empowering those who the likes of welch have helped to dis-empower.

    Does that make the weasel welch’s actions the fault of the news coverage? Absolutely not – welch chose to do what he did (and to this day stands by it) of his own free will.

    The real ACORN aftermath story for Vermont is how untrustworthy and dishonest the weasel welch is.

    The real story here is the one folks seem absolutely resistant to go after: Dubie and “Friends” are passing out a blatant lie based upon this 7 Days story.

  4. You drag in your old “weasel Welch” line in a thread that has nothing to do with ACORN and you’ll have that.

Comments are closed.