Surveys Say? 3 Tips To Help You (And Others) Ignore The Noise
By Kevin Lyons, Co-Founder and Chief Scientist, Flashvote
The two 10-term mayors we’ve met had at least one notable thing in common – they both learned to “ignore the noise” as soon as they were elected. But that is much easier said than done. Read on for some tips to help you and your electeds and colleagues ignore the noise too.
Tip #1: Friends Don’t Let Friends Follow Public Input?
Think of all the public input your agency gets from meetings, social media, online engagement forums and online surveys.
Now ask yourself… how often is that input a good representation of what your whole community thinks? Go ahead, guess a percentage. Got one? If you are like most government communicators, you probably guessed around 25% or less. Many roll their eyes and guess 0%. All of them guess under 50%. And you are correct. We looked at data from hundreds of scientific community surveys and compared it to the input governments got from their “engagement” channels (meetings, social media, online engagement forums and online surveys).
The sentiment from that input matched the sentiment from the scientific survey data only about 5% of the time – the same amount of support on the same side of an issue. Then about 25% of the time that input made a small group of residents seem much bigger than they were, but not enough to appear as a majority.
Finally, about 70% of the time the input received was the opposite of what the whole community thought. That’s a problem. But now think about how often your elected officials think that the input they hear is a good representation of the whole community. Did you guess much higher than what you thought? Correct again. When we ask elected officials they almost all guess above 50%.
That’s the real problem and that is how good intentions lead to bad decisions that disappoint residents and destroy their trust.
So “Tip #1” is to remind your electeds and other colleagues to ignore the noise. Point out that they could be twice as successful at making the community happy if they do the opposite of what they hear from public input.
Tip #2: Local Government is Like Ordering Pizza Toppings?
Another way to help visualize why public input is unrepresentative, is to present this simple story or role play exercise.
Picture this. You are planning a BIG pizza party for 10,000 people, or your whole community! Everyone has chipped in $5 and now your job is to figure out what pizza toppings to order. How would you do that?
You could order your own favorite toppings for everyone, but everyone would think you’re a jerk. You could just order what your friends want, but that would be wrong too. Duh. How about calling a public meeting? Then when the few but well-organized anchovy lovers make up 9 out of 10 attendees, do you order 90% anchovy pizza? Maybe you should look at social media comments or put up an online engagement forum? Sure, if you just want to hear from the folks who are riled up about the toppings they hate.
Clearly you need an online survey. So, you put a questionnaire into the wild and a “Friends of Pineapple Pizza” group finds it and keeps sharing it until their members all take it. Now you are ordering 90% pineapple pizza instead.
OK, so it’s obvious these are all terrible ideas – even though they are exactly what we end up doing in local government. They are all ruined by some people self-selecting to give input at especially high rates based on their special interest in the topic.
The best answer would be to take everyone’s order individually. But with 10,000 or more hungry mouths to interrogate that might cost more than the pizza.
But wait… if you could get orders from a representative sample of folks from your town – not just the loud ones, but a real cross-section of the community – you could get a really good idea of what everyone wants without having to ask each person That’s what a “scientific survey” is and it can be done with just hundreds of responses. Unless you are doing a scientific survey, never count comments or calculate percentages – that’s Tip #2.
Tip #3: The Interactive Card Trick Test Almost Everyone Fails
If you don’t like learning from math or stories, what’s left? A card trick!
Playing along with this 3-minute video might be the best way to understand our vulnerability to focus on the noise. Will you pass each question? Probably not. Which is how everyone learns. Tip #3 is to watch and share this link: https://www.flashvote.com/videos#card-trick
Enjoy and good luck!