I’ve been doing a large number (25 so far) of medical questionnaire surveys in the past few days (in conjunction with getting my genome sequenced).

I have a very low opinion of questionnaire surveys in general but most of these have been excellent. Some exceptional problems have been caused by my own peculiarities:
I have symptomless asthma – I was only diagnosed with asthma when I volunteered for an asthma study as part of the non-asthma control group.
I have quite a few times been misdiagnosed with astigmatism, I first became aware of this when the axis changed by nearly 90 degrees from one optometrist to the next, checked it myself using the “blur circle” visible when viewing distant lights at night – if I had astigmatism then it would be an ellipse, but instead it was a circle with chunks cut out due to floaters. This makes me wonder how many other people are misdiagnosed with astigmatism.

Apart from that, NEVER design a survey with a choice of answers “Yes, no” or “Yes, no, don’t know”. Choice of answers “Yes, no, not sure” is much better because “not sure” includes the case “if I answer the question literally then you will draw the wrong conclusions”, which could also be summarised as “it’s more complicated than that” or “yes and no”. Some good surveys included an “I don’t want to say” option.

There are survey questions that are just stupid, like “do you get seasick?” Everyone gets seasick on a small enough boat in high enough seas unless taking anti-nausea medication. No-one gets seasick on a large enough ship in calm seas.

The questionnaire survey that prompted this post on the Holiday Forum is about side-effects of common medications. It is very badly constructed. Like the sequence, “this is a survey about the side effects of common medications”:
“Have you ever taken aspirin?”
“How long have you been taking aspirin?”
“Has there ever been a period of six months or more when you haven’t taken aspirin?”
“Give dates of all the periods of six months or more in which you were not taking aspirin.”
etc., but never a single question about dosage, the survey made no distinction between 400 mg per year and 1600 mg per day!

Another example:
Question: Are you taking … (an anti-anxiety medication)? Answer: Yes.
Question: Over the past week have you been feeling anxious? Answer: Yes.
Correct conclusion: You may benefit from increasing your dose.
False conclusion: Increased anxiety is an undesirable side effect of … (an anti-anxiety medication).
The survey contained no way for the questioner to distinguish between the false and correct conclusion.

I shudder at the thought that somebody is actually going to publish a scientific paper on the side effects of common medications based on the results of this questionnaire survey.

This leads to my question. How would you design a good questionnaire survey to determine the side effects of common medications?

OR

What’s the worst possible questionnaire survey that you could design?

I’ve never been sea sick on any kind of boat, and I’ve been on things from a 6ft tinnie to a large cruiser type boat.

mollwollfumble said:

What’s the worst possible questionnaire survey that you could design?

How would you rate our product:

good above average great fantastic

(Paraphrased from an actual customer satisfaction survey that was thrust upon me after a course i did)

Carmen_Sandiego said:

mollwollfumble said:

What’s the worst possible questionnaire survey that you could design?

How would you rate our product:

good above average great fantastic

(Paraphrased from an actual customer satisfaction survey that was thrust upon me after a course i did)

An optimistic survey?

Carmen_Sandiego said:

mollwollfumble said:

What’s the worst possible questionnaire survey that you could design?

How would you rate our product:

good above average great fantastic

(Paraphrased from an actual customer satisfaction survey that was thrust upon me after a course i did)

Hmm… formatting fail.

How would you rate our product:

() good
() above average
() great
() fantastic

(Paraphrased from an actual customer satisfaction survey that was thrust upon me after a course i did)

Do they ever use just a survey to collect ‘results’? (Unless you are news.com) Surely a medical survey is one part of a conclusion drawn from other sources well.

I don’t get sea sick either.

Arts said:

Do they ever use just a survey to collect ‘results’? (Unless you are news.com) Surely a medical survey is one part of a conclusion drawn from other sources well.

I don’t get sea sick either.

I’ve only ever been seasick on a small boat on a dead calm sea. Once.

All other times on boats, no sign of seasickness.

roughbarked said:

Arts said:

Do they ever use just a survey to collect ‘results’? (Unless you are news.com) Surely a medical survey is one part of a conclusion drawn from other sources well.

I don’t get sea sick either.

I’ve only ever been seasick on a small boat on a dead calm sea. Once.

All other times on boats, no sign of seasickness.

are you sure it was seasick? you may well have been actually ill and might have vomited on dry land that day too ;)

Arts said:

Do they ever use just a survey to collect ‘results’? (Unless you are news.com) Surely a medical survey is one part of a conclusion drawn from other sources well.

I don’t get sea sick either.

Yes, it’s called a Qualitative study.

http://www.xavier.edu/library/help/qualitative_quantitative.pdf

thanks Poik, that’s interesting.

passenger ships have stabilisation systems that stop the rocking motion – it will be very unlikely that you’ll get sick on a cruise liner

smaller boats don’t have stabilisation system – if you’re susceptible you’ll be heaving your guts out after a few hours

never go whale watching

wookiemeister said:

passenger ships have stabilisation systems that stop the rocking motion – it will be very unlikely that you’ll get sick on a cruise liner

smaller boats don’t have stabilisation system – if you’re susceptible you’ll be heaving your guts out after a few hours

never go whale watching

Things might have changed since the days of the Fairstar I suppose, but I recall people being seasick before we cleared the heads in Sydney harbour.

The thing with questionnaires like this is that they are not supposed to be designed to cater to the responders biases. I totally understand your frustration… So often they don’t offer an option which corresponds to the answer we want to give. But that doesn’t mean they are badly designed or implemented. It just means that the things we the responders want to say are not the things that those conducting the survey are interested in.

The very fact that most surveys like this appear to be badly constructed is a big clue to where the real problem lies. Do you really think the questioners are getting it consistently wrong?

No, the real problem is that people would rather give opinions than straight answers, ad for the most part the people asking the questions aren’t interested in our opinions.

Two things you should avoid getting simultaneously, sea sickness, and lockjaw.