A Good Example

Today I am reviewing a manuscript that includes a qualitative research component. A while back some bad qualitative research rocked the blogosphere, so I thought I would strip the identifiers and provide an example of what good qualitative research sounds like in the methods section:

Two of the authors independently reviewed the open-ended responses to questions…/… and developed a coding scheme. After discussion, a common coding scheme was developed and subsequently reviewed by a third author... The original two authors then independently coded the database.

So scheme developed, confirmed by another researcher, and then applied. That is the way you do a qualitative study.

Art courtesy of PhotoXpress.

Qualitative Quap

The blogosphere has been buzzing about a recent publication on science blogging. The study by Inna Kouper is “an attempt to understand current practices of science blogging and to provide insight into the role of blogging in the promotion of more interactive forms of science communication.”

She examined eleven blogs.

I must admit I have trouble getting my brain around qualitative research. To my mind, research should provide information that can be generalized or applied in some way. Many forms of qualitative research cannot be used this way.

I asked one of my colleagues who really likes this sort of analysis what he thought of a sample size of 11 for this study. He said that was a reasonable number depending on the goals and methodology of the study. A qualitative study would not necessarily require a random sample of a certain percentage of blogs; that’s quantitative research thinking!

So I sent him the paper.

He was not impressed.

[One] disturbing part of this is that the methods do not mention how or why the particular blogs were chosen.  There is no consistency and no specific criteria that is detailed.  That is important for any qualitative analysis because there should be a specific phenomenon that is being analyzed.  This author very loosely commented that the purpose was to look at science-related blogs and the responses. That is very broad and does not guide one to believe looking at only 11 blogs is useful data....even in qualitative studies.

He also comments that the qualitative analysis theory the study used was exceedingly difficult to find within the manuscript. His most damning criticism:

[More detail is needed on the] data reduction that was done.  Yes, the author details how items were coded.  However, this coding process was not validated by an independent reviewer to ensure they were coding in a similar manner.  That is the one way to ensure reliability and credibility of the data collection and data analysis process.  This was not done at all from what I read.

Qualitative research is a different world, but even in that world there are problems with this work.

For earlier posts on this study see:

A Blog Around The Clock

DrugMonkey

On Becoming a Domestic and Laboratory Goddess

I realize I have not credited my qualitative statistics expert; I will let him “out” himself, if he desires, in the comments section.

Genius at Work

My son, Tim, plays many roles. The photo shows his debut at third base last Friday. He did field the ground ball and made the out at first.

I'm not going to tell a baseball story tonight. Unlikely as it seems, I'm going to share a story about AP Statistics. Tim is a sophomore, but has always done well at math. The final assignment this year in his stats class was to design a study and perform it. A fair amount of work went into this, and they worked in pairs. Tim and a buddy decided to test whether colors of gummi bears are equally distributed in packages.

Tim is a gummi connoisseur. Gummi bears, worms, fish, and fruit are all fair game, although the gummi bear has the best surface to interior ratio, in his opinion. He even managed to eat these with braces!

First, they had to assure a random sample. Thanks to the internet, they were able to find out that all gummi manufacturers fill their bags randomly. They did some sample size analysis, purchased a sufficient sample size, and then got together during study hall to count their bears. After covering a table in the library with a newspaper, they poured all 4 pounds into the middle in a giant mountain of gummi. As they were dividing it into same-colored piles, the librarian inquired about their project. I believe the exact quote was "No food in the library!"

They finally convinced her that this was a legitimate homework project, and she consented to let them take the table into the hallway outside the library door. They counted up the bears and have now turned in a paper documenting that red gummi bears outnumber the others, while yellow ones are less common. I'm hoping to get a copy of the paper to post because someone out there needs to know this factoid!

OK, so you've finished your research project. Each boy now has 2 pounds of gummi bears... which each proceeded to eat! In ONE DAY. Tim has decided he doesn't need to see a gummi bear anytime soon.

This story was the funniest thing I heard for several days, so I thought I would share. Remember, these are the rewards of having a kid in an AP class!

Gummi Bears Photo: copyright PhotoXpress.com