Qualitative Quap

The blogosphere has been buzzing about a recent publication on science blogging. The study by Inna Kouper is “an attempt to understand current practices of science blogging and to provide insight into the role of blogging in the promotion of more interactive forms of science communication.”

She examined eleven blogs.

I must admit I have trouble getting my brain around qualitative research. To my mind, research should provide information that can be generalized or applied in some way. Many forms of qualitative research cannot be used this way.

I asked one of my colleagues who really likes this sort of analysis what he thought of a sample size of 11 for this study. He said that was a reasonable number depending on the goals and methodology of the study. A qualitative study would not necessarily require a random sample of a certain percentage of blogs; that’s quantitative research thinking!

So I sent him the paper.

He was not impressed.

[One] disturbing part of this is that the methods do not mention how or why the particular blogs were chosen.  There is no consistency and no specific criteria that is detailed.  That is important for any qualitative analysis because there should be a specific phenomenon that is being analyzed.  This author very loosely commented that the purpose was to look at science-related blogs and the responses. That is very broad and does not guide one to believe looking at only 11 blogs is useful data....even in qualitative studies.

He also comments that the qualitative analysis theory the study used was exceedingly difficult to find within the manuscript. His most damning criticism:

[More detail is needed on the] data reduction that was done.  Yes, the author details how items were coded.  However, this coding process was not validated by an independent reviewer to ensure they were coding in a similar manner.  That is the one way to ensure reliability and credibility of the data collection and data analysis process.  This was not done at all from what I read.

Qualitative research is a different world, but even in that world there are problems with this work.

For earlier posts on this study see:

A Blog Around The Clock

DrugMonkey

On Becoming a Domestic and Laboratory Goddess

I realize I have not credited my qualitative statistics expert; I will let him “out” himself, if he desires, in the comments section.