Replication

I’m having a good conversation on the opensolaris.org community marketing list with Rich Sands of Java community marketing fame.

We’re going back and forth about what community marketing means on an open source project and how to implement such an activity. It’s more complex than you think. Sure, talking about it is easy — just read Cluetrain — but actually opening up your marketing and distributing the function among the diversity of a community is another matter altogether. Rich knows a great deal more about marketing than I do, but I’m coming at this from the communications aspect of marketing, something I do know something about. Anyway, in my last post — at 3 in the morning yesterday — I talked a little about my previous experience working at Tufts University and how the scientists, physicians, and veterinarians there reminded me a great deal of the engineers I’m working with here at Sun. Lots of sharing among the scientific community, the scientific literature is jam packed with studies that can be replicated for accuracy, and the whole thing had an open source feel to it. But I felt that the corporate marketing sitting on top of that system was just as closed as any marketing I’ve ever seen here in high tech. That’s the basic thought I’m exploring … the fact that it’s all coming back to me now due to the OpenSolaris project and it all seems so very familiar.

Then today I trip over this comforting news from the New Scientist — Most scientific papers are probably wrong. Nice. Here are the first three graphs:

Most published scientific research papers are wrong, according to a new analysis. Assuming that the new paper is itself correct, problems with experimental and statistical methods mean that there is less than a 50% chance that the results of any randomly chosen scientific paper are true.

John Ioannidis, an epidemiologist at the University of Ioannina School of Medicine in Greece, says that small sample sizes, poor study design, researcher bias, and selective reporting and other problems combine to make most research findings false. But even large, well-designed studies are not always right, meaning that scientists and the public have to be wary of reported findings.

“We should accept that most research findings will be refuted. Some will be replicated and validated. The replication process is more important than the first discovery,” Ioannidis says.

Well, these findings seem a bit extreme, but I’m not surprised by the general theme here. Back in school at Northeastern, I did a lot of research on the peer review system in scientific journal publishing, so I’m somewhat familiar with this (though my info is certainly out of date at this point). But the key here is that last sentence … that the most important part of the process may now replication. So, the system has to be open enough so you can actually replicate someone else’s study — to refute it, to validate it, to build on it. It’s the openness that drives the conversation that eventually becomes the massive scientific literature we have today, and it’s the openness that actually exposes a great deal of the bias and fraud in science as well. The scientific literature is at its very core a conversation, isn’t it? And openness ties it all together, I believe. Or transparency. Or sharing. Pick your term, but these characteristics are all found in well functioning communities among the people actually doing the work. Healthy competition is in there, too, but without sharing the community itself can’t grow and it can’t check itself, either. It feels so familiar …

Advertisements