Research: Good (sometimes), Often Bad and/or Ugly

A few days back Andrew Gelman favorably mentioned a Bloomberg column ("Why We Fall For Bogus Research") that touched on recent problems of unreplicability in psychology research:
http://andrewgelman.com/2015/09/12/yes/

The specific column referenced is here:
http://www.bloombergview.com/articles/2015-08-31/why-we-fall-for-bogus-research

Then, elsewhere, Tom Siegfried put up a great dove-tailing piece on the problems of so-called "evidence-based" science, centered around medicine here:
http://tinyurl.com/oj6otbx

 These pieces focus primarily, not-so-much on the quality, integrity, or analysis issues (which are significant themselves) of research studies, but on the issue of generalizability: how far out (or how big a population) can the results of a given study really be applied to? This is especially an issue for "social" or "soft" sciences. And the generalizability problem in turn touches on the even broader issue of "induction" in science. Previous work has focused on the "WEIRD" study samples employed in much western research (subjects drawn solely from a "Western, Educated, Industrial, Rich and Democratic" population pool; hardly a random sample). This deserves a longer post, but I don't have time.

I'll just say that people like to discuss "science" as if it were some monolithic practice, when in fact inherent differences reside between various fields of science: psychology, biology, medicine, anthropology, physics, engineering.... (there are even huge differences between practitioners WITHIN any given field!). Science is a continuum (ranging from excellent to good to average to mediocre to poor to piss-poor!), there is NOT some clean binary science/pseudoscience division; plenty of published science barely ranks above pseudoscience, if critically adjudged.
Even though I was a psychology major, I rarely saw psychology research that I'd hold up as good science. As recently as this year I took part in a psychology study under a well-funded, fairly well-known researcher (and author)... that I believe was essentially junk science (as commonly funded by the likes of NIH -- and to those who shudder every time NIH's funding is threatened, I have to wonder how you regard your own research... because chaff (of which there is plenty) is what is intended to be cut from the NIH budget; if you're doing excellent research you have little to fear from such cuts, but if you're just publishing-or-perishing for the sake of publishing-or-perishing pressures, well.... Anyway, enough soapbox, take a gander at the Bloomberg and Siegfried pieces (especially the latter).

One reason, by the way, that I think these issues are SO important is because the growing (even scary) anti-science sentiment in this country is, I believe, the result of a citizenry growing up with a false, idealized version of science in their heads.  If they understood the 'messiness' of science from the get-go they could better accept it, but learning of it only later in life (as if they'd been fed an elitist lie, before) they turn against it. :-(

...ADDENDUM:  Gelman has now done a further followup on this general subject:
http://andrewgelman.com/2015/09/16/harsh/