I'm sometimes amused by 'scientists' on the Web calling themselves "skeptics" only to find that they're aiming most of their fire at what I can only call "low hanging fruit": creationists who think the Earth is less than 10,000 years old, psychics who bend spoons, UFO abductees who've taken trips to the planet Kazaar (or some such), etc. etc. They often proudly call themselves "evidence-based" scientists… but don't seem to acknowledge that scientific "evidence" itself can be highly subjective, manipulated, and tainted fare... requiring skepticism itself.
What I've always wanted to see is more "skeptics" turning their keen eyes on the likes of the Journal of the American Medical Association, New England Journal of Medicine, CELL, Nature, Science, etc. Most "science" is so poorly done it doesn't even see the light of publication, but even research that does make its way into such 'premier' publications (let alone lesser ones) often escapes the scrutiny it deserves, receiving a sort of 'free ride' once published (the data, methods, underlying premises/assumptions, and of course conclusions, never being adequately challenged, nor replicated; and don't assume 3 peer reviewers have done their job either!). Most who have been heavily involved in research, IF they're honest about it, will admit that journal articles, as composed, often don't accurately reflect the actual work, as carried out. But with the digital age upon us, the needed sort of ongoing, ever-watchful skepticism is finally emerging.
Dr. Ivan Oransky has made a job of keeping track of journal "retractions" (often, but not always, for malfeasance) from research journals, with his wonderful, must-read "Retraction Watch" blog (sub headed: "Tracking retractions as a window into the scientific process"). And a recent 'Peer Review Congress' meeting in Chicago, well-tweeted (hashtag #pcr7) by Oransky and others, shined light on many of the important research issues that don't get voiced enough… and these are NOT new, but have been around for a long, long, long time, just finally getting the attention deserved. Anyone who has followed this area will be familiar with the name "John Ioannidis" as one of the most-vocal of those who've questioned the reliability/validity of much research. There are also now a number of twitterers (some better than others) focusing on biomedical/research skepticism, though 140 characters isn't always much to work with!
My sense is that a majority of the retractions that Oransky reports on involve the biomedical sciences, but even mathematics papers or journals occasionally appear among his subjects (sometimes almost laughably). So math, the 'queen of the sciences,' is not immune from sloppiness (or even fraud). Check these out:
http://retractionwatch.wordpress.com/?s=mathematics
WHY the heck do I mention all of this now? Well, nothing earth-shattering, but in preparing the John Casti quote for the post that I did last Sunday on Math-Frolic (thought it would make a nice Sunday meditation), I came across information that disturbed me. Casti (who focuses on "complexity" theory) has long been one of my favorite science writers/thinkers; he is both a good explainer, and a thought-provoking one; I've read him for decades. So I was surprised/disappointed to learn from the internet that he'd been caught extensively plagiarizing material in the past -- multiple times... that doesn't make his material any less interesting or important to me (he plagiarizes from good people ;-), but it does throw a troubling ethical shroud over that material. Here are 3 web links that address Casti's sins:
http://www.nytimes.com/2002/03/09/books/connections-plagiarism-that-doesn-t-add-up.html
http://www.ams.org/notices/200206/commentary.pdf
http://onlinelibrary.wiley.com/doi/10.1002/cplx.20050/pdf
Last year, budding science-writer phenom Jonah Lehrer was disgraced and humiliated when he was found to have plagiarized, and even fabricated, passages in some of his best-selling books. His actions (and Casti's) are an affront to the many high-quality science communicators who choose to write books the old-fashioned way… by actually generating their own words ;-) Casti's lapses may not be as grievous as Lehrer's, but it will now be difficult for me to read him with quite the same respect I did previously -- his repeated actions being such an unexpected/disheartening bit of chicanery to learn about.
Moreover, it harkened back to my prior MathTango post, wherein I wrote about learning that some observers suspect that Daniel Tammet (renowned autistic savant, and another person whose books I've enjoyed) may not be a genuine savant, but only a memory expert who uses his mental skills/tricks to portray himself as he does -- again, disappointing to learn that this is even a possibility. Sigh.... I consider myself a fairly critical reader, yet was caught unaware by these controversies around Casti and Tammet. (While I'm on this whole subject it may even be pertinent to note that the post prior to the Tammet post was a review of the new book, "Magnificent Mistakes In Mathematics," coincidentally, yet another focus on flaws-of-a-sort in science/math.)
On the Web, there are plenty of individuals, with or without some math credentials, who stake out math ideas that aren't credible (Mark Chu-Carroll of "Good Math, Bad Math" occasionally writes entertainingly about them, and their math "crackpottery"): http://scientopia.org/blogs/goodmath/?s=crankery ). But to discover shenanigans going on at a more professional level and/or in journals, is more disconcerting and unsettling… though I s'pose no field is immune from flim-flam artists of all sorts, math/science included.
Luckily, pure math really is probably more resistant to outright fraud, or even unchecked sloppiness, than most fields. In fact when I typed "math" plus "fraud" into Google most of the examples arising were, not too surprisingly, in regard to economics or finance. The other area that popped up, again not too surprising upon reflection, was statistics, which often gets used (though not necessarily deliberately) incorrectly to argue some point.
Anyway, I write all of this as a way of saying that real skepticism needs (unfortunately) to cut across all boundaries -- I'd dare say epidemiology can be critiqued almost as easily as astrology! Don't aim doubt and critical faculties at just the naive, the non-empirical, the 'low-hanging fruit'… but at the 'evidence-based,' the peer-reviewed, and occasionally even the mathematical as well. I applaud Oransky and others for bringing a critical eye to "the process of science" and trying to keep scientists not just skeptical, but honest as well. Science succeeds best through its vigilant, self-scrutinizing, self-correcting functions, which, for a variety of reasons, too often get left on the sidelines... occasionally even in mathematics.
In sum, Margaret Mead was famous for saying: "Never doubt that a small group of thoughtful, committed citizens can change the world; indeed, it's the only thing that ever has."
I'm tempted to parody that by saying, 'Never doubt that self-skepticism, close scrutiny, and doubt aren't key driving forces behind scientific progress… indeed, they always have been.'
Although I blush to write this, may I suggest as a site for "tough skepticism" mine?
ReplyDeleteFor example, this series on what relative risk really means and what it doesn't (often used in JAMA, etc. and by EPA and so on):
Selling Fear Is A Risky Business: Part I
http://wmbriggs.com/blog/?p=8988
Thanks for your excellent post. I agree there's a real need for more skeptics to turn their attention to dodgy science as opposed to the obvious pseudoscience like psychics and UFOs.
ReplyDeleteI'm one of the people who has looked skeptically at the Daniel Tammet case (a lot of the posts at the mnemotechnics site that you mention were mine). One thing that became clear to me when researching that was that there wasn't just a problem with that particular case - the entire field of research on savants was full of extraordinary claims made without solid evidence. And hardly anyone is challenging this flimsy science.
As an example, if you've seen any documentaries on savants, you've probably seen interviews with Darold Treffert, a researcher who is very widely quoted and often described as the world's leading expert on savants. He's often quoted by serious scientists and in serious psychology textbooks. But if you look into his work, you find that he has some views which are at best fringe and at worst outright pseudoscience. For example, he has often written uncritically about savants having psychic powers (see the section "Extraordinary telepathy as a savant skill" at this website here https://www.wisconsinmedicalsociety.org/professional/savant-syndrome/whats-new-2013/ ).
Other scientists have criticized him for these views, as you would expect. The trouble is that he only seems to come in for skeptical criticism for the most outrageous claims, like psychic powers. Slightly less implausible claims, like claims that some savants have a near perfect memory, are accepted without skepticism. They shouldn't be.
Tomas
Thanks so much for checking in here Tomas... I've followed Treffert's work off-and-on for decades, and yes, it is fascinating to get a different, eye-opening (and critical) take on it. For other readers, what is being referenced here is my prior post which focused on Daniel Tammet.
ReplyDeleteTo take a skeptical look at some more claims, I've seen at your other blog (Math Frolics) that you have covered the young "Prodigy" mathematician Jacob Barnett a couple of times. I think that is another good example of a real case that has been exaggerated by the media and deserves much more skepticism. Specifically:
ReplyDelete- It's repeatedly claimed in the media (and on your blog!) that he has "an IQ of 170". A careful reading of where this was first reported (the Indianapolis Star newspaper) and his mother's book ("The Spark") suggests that the test that he scored 170 on was not an IQ test - it was instead a single subtest (numerical operations) of a general test of academic ability (the Wechsler Fundamentals test). Basically, the score of 170 was on a test of basic math ability (addition, subtraction, multiplication etc) rather than an IQ test. His mother does not report his IQ score which is almost certainly lower, as she appears to be selectively quoting the single number of a single test that gives the highest number. I don't blame his mother for this - mothers usually like to talk up their children's achievements. But the media ought to be fact checking claims like this before reporting them.
- He has been shown on TV doing very impressive demonstrations of memory, where for example he rapidly memorises a list of 28 US states and rattles them off in forward and reverse order. The psychologist who tests him (Joanne Ruthsatz) interprets this as evidence of genius. Not mentioned is that normal children can learn to do impressive memory feats using standard techniques and training. Could Jacob have learned these techniques? Yes, because Jacob is a client of a local psychologist who his mother mentions in his book (Dr Carl S Hale) who specialises in teaching these techniques to children. Indeed he advertises on his website: "Using a technique called the Method of Loci, which dates back to the ancient Greeks, your student will gain confidence and learn ways to store and retrieve new information quickly... Mnemonic techniques are fairly easy to learn. They are used by individuals who perform amazing memory feats, such as remembering Pi to the one thousandth digit!... Give your child the advantage of an excellent memory."
- There are a number of wild claims in the media about him disproving Einstein, disproving the big bang, and being tipped as a future Nobel prize winner which frankly don't stand up to the slightest scrutiny. I think scientists and mathematicians could do more to take a skeptical look at these wild claims to keep media reporting sensible.
Jacob is undoubtedly an extremely good student with impressive achievements. But the media reporting of him concentrates on dubious evidence of inborn natural talent, rather than emphasising hard work and good quality tuition, which can give a more plausible explanation for his success.
I don't think these exaggerated media stories are harmless. I think they put off people from learning math because they mislead people into believing that innate talent is much more important for success in math than it really is. A closer look at the evidence suggests that hard work is more important, which is encouraging for any average person who is interested in math.
Jacob should be congratulated for his dedication and hard work, rather than paraded as a superhuman talent by the media. I wish him well.
Tomas
Actually, when I first heard about Barnett I did do some additional research on him and became convinced via statements of the university professors who'd worked with him (and now his work with the prestigious Perimeter Institute) that he was a true prodigy... and I don't believe his talents are merely the result of "dedication and hard work," but include a significant innate component as well, almost certainly related to his autism. I don't doubt that elements of his story are hyped (that's what the media always does), but I imagine that parts of Einstein's and Newton's life stories are hyped as well! For now, I believe the essential parts of Jacob's story are true as told, unless future disclosures indicate otherwise.
ReplyDeleteThe idea that I want to skeptically challenge is that it's easy to recognise innate talent. I don't think it is - I think it's exceptionally hard and requires a lot of evidence (just as it's exceptionally hard to demonstrate if a medicine works or not, which is why we have laborious double blind trials).
ReplyDeleteTry a thought experiment:
- Take a normal moderately capable mathematician (eg typical student at a good university)
- Consider how much total time spent during childhood practicing math, counting only the time that would be really effective. So, for example, time spent solving tough problems of a challenging difficulty would count; time spent bored in math class going over easy problems wouldn't. Given most students study a lot of other subjects, I doubt the number of hours on average throughout childhood and school is all that high - say 10 hours per week on average as a generous maximum.
- Given that, it wouldn't be that hard for another student of equal "talent" to do twice as much effective work - say 20 hours a week, if they were motivated. The extra 10 hours to find is far less than most kids spend watching TV. Other things being equal, the child spending twice as many hours learning math would be progressing roughly twice as fast.
- Would it be possible to accelerate progress further? Try improving the quality of tuition. For example, instead of a normal math class, substitute full time one on one tuition from a highly motivated and capable PhD level mathematician, qualified as a teacher. I think it's plausible that this would at least double the rate of learning. So now we have someone learning twice at twice the hourly rate for twice the hours as the normal mathematician - making four years progress every year.
Our hypothetical mathematician can now learn the same material most would take 20 years to learn in five years. Starting young (say age 3), he might easily be at university level by age 8. No exceptional talent is required.
The true example that I have in mind here is Fields medallist Terrence Tao - one of the best examples of a bona fide math prodigy, and a fitting candidate for the label of "genius". If you look at his biography, I think he fits the mould of our hypothetical mathematician quite well. He is reported to have spent a lot of time on math, so I think double the normal is reasonable. He also had the one on one tuition from a PhD mathematician - his mother, who homeschooled him. And he is reported to have reached university level mathematics at about age seven or eight, and started attending actual university courses at age 9.
I think that any mathematician encountering Tao at that age would assume, wrongly, that he must have immense innate talent. But our thought experiment suggests that he might just have high but not extraordinary talent (similar to the typical university student).
Being a distinguished math olympiad competitor at a young age, and a Fields medallist, he deserves the labels of "prodigy" and "genius" as much as anyone. He might, possibly, have some sort of innate natural talent. But I think it would be wrong to conclude that prodigies like him definitely must have innate talent, because other explanations are also possible.
The idea of innate talent is strongly embedded amongst mathematicians. I think it deserves to be looked at skeptically, because a closer look at the evidence suggests that the true variation in natural ability may not be all that great. And unfounded stories of great natural talent may put off people from math who believe, wrongly, that they lack a special innate skill necessary for success in math.
Tomas
The question remains, when an individual works hard and succeeds in a given area, how much of that hard work was driven by "innate" inclinations/talents in the area versus how much was purely hard work from some other motivation? Moreover, if you argue that individuals have comparable in-born talents but succeed by virtue of the work they put in, you may be labeling a lot of people as lazy when they fail to succeed in some area they attempt (I, for one, much prefer to think of my failure in advanced mathematics to be the result of innate deficiencies, and not poor work ethic on my part!). The whole nature-nurture interaction is way too complex to resolve here (and probably even varies from person-to-person), but I'm skeptical of views that swing too far in either direction.
ReplyDeleteTomas always makes a great argument, but the fact that Terry Tao has an autistic brother looks too much like a coincidence to my eye. Assuming the autism and the smarts are found together due to some common cause, what is that cause? Genetics or some thing that has happened in the family that can give rise to genius and also autism? I’d hate to think the latter. This article describes the Tao family in depth: http://www.theaustralian.com.au/news/features/beautiful-minds/story-e6frg8h6-1111114147837
ReplyDeleteMy position is that the argument about whether or not genius is inborn or is the product of effort is stupid from the start because the question is stupid. Of course genius is the result of hard work, despite any nonsense about unlearned knowledge that Dr Treffert might come out with. The real question is what is behind the hard work. Is it character and iron will? Is it a synergy between success due to innate ability and the rewards of success? Is it parents who drive and reward and threaten the young brainiacs? Is it something in the social background? A lack of other easier opportunities? Or is there some inborn factor that gives the child the motivation to learn? There are just so many anecdotes linking autism with intellectual giftedness, within individuals and families, and I can't criticise Treffert or anyone who studies savant-type achievement for looking at this relationship. I can criticise Treffert for doing little (as far as I can tell) to aid our understanding of how autism is linked with special ability.
"Anyway, I write all of this as a way of saying that real skepticism needs (unfortunately) to cut across all boundaries -- I'd dare say epidemiology can be critiqued almost as easily as astrology! Don't aim doubt and critical faculties at just the naive, the non-empirical, the 'low-hanging fruit'… but at the 'evidence-based,' the peer-reviewed, and occasionally even the mathematical as well."
ReplyDeleteYES! YES! YES! I couldn't agree more, but I'm afraid that if I go looking for folks who pick fault in what we call science I'll end up in bed with deniers and folks with sticky-out ears. Where do the sensible people hang out? I've been asking this my whole life. Are they the rationalists and atheists and skeptics? They are sensible, but so, so boring. Same s*** over and over. I'm so over these skeptical cliques and clubs (seems to be lots of educated white men) who will not hear or say a bad word said about scientists, but who seem to think they are very clever for pointing out flaws in the reasoning of astrologers and homeopathy practitioners. I think this might be what they call "shooting fish in a barrel". They need to get a life and get to grips with some real issues in which people suffer as the result of science that ain't science. In Australia one government after another spends a gazillion on mental health policies which have been heavily influenced by a psychiatrist who has repeatedly failed to declare conflicting interests and is regarded overseas as a renegade who advocates interventions that aren't supported by evidence. In my opinion this is more of a menace to the public at large than the astrology column in New Idea magazine.