Saturday, December 10, 2011

Real science vs. fake science: How can you tell them apart?


Phrenology is a famous pseudoscience that involved determining
a person's personality based on bumps on the skull.

Pseudoscience is the shaky foundation of practices--often medically related--that lack a basis in evidence. It's "fake" science dressed up, sometimes quite carefully, to look like the real thing. If you're alive, you've encountered it, whether it was the guy at the mall trying to sell you Power Balance bracelets, the shampoo commercial promising you that "amino acids" will make your hair shiny, or the peddlers of "natural remedies" or fad diet plans, who in a classic expansion of a basic tenet of advertising, make you think you have a problem so they can sell you something to solve it. 

Pseudosciences are usually pretty easily identified by their emphasis on confirmation over refutation, on physically impossible claims, and on terms charged with emotion or false "sciencey-ness," which is kind of like "truthiness" minus Stephen Colbert. Sometimes, what peddlers of pseudoscience say may have a kernel of real truth that makes it seem plausible. But even that kernel is typically at most a half truth, and often, it's that other half they're leaving out that makes what they're selling pointless and ineffectual.

If we could hand out cheat sheets for people of sound mind to use when considering a product, book, therapy, or remedy, the following would constitute the top-10 questions you should always ask yourself--and answer--before shelling out the benjamins for anything, whether it's anti-aging cream, a diet fad program, books purporting to tell you secrets your doctor won't, or jewelry items containing magnets:

1. What is the source? Is the person or entity making the claims someone with genuine expertise in what they're claiming? Are they hawking on behalf of someone else? Are they part of a distributed marketing scam? Do they use, for example, a Website or magazine or newspaper ad that's made to look sciencey or newsy when it's really one giant advertisement meant to make you think it's journalism?

2. What is the agenda? You must know this to consider any information in context. In a scientific paper, look at the funding sources. If you're reading a non-scientific anything, remain extremely skeptical. What does the person or entity making the claim get out of it? Does it look like they're telling you you have something wrong with you that you didn't even realize existed...and then offering to sell you something to fix it? I'm reminded of the douche solution commercials of my youth in which a young woman confides in her mother that sometimes, she "just doesn't feel fresh." Suddenly, millions of women watching that commercial were mentally analyzing their level of freshness "down there" and pondering whether or not to purchase Summer's Eve.

3. What kind of language does it use? Does it use emotion words or a lot of exclamation points or language that sounds highly technical (amino acids! enzymes! nucleic acids!) or jargon-y but that is really meaningless in the therapeutic or scientific sense? If you're not sure, take a term and google it, or ask a scientist--like one of us (seriously--feel free to ask). Sometimes, an amino acid is just an amino acid. Be on the lookout for sciencey-ness. As Albert Einstein once pointed out, if you can't explain something simply, you don't understand it well. If peddlers feel that they have to toss in a bunch of jargony science terms to make you think they're the real thing, they probably don't know what they're talking about, either.

4. Does it involve testimonials? If all the person or entity making the claims has to offer is testimonials without any real evidence of effectiveness or need, be very, very suspicious. Anyone--anyone--can write a testimonial and put it on a Website. Example: "I felt that I knew nothing about science until Double X Science came along! Now, my brain is packed with science facts, and I'm earning my PhD in aerospace engineering this year! If they could do it for me, Double X Science can do it for you, too! THANKS, DOUBLE X SCIENCE! --xoxo, Julie C., North Carolina"

5. Are there claims of exclusivity? People have been practicing science and medicine for thousands of years. Millions of people are currently doing it. Typically, new findings arise out of existing knowledge and involve the contributions of many, many people. It's quite rare--in fact, I can't think of an example--that a new therapy or intervention is something completely novel without a solid existing scientific background to explain how it works, or that only one person figures it out. Also, watch for words like "proprietary" and "secret." These terms signal that the intervention on offer has likely not been exposed to the light of scientific critique.

6. Is there mention of a conspiracy of any kind? Claims such as, "Doctors don't want you to know" or "the government has been hiding this information for years," are extremely dubious. Why wouldn't the millions of doctors in the world want you to know about something that might improve your health? Doctors aren't a monolithic entity in an enormous white coat making collective decisions about you any more than the government is some detached nonliving institution making robotic collective decisions. They're all individuals, and in general, they do want you to know. 

7. Does the claim involve multiple unassociated disorders? Does it involve assertions of widespread damage to many body systems (in the case of things like vaccines) or assertions of widespread therapeutic benefit to many body systems or a spectrum of unrelated disorders? Claims, for example, that a specific intervention will cure cancer, allergies, ADHD, and autism (and I am not making that up) are frankly irrational.

8. Is there a money trail? The least likely candidates to benefit from conclusions about any health issue or intervention are the researchers in the trenches working on the underpinnings of disease (genes, environmental triggers, etc.), doing the basic science. The likeliest candidates to benefit are those who (1) have something patentable on their hands; (2) market "cures" or "therapies"; (3) write books or give paid talks or "consult"; or (4) work as "consultants" who "cure." That's not to say that people who benefit fiscally from research or drug development aren't trustworthy. Should they do it for free? No. But it's always, always important to follow the money.

9. Were real scientific processes involved? Evidence-based interventions generally go through many steps of a scientific process before they come into common use. Going through these steps includes performing basic research using tests in cells and in animals, clinical research with patients/volunteers in several heavily regulated phases, peer-review at each step of the way, and a trail of published research papers. Is there evidence that the product or intervention on offer has been tested scientifically, with results published in scientific journals? Or is it just sciencey-ness espoused by people without benefit of expert review of any kind?

10. Is there expertise? Finally, no matter how much you dislike "experts" or disbelieve the "establishment," the fact remains that people who have an MD or a science PhD or both after their names have gone to school for 24 years or longer, receiving an in-depth, daily, hourly education in the issues they're discussing. It they're specialists in their fields, tack on about five more years. If they're researchers in their fields, tack on more. They're not universally blind or stupid or venal or uncaring or in it for the money; in fact, many of them are exactly the opposite. If they're doing research, usually they're not Rockefellers. Note that having "PhD" or even "MD" after a name or "Dr" before it doesn't automatically mean that the degree or the honorific relates to expertise in the subject at hand. I have a PhD in biology. If I wrote a book about chemical engineering and slapped the term PhD on there, that still doesn't make me an expert in chemical engineering. 

There is nothing wrong with healthy skepticism, but there is also nothing wrong in acknowledging that a little knowledge can be a very dangerous thing, that there are really people out there whose in-depth educations and experience better qualify them to address certain issues. However, caveat emptor, as always. Given that even MDs and PhDs can be disposed to acquisitiveness just like those snake-oil salesmen, never forget to look for the money. Always, always follow the money.

Emily Willingham, Double X Science Editor
@ejwillingham


Here is a handy short version, too!



ETA: I've also blogged about pseudoscience before here and here; the latter formed the basis for this post. There's also a much longer and very good primer on what the signs of a pseudoscience are here; it's now 10 years old, but it's all still applicable, which just goes to show that some things don't change. 

29 comments:

  1. Awesome! This post should be made mandatory reading for every science researcher and science educator.

    ReplyDelete
    Replies
    1. Real science researchers already know this. It's the non-science public who needs the educating.

      http://mindsmachinesmom.blogspot.com/

      Delete
  2. I agree with Suirauqa - mandatory reading - probably I'd start with the delegates at Durban's climate change meeting. #durban #climatechange

    ReplyDelete
  3. I appreciated this. I really wish I could have shared this with my family a few months ago, before they started getting conned...it's too late now, but thanks for working to spread the truth.

    ReplyDelete
  4. This is a great blog post because it will help readers to avoid getting caught up in endless arguments about pseudoscience with those that reject actual science. Pseudoscience proponents tend to regard anything other than their views as heresy, and they squabble quite a bit online and points made in this blog post help readers to avoid pointless and time-wasting situations.

    ReplyDelete
  5. Thanks for all the comments. @showjumper...you make a good point and remind me of another key feature of pseudoscience: the cult of personality trumps any need for evidence.

    ReplyDelete
  6. Great post! I think the point about testimonials is spot on. It caught my eye because I've researching nutrition advertising in the 1920s, and they use a lot of testimonials. And people wrote then about how you can't trust testimonials. Now, 90 years later, we still have the same problem.

    ReplyDelete
  7. Excellent post.A helpful beacon to use in the fog of the internet....

    ReplyDelete
  8. Nice job putting this together. Great insights. I think that I will share this with my high schools students.

    ReplyDelete
  9. Great post! The same principles could be applied to any parenting advice that begins with, "Studies have proven," or even "They've proven..." whoever "they" is. I had not idea how many parenting practices are backed by pseudoscience until I became a parent and started looking for answers.

    ReplyDelete
  10. #3 made me laugh because I've had a conversation recently with a family member about some current event (probably government or tax related) and we each were sharing our sources back and forth until I received a link to a site that had rotating, golden dollars with blinking stars on the background. At that point I ceased my argument - just didn't see any reason to continue.

    ReplyDelete
  11. You might be interested in Sense About Science, a charitable trust based in the UK that works to equip people to make sense of scientific and medical claims in public discussion, and its current ‘Ask for Evidence’ campaign. This is encouraging people to ask for the scientific evidence behind claims made in newspaper articles, ads or for products. It also has an annual Celebrity and science review, where it highlights and corrects misleading scientific claims made by celebrities.

    ReplyDelete
  12. Great post (typo in para 3 : 'jagrony '), have shamelessly plundered it (with attribution and links of course) for my post today.

    Well, I used it for a case study of a well-known cancer clinic :D

    Excellent checklist, have adopted it.

    ReplyDelete
  13. Thanks for all the comments and feedback. @Irene, thanks for those links. I will be reading through.

    @anarchic teapot--thanks for typo heads up. Always welcome. I'm pretty sure I know the clinic to which you refer. :)

    ReplyDelete
  14. Considering point 8: Most scams use separate websites for selling stuff and the medicinal claims. You won't find any absurd claims on the sellers site. There, you will typically find no other claims than it's an "all natural food supplement" that is "beneficial for your heath", along with a phrase like "use google, do your own research."

    Then you google, and you find a seemingly unrelated site telling you all the absurdities and that it even cures you from non-existant HIV.


    Also, I miss the term "natural" in your list. Quacks use the term as simplification of medicine. Natural is good, while chemical is bad. Well, peanuts are natural, yet they kill people. Amanita phalloides is an all natural mushroom. Tasty too. And it does not kill you if you get your liver transplant in time.
    And they consider a lot of treatments unnatural, while they in fact are as natural as it gets: Vaccines and penicillin can in fact be living. How is that not a natural cure?

    ReplyDelete
  15. Dave...all great points. I debated getting into specific terminology or situations but decided to keep it more general. But yes...terms like natural, toxic, "chemical free" (how?!?)...all serve as buzzwords and jargon to attract people seeking something that seems at first look to be more, well...natural.

    ReplyDelete
  16. Love the guide! We did something similar trying to squeeze it onto a flier. http://pythagoreancrank.com/?p=1783
    This is a great further reference, much appreciated!

    ReplyDelete
  17. Thanks for the post, Emily.

    My favorite book on this topic is written by a science educator and has gone mostly overlooked, called "Lies, Damned Lies and Science" by Sherry Seethaler (Yes, I did do a video review of it, along with Ben Goldacre's "Bad Science"). I do have a goal to make a video series based on this book. When a major science TV channel wants to avoid showing an important program that they helped to create because Americans are against climate change, we know that anti-science and pseudoscience has taken hold in the collective psyche.

    I'll see you in person at @scio12! I'll be giving a twitter shoutout for this blog, too! :)

    ReplyDelete
  18. great pints. the public needs to hear this more often.

    ReplyDelete
  19. Emily, it's a great great post.
    Would you mind if I translate your list and concepts (with links back here and all the deserved credits, of course) to portuguese and put it at my blog?
    It will be so useful for discussion and for raising awareness around here.

    ReplyDelete
  20. I do not mind at all, and thank you for asking.

    ReplyDelete
    Replies
    1. Great!
      Thanks for the attention! :)

      Delete
    2. Emily,
      thanks again, here is the post (in Portuguese):
      http://dopaperapauta.wordpress.com/2012/04/25/dez-mandamentos-para-nao-ser-enganado-por-pseudociencias/
      Hope someday we arrange an e-mail interview, could be nice. Your blog is one good example of how scientists can become more communicative and accessible to their audience - and IMHO that includes journalists like me.

      Delete
  21. Only one problem with the MD comment, MDs, even good ones, are subject to bounded rationality. As someone who was put on the maximum dosage of Zocor when "modest" CAD was discovered, and who later started having one of the "rare but significant" side effects that turned out to be not so rare, and led to the maximum dosage of Zocor being lowered. ... MDs aren't perfect, and are learning, and after their 24 years of schooling can be subject to fads (and they can be overly impressed with their 24 years of schooling).

    ReplyDelete
  22. That's not a problem with the MD comment. I don't think that anything there says that they're immune to fads or that they can foresee future outcomes. The point is, education does matter.

    ReplyDelete
  23. I'm late to the party, but had to write in and say I thought this was an outstanding piece. And much needed.

    ReplyDelete
  24. Even when it's obvious, though, it's surprising how many people not only can't tell the difference, they get it exactly backwards.

    (I suppose that's only to be expected, though, since fake science goes to great lengths to look like real science, only better; while real science just is what it is, warts and all.)

    ReplyDelete
  25. Scams based on physics almost always involve one of two claims. One is going faster than the speed of light; the other is a device or process that makes more energy than it uses (violation of the second law of thermodynamics). Google 'hydrino' for a good example - it's utter BS.

    ReplyDelete
  26. I would add, it is not because it is public money that it isn't biaised.

    ReplyDelete