How to Tell a Good Scientist from a Bad Scientist

In a series of interviews with New York Times science writer Gary Taubes on scientificblogging, psychology professor Seth Roberts turns to the question of how do you go about making the judgment as to whether a scientist is trustworthy, especially when the topic is controversial. Taubes responds:

I’m a stickler about the use of words like “evidence” and “proof”. So if someone tells you there’s no evidence for some controversial belief, you can be fairly confident that they’re a bad scientist. There’s always evidence, or there wouldn’t be a controversy. If somebody says that “we proved that this was true” or “we set out to prove that this was true” that’s another bad sign. The point here, as [Karl] Popper noted, among others, is that you can never prove anything is true; you can only refute it. So researchers who talk about proving a hypothesis is true rather than testing it make me worried.

SETH: Yeah, I see what you’re saying. They overstate; they twist things around to make it come out the way they want. They are way too sure of what they…

TAUBES: Yes, and the really good scientists are the ones, almost by definition, who are most skeptical of evidence that seems to support their beliefs. They’re most aware of how they could have been fooled, how they could have screwed up, or how they might have missed artifacts in their experiment that could have explained what they observed. They’re very careful about what they say. If you ask them to do play devil’s advocate, and tell you how they could have screwed up, then at the very least, they’ll say “Well, if I knew how I could have done it, I would have checked it before I made the claim”. So when I’m talking about discerning the difference between a good scientist and a bad scientist, I’m talking about how they speak about their research, the evidence itself, it’s presence or absence.

Worth bearing in mind when you hear something which appears to overturn consensus expressed in strident terms: Where all the other possible explanations for the phenomenon considered? How did the researchers test their theory and data against the best possible countervailing research? Why do their conclusions offer better explanatory power?

12 Responses to How to Tell a Good Scientist from a Bad Scientist

  1. raincoaster says:

    Wise. Thank you for that.

  2. […] an article from TheStatsBlog that I found interesting. Here is what they say about the organization behind it. Since its […]

  3. yungchin says:

    Pfff, I disagree with Taubes. Sure, he’s right that people preferably shouldn’t say they proved stuff when they didn’t really – but that implies that all good scientists would bother about the philosophy behind that choice of words.

    Some don’t. Most haven’t read Popper. They may not be versed in philosophy of science at all. Does that invalidate their work?

  4. Hi Yungchin,

    I don’t think one needs to have absorbed Popper in order to have an intellectually scrupulous approach to one’s work: it’s more a matter of asking, have I honestly accounted for the strongest arguments against my hypothesis or theory? Can I address them with better data and explanatory power?

    I think the problem is when scientists start believing in their research at the expense of testing their fundamental premises.

    As for Popper, his theory of falsifiability is very useful in that it allows us a quick way of deciding what is scientific and what isn’t: if there is no possible way to disprove, say, certain tenets of Freudianism, then there is no possible way of proving them either. Popper’s real problem is in his logical absolutism on inference, which, ironically, makes him more of a mystic than a hard-nosed rationalist.

  5. yungchin says:

    Hi, thanks for your reply. I see what you mean, and agree. I guess what I dislike about the strong position taken above is just the choice of words: “bad scientist”. I’d rather say this would make an immature scientist.

  6. Bill in Vigo says:

    I’m afraid that some of those this might apply to are quite mature scientists. What bothers me most is the unavailability of raw data, unverified methodology, use of old out of date and sometimes disproved proxies. A good scientist would be searching for the truth and would appreciate learning of any error or incorrect data. That is how science is supposed to be self correcting.

    just my 2 cents


  7. Maia Szalavitz says:

    Yeah, I think Taubes is wrong here as well because there are many things where there really *is* no evidence. By that, scientists mean “no good evidence.”

    For example, the autism/MMR link: if someone said there was no evidence supporting it, they would be right! If someone said there’s no evidence supporting boot camps for kids, they would be right: all of the existing data does not show that they hold any advantage over alternatives. The only evidence favoring them– and creating ‘controversy’– is anecdote and commercial promotion. I think it’s fair for a scientist to not count that as evidence!

  8. kingsley says:

    i love the explanation and now i can tell a good scientist from a bad scientist

  9. Manyu says:

    Hey there,

    A good scientist is one who is a better philosopher, Science becomes interesting when you compare the fact you are studying with the normal/ behavioural science in humans and nature for example, the bond between covalent can be compared to the shared friendship among true friends, Besides a TRUE SCIENTIST IS one, who always tells, WE HAVE PROVED THIS, But this DOESNT mean that the other is wrong, To prove it to be Wrong you actually have to prove it.

  10. A bad scientist says:

    I am a industrial scientist formally trained in the field of optics. Think lasers, cameras, microscopes, medical devices, space telescopes (for keeping an eye on you the people from afar, as well as the aliens); these are the things I profess to know. Too often we industrial scientist get overlooked in these debates, but you may want to hear what I have to say, as it is industrial scientist that put the ceramics in your kitchen and the soap in your bathtub. I think the criteria is wrong for determining good vs bad. A scientist is an inquisitive person who seeks truth from nature, and even the most rogue of us (yes, I’m talking about the Teslas and Oppenheimers of our time) don’t qualify as “bad” by the criteria you have set out, even if they are mad and lead us into nuclear crises and create devastating projected energy weapons. Even a poorly trained scientist with no knack for keeping his foot out of his mouth will eventually find some objectively verifiable truth out there, which makes that person a success in my book!
    A bad scientist is one who uses experimental evidence and theoretical constructs to support things in this world we all know to be wrong: bad products, quack medicine, anti-anything campaigns that do not help the general populous have a better understanding of all risk factors (like putting cancer patients in jail for marijuana use). A bad scientist is an enabler. A bad scientist is anyone that merely takes orders, just like me, and likely >90% of the scientists out in American industry that have no leg to stand on against upper management. Look at the inferior stuff we make in this country. Our cars fall apart, our clothes are over priced and are generally made of cheap material, and our food is just terrible! I like my job, but I hate some of our products, and I see little need for them in the world. When the words “I think we should plan for this to break outside of warranty” come out of my manager’s and his manager’s mouths, I know what’s running the show, and it sure as hell isn’t good science. My superiors use predictions that I make based upon data that I take to form a decision that I know to be inherently predatory on the rest of humanity. The self-serving nature of corporations and governments to use good science for ill means drives me up the wall. The problem is that scientists are segregated from one another, and otherwise repressed and forced to fall in line lest we lose our jobs. I have a very sick spouse and need the benefits. Can you blame me for planning the demise of your digital camera outside of warranty? What about designing an inferior quality microscope for diagnosing the cancer in your throat from all those years of smoking? What about not telling you about the incredible waste of resources I encounter every time I interact with the DoD, because then I could go to jail? We need a union of ethical scientist in this country so that we may have a reporting system for the gross inconsistencies in how our industries behave. Most of all, the citizens of this country need to get it together and decide if we are all on the same team, because I am here to serve you all, and if you don’t ask me directly for quality products, you sure as hell won’t get it from my manager. All he cares about is money.

  11. A bad scientist says:

    By the way, people like my managers keep this kind of information from you…for years and years.

    • Poppers Advocate says:

      @A bad scientist, i think you missed the point of this post! First, you shouldn’t be mixing science with business! Science is purely finding the truth, there is no good or bad in it. The idea of using science to sell products is concerned with advertising and other business related disciples (areas that manipulate and twist knowledge), and not science! In fact, a good quote I read (just today) about science is:

      Gustave Le Bon:
      “Science has promised us truth. It has never promised us either peace or happiness.”

      I guess what you should take from this is that science can be used for good or evil. Yes, i can see how you can call someone a “bad scientist” in terms of how they use their data, so you do have a valid point. But i think that is missing the point of this article. I think they are trying to discern the difference between a good and bad scientist in terms of their ability as a scientist…

      For me, the only real measure in science is Empirical Observations. I think a bad scientist is someone who allows their own beliefs to run the show. They will twist the empirical evidence (often without even realising it!) through discounting some peoples experiments, and accepting poorly performed ones, to ensure their beliefs (and often their ego too) are maintained. Also, these beliefs are often made from what other people have told them!!

      I think the key to a good scientist is someone who firstly takes on board all of the empirical evidence. This is our only real glimpse at the truth – no theory or hypothesis can over rule that. Secondly, a good scientist will be able to discern between what in the empirical evidence is right, and what is wrong through experimental error, but only if there are any data that conflicts… Usually it is best to find a solution to the problem that accounts for all the evidence, as this is likely to be closer to the truth.

      Its a subtle difference, but it certainly exists. To me, it is basically letting go of your ego and your own beliefs, and being open to any possibility to explain your data. Being a neuroscientist in academia I see this all the time! I can see that Taubes has certainly implied these ideas in his answer, so I pretty much agree with him…

      Second point – i think anyone who is going to take science seriously, and really wants to be a good scientist, has to understand the philosophical principles on which it is operated… After all PhD stands for Doctor of PHILOSOPHY.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


Get every new post delivered to your Inbox.

Join 39 other followers