Sunday, May 22, 2016

Do Your Own Research: The Scientific Research & Review Process

[Disclaimer: This is a personal post drawing upon my 8 years of experience in scientific research, at the graduate and post-doctoral level. None of these views reflect my employer (whoever that is), or my previous institutions. I have a Ph.D. in chemistry.]

Science literacy is one of my passions. I think that everyone should have the tools to do their own research (on beauty products, healthcare, environmental and food choices, everything), and that it's important to exercise the right to stay informed.

So, I decided to do an overview of the scientific research and review process. This is the international standard, and is followed in industry, academic, and government research. John Oliver did a really good segment on scientific studies that brings to light some of the strengths and weaknesses of the current system, and the way research is portrayed in media. I'm going to continue in that vein but also share some of the nuts and bolts of the process, and show you how science travels from bench to the public. It is lengthy, but I hope it helps you understand the restraints science is under, and how rigorous the process should be. Bookmark it, come back to it, whatever you want to do - and if you have any questions, please ask. I'm happy to talk for hours on this topic, and if I don't know the answer, I can hopefully at least point you in the right direction.

Essentially, as a scientist, I am trained to be skeptical and to evaluate problems based on evidence. When confronted with a problem or a claim, important considerations:

  • Sample size: how many people did we survey/test/examine? 5? 15? 1500? 150,000? This matters, both because it changes the actual statistics, and also, logically, what are the chances of something randomly happening in 5 people, vs. 150,000? Vs. 1.5 million people? 
  • Was the study conducted in humans?: This is more relevant for health-related issues, but to be clear, a mouse model is absolutely not the same as a human model. I'm not going to get into the ethics of animal studies in medicine, and I ask that you not leave flaming comments here about that. There are a host of issues involved: timeline, scale, human ethics, etc., but the bottom line is that humans and animal models are not the same. Showing that something worked in mice is not the same as showing it worked in humans. It is evidence that can help to support and sponsor a similar study in humans, but that study still needs to be done if you are going to draw the same conclusion in humans.
  • If the study was conducted on a sufficiently large number of humans, are those subjects representative of the population of interest?: I.e., if you are concerned about the effects in children, you don't test adults. The effects in men are not always the same in women. Age matters. Genetics matter.
  • Is there a control?: A lot of times, we are testing the effect of some stimulus or product or drug on people. Ok, so the product says that "women who used this product saw skin brightening in 2 weeks." Did they test a control - a group of people who didn't use the product, who were also evaluated for skin brightening? Unless they did, you can't be sure that the effect that was observed, was actually due to the product being tested.
  • Correlation is not causation: Along those lines, correlation (two or more things being related) is not causation. Just because people who tend to drink coffee also tend to run fast (made-up example!), does not mean that coffee makes you run fast. Maybe people who drink coffee are up earlier in the morning, and have more time to train. Or maybe they just have breakfast, as opposed to people who don't drink coffee and perhaps also don't have breakfast. 
Now that we are thinking about these things, let's go over the research pipeline. A principal investigator (boss dude/tte) has an idea - a hypothesis about why some observation is the way it is -  and has to ask the government or industry for money to fund that research. S/He does this by applying for grants, and writing out long proposals about what they want to do, and why, and here's some preliminary evidence that suggests this is a good idea, and here's a review of the current literature in the field to show that it hasn't been done before. The government or grant institution then has other scientists with expertise in that field review the grant, and make sure it's feasible and worthwhile.

If all goes well, the PI gets funded. They start up a lab, hire some peeps, and get the ball rolling. When they perform some experiments (following established safety and ethical rules), and get some results, that either support or don't (notice we're not "proving" anything) their hypothesis, they start getting ready to publish it. This is the crux of scientific research. You want to publish your results in international, well-regarded, subject-specific scientific journals, so that other people can read your results, and the science community learns something new. The journal process is something like this:
  • Pick a journal to submit to. There are "top-tier" journals that have high impact factors (a numerical index of the journal's influence) and good reputations, and these are what most people would love to shoot for. Of course, they are absolutely competitive, and your science has to be top-notch, new, and very useful. So, the majority of good work gets published in mid-range journals, that are still solid representations of well-conducted science, but not BOOM. Top journals include Nature, Science, Cell, The New England Journal of Medicine. You can google their impact factors to get a feel for the range.
  • Prepare your manuscript, include any authors who contributed meaningfully to the work, and then send it in. The first author is usually the person who did most of the work and wrote the paper. The corresponding author (usually the last one, but this is usually indicated) is the PI for the laboratory. Usually, the first author works as a post-doc or graduate student in the PI's lab.
  • The editor decides whether it should go out for review, and if yes, sends it to 3-5 other scientists in the same field of expertise. This is called the peer review process, and is the same as for grants. Reviewers read the manuscript, decide whether the science is sound, the experiments and methods are believable and reliable and well-documented (could this experiment be repeated?) and that the results are relevant and impactful. Sometimes they send it back for edits, or additional experiments. Sometimes they reject it. Sometimes they accept.
  • If you have to edit or add your paper, you do that. You send it back, it can go back and forth a few times, but finally, your paper gets published! 
  • Now it is available to the public, but really what this means, is that it is available to people or institutions with journal subscriptions. The landscape is changing and a lot of journals are becoming open access, and free to the public, but the majority of journals are still subscription only. If you want to look up a paper, I suggest going to the nearest university library.
What does this mean for you, an interested reader? Things to consider:
  • Who wrote the article? Are they well-respected, established? Do they have a webpage for their lab at their institution? What other papers have they published?
  • Is the journal reputable? What is their impact factor, how long have they been in existence, who are the editors? 
  • Do they give a detailed outline of their methods? This is generally a requirement of most scientific manuscripts, but the details can definitely range from sparse, to people including their data and codes.
  • Are there conflicts of interests? Journals require that these be declared. Who funded the project (A large soda company funded a very suspicious obesity project, for example), who do the authors work for, are they affiliated with any projects or institutions that could lead to a bias?
  • How old is the article? Have any new supporting or conflicting results been published?
Most citations that you see in science media (good science media, anyway) link back to journal articles - also called papers, manuscripts, technical literature. Going through them can be a nightmare (they are written for other scientists in the field, not the layperson), but they should contain everything: an abstract, an introduction that summarizes current literature in the field, and why this is a relevant result, the methods (how the experiment was performed, what tools and instruments and subjects/samples used, how often, etc.), and then the results and discussion. The formats differ from journal to journal, but that is the gist of it. Most research scientists publish. That's how their work is documented and shared with the world.

Edit (thanks to Amy's great comment below): This isn't to say that everything in science is perfect, not at all. This is just to give you an idea of the checks in place and the rigors of the scientific method. There are definitely problems with the current operation model, and many scientists don't agree with the current procedures. Peer review is also not infallible. Papers can be and are retracted after publication (though this is rarer and exceedingly embarrassing for all parties). But this is the model we are working with, and I just wanted to put this information out there for people who aren't familiar with it.

I apologize for the length, but I figured it was easier to get it out all at once, rather than segment it out into separate posts that would need to be cross-referenced, etc. You can tackle it in chunks, over tea. If you have any questions, please don't hesitate to ask. Larie, out.