Autor: markyoung

~ 15/03/10

Hey You!

 

Yes…YOU!

 

Put up your hands and step away from the journals.

 

bookcops

 

It may be no surprise to anyone who frequents this blog that I am an information addict.  As such, I scour blogs all over the place to find useful thoughts and ideas relating to training, nutrition, and overall health.  However, over the past few months I’ve noticed something I’ve never really noticed before.  Everyone seems to be quoting studies regarding training, nutrition, and supplements!

 

Excellent…or maybe not?

 

As a complete science geek you might think I’d be happy about this, but truthfully, it all has me a little annoyed.  I can’t tell you how many times I’ve pulled the study mentioned in an article or blog post only to find that the study design was completely inappropriate or the statistics used by the authors were a hot mess.

 

While I appreciate the intentions of well meaning coaches and trainers, I’m concerned that the information being presented is often a little incomplete.  With free access to places like Pubmed finding scientific abstracts is easy.  But reading the actual studies and understanding them is a different matter.   

 

Here are a few things you should note when reading a journal:

 

1.  Impact

 

Impact is basically a measure of how frequently an average article in a journal is cited in a given period of time.  Higher impact journals tend to be the most reliable in terms of quality of research.

 

Journals like Nature, Science, and Cell are very high impact journals.  The journal of “How Much Hyooge Muscles I Can Get by Takin’ Dem Supplements” is probably not a journal I’d bother reading.

 

ajpem_cover_final

 

 

 

2.  Alpha

 

Although I’m simplifying a bit here, alpha is a statistical figure that scientists set to determine if their research is considered significant or not.  (Stats experts please don’t crucify me here.  I’m trying to simplify without losing the message.)

 

If alpha is set at 0.05 and the researcher confirms his or her hypothesis that training program X is better than training program Y, there is a 95% chance that this is true.  This is good news.

 

Unfortunately, there is also a 5% chance that this happened due to random influences or other sources of variation.  If this happens it is called a Type 1 error and journals tend to hate when this happens.  This is bad news.

 

If alpha is set lower (at 0.01), there is a 99% chance that the result is true and is less likely due to random influences.  If a study has reached significance at a lower alpha level you can be more sure that this type of error has been avoided.  Good news.

 

On the downside, because you’re being so careful to avoid saying something is true when it is not, you might miss a training effect that is actually there.  This is called a type 2 error.  This is bad news.

 

Generally speaking, journals tend to like a certain level to be met so the authors sometimes have no say where they set alpha.  Setting it in one place says a protocol is significant while setting it elsewhere might suggest it is not.

 

If a study says that a certain protocol wasn’t significant you sometimes have to look at alpha to see if it is set at 0.01 because they may have missed an effect that they would have deemed significant if they’d set it at 0.05.  Sometimes you’ll even see a researcher sneak in an alpha value of 0.06 just to make their results significant.

 

3.  Study Design

 

Generally speaking this is probably one of the most important things to consider and this is where I think many studies fall short.  Simple flaws in nutrition studies can often be as small as not getting subjects to record their food intake and relying on subject reporting that their intake has remained the same.

 

A bigger flaw (mentioned recently in Alan Aragon’s Research Review) was that a study examining the effects of branched chain amino acid supplementation used a control group who got 64 grams of protein for the day and a BCAA group who got 109 grams of protein for the day (and an additional 90 grams of carbs).  Obviously they found that the BCAA group was superior, but if you didn’t look closely you’d probably assume that this is because of the BCAAs instead of just total protein intake or carb intake.

 

Even things seemingly as trival as whether you use the same subjects for both protocols are important.  For example, if you were looking at the differences between one type of contraction or another on hypertrophy you might be inclined to assign one group to train with concentric contractions and another group to do purely eccentric contractions.

 

However, when you go to do your statistics (you’d probably run something called an ANOVA), you might miss something because there is usually more variability between two people than might be introduced by the training protocol.  Instead, your study would have more power to detect a difference in training type if you did one type of contraction with one arm and a different type with the other arm.

 

What’s that?  You think it would be hard to get trained people to train each arm differently?  You bet your ass it is!  And that is part of the reason why you don’t see a lot of studies being done on trained individuals.  Well…that and the fact that most training study participants are university students who are living on Kraft Dinner and beer because they’re the only ones who will let you do 16 muscle biopsies and train each arm differently for 12 weeks for $300.

kd 

 

 

There are so many possible issues here that I can’t even come close to touching on all of them, but I think you get the idea.

 

4.  Funding Source

 

Personally I think there is a lot less to make of this than most people suggest, but it is still worth mentioning.  In my experience, when a researcher is seeking private funding for a study they’ll design a study first, create a hypothesis AND THEN contact a company for sponsorship.

 

Obviously the researcher is going to reach out to a company who serves to benefit if the project proves correct (which it usually does if the researcher has done his or her reading of previous literature and knows what outcome is most likely).  Of course, a company is also only likely to sponsor a study it thinks will serve to benfit them.  Why on earth would they sponsor something that would disprove their product???

 

However, all if this has little to do with the study itself which is usually run by a grad student in pursuit of their Masters or Ph.D. and they have little dealing with the funding agency in the first place.  They are not “hired” by the agency to produce outcomes.

 

In the end, the researcher expects a certain outcome, they ask a company for money to demonstrate this, and then they publish it.  I’m sure there are some shady dealings out there, but I don’t think they’re as common as you might expect.

 

Summary

Utimately, I think what I’m trying to say is that unless you understand things like post-hoc analyses, calculation of statistical power, and the specific protocols used during the studies (and their assumptions) please leave the summarizing of studies to those who do.

 

Thoughts?  Comments?  Questions?  Please post them below.

Post tags:

10 Comments »

  1. Good stuff Mark!

    I like the review and they are all important things to take into account!

    I wish more negative studies were published though. It is just as important to know what does NOT work. I think they were going to actually start at Journal of Negative Results or something similar. Granted, if you are the top publisher in that journal you are committing academic suicide.

    While having a high rejection rate is ok, at times it gets to be crazy with the revisions you have to do. Many top journals pride themselves on their rejection rate too.

    True, training studies on athletes are very rare. If I was a college coach and a grad student asked me to study my athletes I would say hell no!

    The average college kid is much easier to get into a study, but not as easy as you may think. The studies I ran went great (thanks to my wonderful volunteers–seriously, they were great), but I have heard of horror stories. If half of your subjects drop out at the end of the study, you are screwed. Sometimes even just a few dropping out = massive headaches, esp if you have to go back and plead to the IRB why you need more people now.

    Funding is always a tricky one. Some times the supplement company will actually OWN the data and if you as a PI don’t state that no matter the results you are going to publish it; the odds of it seeing the light of day are slim.

    I agree that most grad student’s don’t give two stick’s shake about who funded it. Many times they don’t even control the study design.

    I opted to do a study in the lab with current equipment and then pay for the energy drinks out of my own pocket vs try to contact the energy drink companies. I wanted to make sure that it may cost me more money, but less hassle. Although, Red Bull has sponsored quite a few studies (I used Monster Energy drink though)

    Off my soapbox I go.
    Rock on
    Mike T Nelson PhD(c)
    http://ExtremeHumanPerformance.com

    Comment by Mike T Nelson — March 15, 2010 @ 12:40 PM

  2. Mark,

    How much time would you say you spend reading journals, and what type of process goes into what you decided to pick out and read in full (or is it whatever strikes your fancy in any given week?)

    As someone with a very modest scientific background, I often find myself stuck in between…………..I don’t want to have to read a bunch of studies just to determine if something is the best way to do things according to current knowledge, but on the other hand, I don’t want to be the guy who ignores science completely or merely skims a few abstracts or reads things in such a way as to get a very incomplete picture of what is going on.

    It would be interesting to see how people like you wade through what is out there. On the nutritional front, you have guys like Alan Aragon, Lyle McDonald, and Martin Berkhan who always seem to be on top of things, almost to the point of appearing to live on Pubmed (at least to a novice like me, that is), but they don’t strike me as the type who do nothing but whole up and read studies 23 hours out of the day, either. So it has always interested me to know how guys who stay very current with what is going on manage to find time to live life, work, have a bit of down time, and still stay up to date with research in a given area.

    Comment by Rhett Corley — March 15, 2010 @ 7:05 PM

  3. Wow. Loved reading this post.

    Too bad opinions being formed from abstracts are far too common. I know you both feel the same way as I do and would NEVER dare think about doing so. This is just something you do not do. PERIOD.

    I agree with you Mike that I, too, would like to see more negative, or lack of significance, studies published…come to think of it, my study actually trended this way (in large part due to the wrong subject population utilized).

    Overall, in my opinion, the biggest issue is lack of access to scholarly research…

    Anyway, good job boys.

    Comment by Jeff Cubos — March 15, 2010 @ 10:21 PM

  4. @ Mike – I totally agree. More publication of negative results would be a huge plus in my books too. Unfortunately, I don’t forsee this happening in the near future. Your comment about coaching a team and refusing to let them be used for research cracked me up…partially because it is so true.

    @ Rhett – When I was in grad school I had to read and review AT LEAST 10 journal articles per week. When you have to do this for a couple years you get pretty good at it. At first it took me hours to read a single study in detail, but over time I got much faster. It really is a skill to be learned. I probably put in at least 5 hours per month reading journals, but I can cover a lot of material in that time.

    These days journal articles just seem to appear in my inbox. Friends are always sending stuff my way that they think I’ll be interested in. I also subscribe to email alerts from many journals that send me the table of contents each time a new issue comes out and I can look into the articles that sound interesting. If I’m searching Pubmed I just tend to come up with a theme that I’m interested in for a while and focus on that. However, I rarely reach beyond my scope of understanding. If I don’t understand the methods that means I don’t understand the assumptions associated with those methods and I could easily make a mistake.

    Truthfully, I’ve given some thought to doing something like Alan Aragon’s Research Review, but specifically geared towards training for hypertrophy, performance, and strength, and rehab, but I’m not sure what the interest would be like.

    @ Jeff – Why do you think the information is not accessible? Anyone can usually get their hands on it at a university library can’t they?

    Comment by markyoung — March 23, 2010 @ 4:16 PM

  5. @mark

    Yes…if they live in large urban centers.

    I also cannot confirm this but I would hazard a guess that smaller universities without medical or rehabilitation programs may not carry the same subscriptions as Mac or U of T…

    Lastly, for those of us who work in the clinical setting Monday to Friday…AND…in a rural area, getting to a major university would be difficult to do. (This also applies to those in the gym setting, etc).

    Obviously we live in an electronic age and fortunately I have access via something that starts with “pass” and ends with “words”. I was lucky for the last 12 years in undergrad, chiro school, and grad school…now,…different story. Being a member to every single association and subscribing to all others just isn’t feasible.

    I have abstracts coming into my inbox all the time and I don’t mind searching for them. But on occasion I’ll hit a journal that I don’t have access to.

    Anyway…I may be completely wrong…

    Comment by Jeff Cubos — March 23, 2010 @ 6:11 PM

  6. @Mark I would be interested in your version of Alan Aragon’s Research Review. While you make really salient points about how hard it is to find studies on trained individuals, I’ve found it hard to find well-designed studies on athletic performance in general. Often it’s a matter of “creatine vs. non-creatine” study #308.

    Comment by Derek — March 24, 2010 @ 8:26 AM

  7. @ Derek – You’re bang on that they are significantly less common. The funding for athletic performance simply doesn’t match that for health/rehab. While government funding for performance research is less than optimal, whenever an independent sponsor (usually a large corporation) steps up to fund research it is usually viewed with skepticism because of the “conflict on interest”. Still, there is some good stuff out there if you are dedicated to finding it. If I continue to get good feedback about doing my own review I might just go that route.

    @ Jeff – I never really thought about that given that I’ve always had access to great libraries. Honestly, I think the prices per article are just plain ridiculous if you don’t have a subscription (which are always overpriced). If they reduced the cost I’ll bet that they’d sell a whole lot more individual papers.

    Comment by markyoung — March 24, 2010 @ 11:35 AM

  8. Mark – awesome blog post man. This stuff is right up my alley as i’ll be ending undergrad and begining grad school this coming fall. I wish you would talk more about research and its implications on performance training.

    One question i have is this. How/Who/What decides alpha? Why do the authors pick .05 verus .01? Do they even get a say or is the journal/publisher the one doing that? This is deep stuff man and i like hearing about it :)

    Comment by Sam Leahey — March 27, 2010 @ 4:56 PM

  9. Hey Sam,

    Journals usually have a requirement for alpha so this plays a big role. Also, where alpha is set depends on the consequences of getting a wrong answer. For example, if you’re testing a new drug you’d probably want to set alpha at 0.01 because if you found an effect you’d want to make sure that it wasn’t due to chance.

    On the other hand, with most research you don’t want to miss an effect if it is there so you can afford to set alpha a little higher. Generally speaking, if you set alpha above 0.05 it likely means that you weren’t able to see an effect so you’ve adjusted alpha afterwards so you can say that your results are statisitically significant. Most people “in the know” will pick up on this rather quickly though.

    Comment by markyoung — March 28, 2010 @ 8:35 AM

  10. [...] -  As engaging as Tony is, Mark Young may be the most pragmatic.  He always reads things with a critical mind, and I love that about him – it’s something I’m always working on myself.  In this piece, Mark discusses how you can better understand research articles. [...]

    Pingback by Random Friday 3-26-10 | Robertson Training Systems — September 11, 2010 @ 7:13 AM

RSS feed for comments on this post. TrackBack URL

Leave a comment