The insignificance of statistical significance

M

Masticate3xAday

Registered Member
I see tons of bad literature. Readers and the general public SHOULD BE INFORMED on how to handle and interpret data. Too often I see biomedical articles that try to reach some sort of conclusion due to "statistical significance" based off of some sort of statistical test to calculate a "P" value. The only problem is that statistically significant results might not mean anything biologically or medically.

Readers should inform themselves briefly (very easy read) on this topic:

Br Med J (Clin Res Ed). 1986 Mar 15;292(6522):746-50.
Confidence intervals rather than P values: estimation rather than hypothesis testing.

Reader beware, researchers and institutions DO have agendas, and many times publish results that are nothing but hype, when in reality their statistically significant findings are absolutely worthless. Why is it that despite decades of effort, the research community STILL has not come up with a standardized way to publish and present data? Why is there soooooooo much over reliance on statistical significance and null hypothesis testing? It should absolutely be mandatory of all published science to say something about the actual magnitude of the effect that one is measuring.
 
I don't know you tell me? How does literature in those fields usually report a relationship between two variables? One must be very careful to separate magnitude of effect due to an experiment vs. statistical significance. You can make ANYTHING statistically significant provided you have a enough sample measurements since P values are very sensitive to the number of samples being tested. However, P values and all the stats tests that go into it say absolutely NOTHING about the magnitude of effect you're measuring. In other words, you can come up with an experiment that will measure statistically significant findings, findings which are absolutely irrelevant to the real world because the magnitude of the effect that you're measuring is very small and of no importance.
 
Generally one includes means and SEs in addition to P. That's a fair representation of difference - although strangely not all med journals do this, even today. Odd, no doubts there.

For me, the bigger issue is the abandonment of Bonferroni for a statistically workable universal establishment of permutational significance thresholds.
 
But what do standard errors of the mean even mean? SEs can be highly misleading and researchers often use them because they appear smaller if enough samples are taken. Many researchers, reviewers, and journal editors don't understand the fundamental concepts behind SEs, CIs, and SDs or error bars in general. SEM is not descriptive and should not be used at all to describe variability for a sample. Confidence intervals are far more valuable and convey more information that's often easier to interpret.

If one includes SE and P, I fail to see why they just don't simply provide CIs instead along with more important statistics for biologists (such as stats that indicate a biological effect is taking place, not just statistical significance) such as Cohen's d and r values. In a nutshell provide it all! CIs, statistical significance, and effect statistics. If one is running a common test like ANOVA, ANOVA should spit out an estimate of effect size anyway, so why not mandate reporting it?
 
Statistics helps make predictions, at the group level, but can be misleading when block predictions are applied at the individual. For example, there is a risk I can get struck by lightning. If I go my entire life and never get struck by lightning, was there ever really any risk to me, based on the summation of reality data?

Even though there was never any risk, based on my data, there will still be a tendency to force conformed behavior like there was constant risk to me. Everyone is expected to abide by the reading from the oracle. This can be good for sales.

I can say there is a risk you will run out of ice, someday, and not be able to make cocktails. The oracle has spoken. If we average this over the entire human population, I can use this to sell ice cubes to eskimos. In the hands of scientists, statistics can be helpful, but once this is used by politicians and salesmen, they can use these numbers to manipulate non casual individuals.

In NY City, the Mayor is banning large soft drinks beyond a certain size. Statistically, these larger drinks do have an impact on the weight gain of the average Joe. But we can also point out individuals, who either moderate larger drinks and/or don't have this effect (stay thin). Yet, they too will be forced to conform to the oracle, even though non casual. The scientists may see this but politicians can use this like any other lie.

The question is, if I am given a risk number, but I never develop the final condition, was there really any risk? How could I be assigned something that was never real to me? Is this science?
 
statistics is a valuable tool once its limitations are known.
one of those limitations is that statistics deals with probabilities.
another is correlating a third variable to 2 other unrelated variables.
 
Masticate3xAday said:
But what do standard errors of the mean even mean? SEs can be highly misleading and researchers often use them because they appear smaller if enough samples are taken. Many researchers, reviewers, and journal editors don't understand the fundamental concepts behind SEs, CIs, and SDs or error bars in general. SEM is not descriptive and should not be used at all to describe variability for a sample. Confidence intervals are far more valuable and convey more information that's often easier to interpret.

If one includes SE and P, I fail to see why they just don't simply provide CIs instead along with more important statistics for biologists (such as stats that indicate a biological effect is taking place, not just statistical significance) such as Cohen's d and r values. In a nutshell provide it all! CIs, statistical significance, and effect statistics. If one is running a common test like ANOVA, ANOVA should spit out an estimate of effect size anyway, so why not mandate reporting it?
Click to expand...

Well, I don't see a problem with that. I don't think the object is to necessarily become more conservative though, even at small effect size. You have to remember, there are any number of other effects in living systems that may introduce error; and effects in small samples are subject to Beavis effect error, true. I've seen it myself: a locus having major effects in a pair of half-sib families was significant but with smaller effect across an entire population. I don't think it makes the effect invalid. Numerous genetic systems are polygenic, and refuting small effects systems means that we move ourselves away from the proposition of neo-Darwinian microevolution.
 
I think OP is wrong and misunderstands statistics, P is an all-encompassing number that calculates exactly what the chances are that this occurred due to statistical abherration alone. Other numbers are used for more advanced information gleaning which would only be a distraction for a simple study testing a single hypothesis. In simple yes/no studies it would be literally impossible for any number to tell you anything more than P does, since P tells you exactly the percentage chance that it's a statistical anomaly, and you factly can not get any more information than that on how likely the study is to be true.

Masticate3xAday said:
You can make ANYTHING statistically significant provided you have a enough sample measurements since P values are very sensitive to the number of samples being tested. However, P values and all the stats tests that go into it say absolutely NOTHING about the magnitude of effect you're measuring. In other words, you can come up with an experiment that will measure statistically significant findings, findings which are absolutely irrelevant to the real world because the magnitude of the effect that you're measuring is very small and of no importance.
Click to expand...

This is all nonsense, obviously the magnitude of an effect is going to be something you register when you read a study.
 
typical animal said:
I think OP is wrong and misunderstands statistics, P is an all-encompassing number that calculates exactly what the chances are that this occurred due to statistical abherration alone. Other numbers are used for more advanced information gleaning which would only be a distraction for a simple study testing a single hypothesis. In simple yes/no studies it would be literally impossible for any number to tell you anything more than P does, since P tells you exactly the percentage chance that it's a statistical anomaly, and you factly can not get any more information than that on how likely the study is to be true. .
Click to expand...

But this is all relative to a null hypothesis that's assumed to be true, which is what most people ignore or forget. Assume theory A predicts that change in X causes Y. The researcher therefore manipulates X in an experiment and sees a change in Y. The stats all say this occured w/ P<0.05. Therefore the researcher concludes that theory A is supported. The researcher is in fact falling into a fallacy, which was noted by Aristotle. Theories B, C, D, E, F, G, ..... could all also predict that X changes Y (and they might all even do so better than A), therefore the conclusion that A is supported is rather weak. Most studies only pit the null hypothesis (which in reality in the vast majority of studies is going to be completely false anyway) vs the desired one (not all possible B, C, D, E etc.), P values only represent the probability of the data if the null hypothesis were true. The more interesting question, what's the probability of the hypothesis you're testing being true given the data is NOT answered. Effect size is much more important to a biologist I'd argue, since effect sizes are much more relevant to what a biogist is even trying to do, and effect sizes can be determined without relying on a null hypothesis that comes with all sorts of pitfalls for interpretation. Again, if you just simply make your sample sizes large enough, you can always obtain a non-zero statistically significant effect. It is entirely possible to have biologically important as well as clinically important significance, even if your data says that your findings are not statistically significant. Not sure why you believe there isn't more information out there than some arbitrarily chosen P value.


This is all nonsense, obviously the magnitude of an effect is going to be something you register when you read a study.
Click to expand...

Oh yeah? Browse through biological journal articles outside of clinical medicine and certain fields like ecology and tell me how many stats you see on effect size being listed. You can't always eyeball effect size and importance relative to a control or untreated sample easily.
 
Agreed. CIs may be more conservative, it is absolutley up to the researcher to determine whether a small effect size may be important.
 
One of the reasons we need statistics in biology is we largely ignore the impact of water. There are 100 times more water molecules in your body than all the other molecules combined. Yet the dominant molecular influence is largely ignored. The result is we need to use statistics to compensate for ignoring the water scaffolding, as we assume life is only the 1% organic.

For example, the DNA double helix contains a double helix of hydrogen bonded water within the major and minor grooves of the helix. While beta-DNA, which is the most common DNA configuration for bioactivity, has the highest degree of hydration. If you ignore the cause and effect of the water, you end up with is an incomplete picture of the DNA, that needs statistics.

Proteins fold into unique folds. This uniqueness of folding precludes statistics, since the folds are not based on average energy nor are they based on a random event, in spite of the small energy that holds unique folds together (equal to 1-2 hydrogen bonds worth of energy). The fold is based on a cause and effect within water. This is an example of water eliminating chaos. If we ignore the water we need to add chaos back thereby requiring statistics.

Life is composed of a symbiosis between water and organics. Nothing can move in the cell unless there is a displacement of water. Conceptually, one can simplify the cell by expressing the wide variety of organics (1%) in terms of a 99% aqueous reflection. This simplifies scale-up since water is one thing with only a handful of possible states that reflect the 1%.
 
You must log in or register to reply here.
Back
Top