P-values and all that
The p-value is the probability of the test-statistic, not the probability that the null hypothesis is true. Probability(D|H) reads: the probability of the test statistics, given the hypothesis; research does not predict the happening of a hypothesis: Probability(H|D). In other words: the probability of Death by Hanging is not to be confused with the probability of suicide by Hanging as cause of Death. ...
Null- en Nil-Hypothesis
In general the nil hypothesis is the null hypothesis. But the "absolutely no effect whatsoever" hypothesis will almost always be rejected so that a Type I error (falsely rejecting the null) cannot occur, but the probability of a Type II error (falsely not accepting the alternative) often is about 50%. Correcting for multiple testing decreases the power of the test even more. What are we testing? ...
Troubles with test statistics
Using test statistics without checking assumptions can give trouble. For instance: percentages follow a binomial distribution and differences between groups can be tested with a simple t-test only when the percentages are not too close to 0% or 100%. T-tests and ANOVA's should not be used on counts and log-tranforming these counts to correct skewed data creates it's own bias. More troubles ...
Model misteries
Statistical tests can be viewed as models. A two-sided Student's t-test for comparing the means of two groups can be formulated as a simple (not easy but with one independent variable) regression model. The test evaluates if the estimated parameter in the model is different from 0. More complex models are more difficult to interpret and assumptions harder to test. So much can go wrong ...
No-effect Interpretation
The term "p-value fallacy" covers a range of misinterpretations of the meaning of p-values. One common mistakes is that if the result is not statistically significant, the null-hypothesis is accepted and we "have to" conclude that there is no difference between samples. But we should never interpret large p-values as establishing that the null hypothesis is true. Interpretations can go off-base ...
Classification logic
The classification table in logistic regression shows how many cases are correctly predicted (# cases are observed to be 0 and are correctly predicted to be 0; # cases observed to be 1 and correctly predicted to be 1), and how many cases are not correctly predicted (# cases observed to be 0 but are predicted to be 1; # cases observed to be 1 but are predicted to be 0). Or does it? ...