In general the nil hypothesis is the null hypothesis. But the "absolutely no effect whatsoever" hypothesis will almost always be rejected so that a Type I error (falsely rejecting the null) cannot occur, but the probability of a Type II error (falsely not accepting the alternative) often is about 50%. Correcting for multiple testing decreases the power of the test even more. What are we testing? ...

Using test statistics without checking assumptions can give trouble. For instance: percentages follow a binomial distribution and differences between groups can be tested with a simple t-test only when the percentages are not too close to 0% or 100%. T-tests and ANOVA's should not be used on counts and log-tranforming these counts to correct skewed data creates it's own bias. More troubles ...

Statistical tests can be viewed as models. A two-sided Student's t-test for comparing the means of two groups can be formulated as a simple (not easy but with one independent variable) regression model. The test evaluates if the estimated parameter in the model is different from 0. More complex models are more difficult to interpret and assumptions harder to test. So much can go wrong ...

The term "p-value fallacy" covers a range of misinterpretations of the meaning of p-values. One common mistakes is that if the result is not statistically significant, the null-hypothesis is accepted and we "have to" conclude that there is no difference between samples. But we should never interpret large p-values as establishing that the null hypothesis is true. Interpretations can go off-base ...