A note of caution when selecting ROIs from orthogonal contrasts

In neuroimaging, fMRI, EEG and MEG, it is quite common to circumvent the requirement to correct for multiple comparisons by reducing the multidimensional data to a single value per subject per condition by selecting a region/electrodes/time window of interest. Previously, I have shown that the results of statistical tests are biased if this is based on where any effect is maximal (http://www.ncbi.nlm.nih.gov/pubmed/23639379). However, it is common practise to select these region/electrodes/time window of interest from an orthogonal contrast to the one of interest in a factorial design. In other words, if we have a 2X2 design with factors A and B we might select our region/electrodes/time window of interest where the main effect has the highest statistical difference and then test whether the interaction is significant, averaging across this region/electrodes/time window of interest. However, is this biased? The answer is maybe.  What follows is covered in "Circular analysis in systems neuroscience: the dangers of double dipping" by Kriegeskorte et al. (2009) http://www.ncbi.nlm.nih.gov/pubmed/19396166.

To test this I simulated data from the 4 cells of a 2X2 factorial design, A1, A2 B1 and B2, such that the data for each trial and each subject and each time point was drawn from the same normally distributed random population. I then selected either the time point where either (i) the main effect of A vs B was maximal or (ii) randomly selected a time point. Then for this time point I tested whether the interaction between A and B was significant. This was repeated 500 times and I calculated the percentage of false positives. At a significance level of 0.05 we should expect false positives 5% of the time. This is precisely what was produced. Irrespective of how the time point was selected the percentage of false positives was 5% (see left hand panel of the figure below). Therefore, choosing a region/electrodes/time window of interest from an orthogonal contrast is not biased? Well the answer is yes but only if all the cells have the same variance. If I rerun the same simulation but now I reduce the variance of A1 by a factor of 10, keeping the means of all the cells the same, equal to zero, I get a very different result. Now the statistical test for the interaction is biased with over twice as many false positives as predicted (see right hand panel of figure below)

This simulation assumed that cell A1 had a different variance to cells A2, B1 and B2. If I now rerun the simulation but now assume that all cells have unequal variances by dividing the variance of A1 by 1, A2 by 2, B1 by 3 and B2 by 4 then the proportion of false positives rises above 50%.

.