Dear PLS experts,

I would very much like your opinions on a discussion below, in order to clarify some things about how bootstrap testing works in PLS.

I'm using PLS on ERP data. My understanding of the bootstrap procedure in PLS is that the standard errors of the electrode saliences are estimated through bootstrap sampling, using sampling with replacement and keeping the experimental conditions fixed for all observations, and the electrode saliences are then recalculated for each bootstrap sample. The ratio of the observed electrode salience to the bootstrap standard error provides a standardised measure of the reliability of the electrode salience, which is approximately equivalent to a z score (if the bootstrap distribution is normal).

 We had in a manuscript used an absolute bootstrap ratio value of 1.96 as a cut-off for determining whether the electrode salience was reliable, because values above 1.96 and below -1.96 are significant at the P< 0.05 level (in line with prior research, although some people use a higher cut-off, e.g. 3 for P < 0.001).

A reviewer of our manuscript suggested that we should refer to the ratio of 1.96 a corresponding to a 95% confidence interval, which emphasizes the issue of reliability instead of referring to significance values.

 Am i correct in thinking that what they mean by this is that if the bootstrap ratio is below -1.96 or above 1.96, the 95% CIs of the electrode salience does not encompass zero, so it is hence reliably different from zero with 95% confidence? Is this latter statement actually the case?

I asked these questions to a statistics expert in our department who responded with the following comments (I have shortened these a bit):

-----------------------------------------------------------------------
"In essence, the issue is quite a subtle one. Significance testing is based on the null hypothesis ie assume there is no effect and how big could the statistic be by chance. hence: "Oh look, z is >2; that's too rare to be chance" (accepting a 5% error rate in that statement). bootstrapping is not a null hypothesis procedure as you randomise with replacement: permutation tests are the resampling method appropriate to null hypothesis significance testing. So the reviewer didn't like your use of bootstrapping apparently to test significance. so what does the bootstrap do? well in this case it is providing confidence intervals for the electrode saliences -- and so asks are these electrode saliences reliable / reliably different from zero?

The weird thing in the PLS approach is that it doesn't use the bootstrap in a conventional way which probably antagonised/confused the reviewer a bit. normally, the bootstrap distribution would directly give one the 95% confidence intervals (ie take 10000 samples and compute the lowest 500 and highest 500 to get the cutoffs). In PLS the bootstrap seems to be being used as a method of estimating the standard error of the normal distribution numerically rather than using a theory to derive a formula for it. This is a bit odd wrt the standard bootstrapping approach.

I wonder if this odd usage of bootstrap in PLS is just a short-cut thing. To use the bootstrap conventionally it would be typical to take 10000 samples in order to get a proper distribution of the electrode saliences and so estimate the CIs. but with many electrodes to check that would take ages even on a fastish computer. If you were just assuming normality and using the bootstrap to calculate the standard error, you could probably get away with a smaller sized bootstrap sample. What do you/they use in PLS for this step?"

----------------------------------------------------------------------------

I also have two further questions:

First, whether an additional reason for reporting the bootstrap ratio rather than the CIs of the electrode saliences is that the electrode saliences are not standardised? Since they are dependent on the raw amplitudes of ERP data, directly reporting the CIs of the electrode saliences themselves might be more confusing, because they would be variable across time and location only on the basis of overall amplitude differences?

Second, whether there is any reason to believe that a bootstrap distribution of electrode saliences would be non-normally distributed under certain cases? And if so, when?

I hope my questions make sense, and thank you very much in advance,

Zara