Testing hypotheses via orthogonalization

Classical hypothesis testing frameworks break down in contemporary settings in which null hypotheses are increasingly abstract, the same data are used to both generate and test hypotheses, and minimal assumptions about the underlying data are made. In this work, we propose a new framework for conducting valid hypothesis tests in broad contexts. We propose to add and subtract external noise generated from a symmetric shift-family to our data, X, to partition it into two pieces, X1 and X2. We provide a generic strategy for orthogonalizing X2 against X1 under the null hypothesis H0, then show that testing whether the orthogonalization was successful provides a valid test of H0 under mild assumptions. Remarkably, this framework extends naturally to the post-selection inference setting with minimal modifications: we simply select a hypothesis on X1, then perform orthogonalization under the selected null. As our approach neither requires pre-specification of the selection mechanism, nor is it restricted to a small class of data-generating distributions, it dramatically expands the settings for which valid post-selection inference can be conducted. We showcase the flexibility of our proposal in four case studies: two involving challenging pre-specified null hypotheses and two involving post-selection inference scenarios.

This is joint work with Ameer Dharamshi (University of Washington)