A spectral parameterisation of log-linear models

We have a new paper appearing in the forthcoming AISTATS2012 - you can get the paper here:

D. Buchmann, M.Schmidt, S. Mohamed, D. Poole, N. de Freitas. On Sparse, Spectral and Other Parameterizations of Binary Probabilistic Models. AISTATS, April 2012.

David has done some great work and the paper provides a nice new way of studying the natural statistics of binary data, in a similar way in which we study the natural statistics of other data, such as images. The paper shows a neat spectral representation of log-linear models and some useful results. It also provides a nice empirical argument for using lower order potentials in such models.

Here is the abstract:

This paper studies issues relating to the parameterization of probability distributions over binary data sets. Several such parameterizations of models for binary data are known, including the Ising, generalized Ising, canonical and full parameterizations. We also discuss a parameterization that we call the "spectral parameterization", which has received significantly less coverage in existing literature. We provide this parameterization with a spectral interpretation by casting log-linear models in terms of orthogonal Walsh- Hadamard harmonic expansions. Using various standard and group sparse regularizers for structural learning, we provide a comprehensive theoretical and empirical comparison of these parameterizations. We show that the spectral parameterization, along with the canonical, has the best performance and sparsity levels, while the spectral does not depend on any particular reference state. The spectral interpretation also provides a new starting point for analyzing the statistics of binary data sets; we measure the magnitude of higher order interactions in the underlying distributions for several data sets.