Data Distribution Dependent Priors for Stable Learning

PAC-Bayes is a state-of-the-art framework for analysing the generalisation of learning algorithms that incorporates the Bayesian approach, yet provides statistical learning bounds. One feature of the approach is that though the prior must be fixed, we do not need to have an explicit expression for it, only to be able to bound the distance between prior and posterior. Furthermore, the choice of prior only impacts the quality of the bound and not the validity of the results. We will discuss the implications of these observations describing ways in which the prior may be chosen to improve the quality of the bounds obtained. The application of these ideas to the stability analysis for SVMs delivers a significant tightening of the well-known stability bounds.