conclusion.tex (5780B)
1 \setchapterpreamble[u]{\margintoc} 2 \chapter{Conclusion} 3 \labch{conclusion} 4 5 \section{Summary and Key Results} 6 7 In this project, we aimed to develop a scalable full Laplace approximation for Bayesian neural networks. 8 In doing so, we developed a novel method for approximately sampling from the full Laplace posterior and methods for preconditioning the Hessian of the negative log-posterior, which involved attempting to better understand the spectrum of the generalised Gauss-Newton Laplace posterior precision. 9 To test the quality of our sampling procedure, we showed how to evaluate the quality of the samples it produces while only accessing the samples themselves and the desired posterior precision, enabling us to choose a suitable preconditioner for it. 10 This evaluation method showed that, with an appropriate preconditioner, we can reliably produce samples which are Gaussian-distributed with the correct mean and covariance, as desired. 11 As such, we have shown that our approximate sampling method is a valid (albeit slow) method for sampling from the full Laplace posterior. 12 13 We also developed a novel method for training the full Laplace approximation by maximising the Laplace-approximate marginal likelihood of the data. 14 To compute this evidence term, we developed a method for computing an upper bound on the log-determinant of the Laplace posterior precision without storing the posterior precision itself. 15 This method allows us to compute a differentiable log-marginal likelihood bound, which we can train with standard gradient descent optimisation methods. 16 We showed that this method is a valid method for training the full Laplace approximation, and that it can be used to train the full Laplace approximation on a convolutional neural network. 17 However, we also found that the marginal training procedure to optimise the hyperparameters of the full Laplace approximation is not yet feasible, since the hyperparameters converge to unreasonable values. 18 19 Finally, we showed that it is possible to perform the full Laplace approximation with marginal likelihood training, and sample from its posterior. 20 We demonstrated on a simple sine wave regression problem and an MNIST classification problem that our method can be used to train the full Laplace approximation and yield predictive performance which rivals deterministic training. 21 However, we found that the sampling procedure is too slow to sample outside of a post-hoc setting, and may become a significant bottleneck for inference even under post-hoc Laplace on large-scale problems. 22 23 Overall, while we have developed novel methods for training and inference in the full Laplace approximation, we have yet to demonstrate its improved performance over the diagonal Laplace approximation nor its practical application to large-scale problems. 24 25 \section{Outlook and Future Developments}[Outlook and Future Work] 26 27 As mentioned in the previous section, we have not yet demonstrated the practicality of our method for full Laplace approximation, particularly for large-scale problems. 28 Hyperparameter selection typically requires performing a grid search over a range of hyperparameters, which is computationally expensive. 29 As such, the practicality of our method depends on the ability to perform marginal training to optimise hyperparameters via backpropagation. 30 Additionally, marginal training is prohibitively slow for training convolutions, and we have not yet found a way to speed it up. 31 Furthermore, estimation of the rank of the posterior precision is important to guarantee the closeness of the upper bound on the log-determinant term in the marginal, and this is still an open problem. 32 This is closely related to the fact that we do not have a solid understanding of how much the eigenvectors of the posterior precision matrix vary across different observations nor what affects the shape of the spectrum of this Hessian. 33 This point is also related to the structure of the Hessian in the Laplace approximation more generally. 34 As such, there is still much work to be done to understand the methods we have developed and to improve them. 35 36 Since we have not yet been able to train hyperparameters on the log-marginal likelihood, we have not yet been able to perform the full Laplace pipeline, which includes training the model parameters and hyperparameters and sampling from the posterior. 37 Because of this, we have not yet run experiments which attempt to demonstrate that the full Laplace approximation shows improved predictive performance or uncertainty calibration when compared to the diagonal or Kronecker-factored Laplace approximations. 38 Furthermore, the full Laplace approximation should be compared to other methods for Bayesian inference, such as variational approximations, deep ensembles, Monte Carlo dropout, and Markov chain Monte Carlo---both in terms of performance and runtime. 39 40 Furthermore, the feasibility of the full Laplace approximation must be demonstrated on a large-scale problem. 41 It is not clear how our method for the full Laplace approximation will scale with the number of parameters, data points, and model outputs. 42 This concerns not only the actual predictive performance and uncertainty estimation of the method, but also the post-hoc inference time for the approximate posterior. 43 As such, important experiments must be performed to determine the scalability of the full Laplace approximation. 44 This includes experiments on a large-scale problem, such as a large-scale image classification problem on a dataset with hundreds of thousands to millions of rows, with significantly higher resolution than the MNIST dataset, and more than 10 classes, such as in the case of zero-shot learning. 45 These experiments would thus also require models with many parameters, such as a ResNet.