commit 7344f0fca9dbbd40b47e97004b163ec3ceab9cdc
parent 45831861696087adb4c89c4e447cad87549cd062
Author: Silas Brack <s174433@student.dtu.dk>
Date: Thu, 2 Feb 2023 08:54:19 +0100
Not full rank.
Diffstat:
3 files changed, 17 insertions(+), 17 deletions(-)
diff --git a/chapters/conclusion.tex b/chapters/conclusion.tex
@@ -4,23 +4,23 @@
\section{Summary and Key Results}
-In this project, we aimed to develop a scalable full-rank Laplace approximation for Bayesian neural networks.
-In doing so, we developed a novel method for approximately sampling from the full-rank Laplace posterior and methods for preconditioning the Hessian of the negative log posterior, which involved attempting to better understand the spectrum of the Generalised Gauss-Newton Laplace posterior precision.
+In this project, we aimed to develop a scalable full Laplace approximation for Bayesian neural networks.
+In doing so, we developed a novel method for approximately sampling from the full Laplace posterior and methods for preconditioning the Hessian of the negative log posterior, which involved attempting to better understand the spectrum of the Generalised Gauss-Newton Laplace posterior precision.
To test the quality of our sampling procedure, we showed how to evaluate the quality of the samples it produces while only accessing the samples themselves and the desired posterior precision, enabling us and to choose a suitable preconditioner for it.
This evaluation method showed that, with an appropriate preconditioner, we can reliably produce samples which are Gaussian-distributed with the correct mean and covariance, as desired.
-As such, we have shown that our approximate sampling method is a valid (albeit slow) method for sampling from the full-rank Laplace posterior.
+As such, we have shown that our approximate sampling method is a valid (albeit slow) method for sampling from the full Laplace posterior.
-We also developed a novel method for training the full-rank Laplace approximation by maximising the Laplace-approximate marginal likelihood of the data.
+We also developed a novel method for training the full Laplace approximation by maximising the Laplace-approximate marginal likelihood of the data.
To compute this evidence term, we developed a method for computing an upper bound on the log determinant of the Laplace posterior precision without storing the posterior precision itself.
This method allows us to compute a differentiable log-marginal likelihood bound, which we can train with standard gradient descent optimisation methods.
-We showed that this method is a valid method for training the full-rank Laplace approximation, and that it can be used to train the full-rank Laplace approximation on a convolutional neural network.
-However, we also found that the marginal training procedure to optimise the hyperparameters of the full-rank Laplace approximation is not yet feasible, since the hyperparameters converge to unreasonable values.
+We showed that this method is a valid method for training the full Laplace approximation, and that it can be used to train the full Laplace approximation on a convolutional neural network.
+However, we also found that the marginal training procedure to optimise the hyperparameters of the full Laplace approximation is not yet feasible, since the hyperparameters converge to unreasonable values.
Finally, we showed that it is possible to perform the full Laplace approximation on a convolutional neural network, train its weights with marginal likelihood training, and sample from its posterior.
We demonstrated on a simple sine wave regression problem and an MNIST classification problem that our method can be used to train the full Laplace approximation and yield predictive performance which rivals deterministic training.
However, we found that the sampling procedure is too slow for online Laplace, and may become a significant bottleneck for inference using post-hoc Laplace on large-scale problems.
-Overall, while we have developed novel methods for training and inference in the full-rank Laplace approximation, we have not yet demonstrated its improved performance over the diagonal Laplace approximation nor its practical application to large-scale problems.
+Overall, while we have developed novel methods for training and inference in the full Laplace approximation, we have not yet demonstrated its improved performance over the diagonal Laplace approximation nor its practical application to large-scale problems.
\section[Outlook and Future Developments]{Outlook and Future Developments}[Future Work]
@@ -38,14 +38,14 @@ As such, there is still much work to be done to understand the methods we have d
% Also diagonal raw values maybe, compare the two?
% Diagonal lets you know exactly which parameters contribute to the largest eigenvalues
-We also have not shown that the full-rank Laplace approximation shows improved predictive performance or uncertainty calibration when compared to the diagonal or Kronecker-factored Laplace approximations.
-Furthermore, the full-rank Laplace approximation should be compared to other methods for Bayesian inference, such as variational approximations, deep ensembles, Monte Carlo dropout, and Markov chain Monte Carlo --- both in terms of performance and runtime.
+We also have not shown that the full Laplace approximation shows improved predictive performance or uncertainty calibration when compared to the diagonal or Kronecker-factored Laplace approximations.
+Furthermore, the full Laplace approximation should be compared to other methods for Bayesian inference, such as variational approximations, deep ensembles, Monte Carlo dropout, and Markov chain Monte Carlo --- both in terms of performance and runtime.
% RELATED TO ACTUAL USEFULNESS OF THE METHOD
-Furthermore, the feasibility of the full-rank Laplace approximation must be demonstrated on a large-scale problem.
+Furthermore, the feasibility of the full Laplace approximation must be demonstrated on a large-scale problem.
It is not clear how our method for the full Laplace approximation will scale with the number of parameters, data points, and model outputs.
This regards not only the actual predictive performance and uncertainty estimation of the method, but also the post-hoc inference time for the approximate posterior.
-As such, important experiments must be performed to determine the scalability of the full-rank Laplace approximation.
+As such, important experiments must be performed to determine the scalability of the full Laplace approximation.
This includes experiments on a large-scale problem, such as a large-scale image classification problem on a dataset with hundreds of thousands to millions of rows, with significantly higher resolution than the MNIST dataset, and more than 10 classes, such as in the case of zero-shot learning.
These experiments would thus also require models with many parameters, such as a ResNet.
diff --git a/chapters/introduction.tex b/chapters/introduction.tex
@@ -41,14 +41,14 @@ This last point is important, since it also means that the Laplace approximation
In the Laplace approximation, to compute the posterior distribution of the model parameters, we need to compute the Hessian of the loss with respect to the model parameters.
Typically, this is done by storing the Hessian matrix explicitly and computing the inverse of this matrix.
-Since the Hessian matrix is a \(D \times D\) square matrix with \(D^2\) elements, where \(D\) is the number of parameters, this is intractable for large models, which can contain millions or billions of parameters, thus limiting the applicability of full-rank Laplace outside of toy problems of a few thousand parameters.
+Since the Hessian matrix is a \(D \times D\) square matrix with \(D^2\) elements, where \(D\) is the number of parameters, this is intractable for large models, which can contain millions or billions of parameters, thus limiting the applicability of full Laplace outside of toy problems of a few thousand parameters.
To overcome this, we can use a crude approximation, such as only storing the Hessian matrix's diagonal~\cite{lecun1989optimal,denker1990transforming}, its Kronecker factorisation~\cite{heskes2000natural,martens2015optimizing,botev2017practical}, and other low-rank approximations.%~\cite{}.
Many of these Hessian approximations have successfully been applied to the Laplace approximation~\cite{ritter2018scalable,ritter2018online}, though are limited by the quality of the approximation.
-To overcome this, we propose a method which performs the full-rank Laplace approximation without requiring the Hessian to be stored explicitly.
+To overcome this, we propose a method which performs the full Laplace approximation without requiring the Hessian to be stored explicitly.
\section{Large-Scale Laplace}
-We are seeking an implementation of the full-rank Laplace approximation which doesn't require instantiation of the full posterior precision matrix.
+We are seeking an implementation of the full Laplace approximation which doesn't require instantiation of the full posterior precision matrix.
To perform inference using the Laplace approximation we need to sample from the posterior distribution, which is approximated by a normal distribution with a precision given as the Hessian of the loss with respect to the model parameters.
Sampling thus requires computing the covariance matrix as the inverse of the Hessian of the loss with respect to the model parameters, and then taking the square root of this covariance matrix.
However, for large models, which can contain millions or billions of parameters, we cannot instantiate the Hessian matrix, for reasons mentioned above.
@@ -72,7 +72,7 @@ JAX enables us to efficiently compute Hessian-vector products without explicitly
% \section{Research Objectives}
-We would thus like to develop a method which performs the full-rank Laplace approximation without requiring the Hessian to be stored explicitly.
+We would thus like to develop a method which performs the full Laplace approximation without requiring the Hessian to be stored explicitly.
To do this, we will use JAX and XLA to compute Hessian-vector products without explicitly storing the Hessian matrix.
We will then use these Hessian-vector products to compute the log determinant and inverse square root of the Hessian.
These quantities will be used to perform training and inference using the Laplace approximation.
diff --git a/chapters/laplace.tex b/chapters/laplace.tex
@@ -358,8 +358,8 @@ This is also possible with the Laplace approximation~\cite{eschenhagen2021mixtur
However, this is not a scalable solution, as it requires training multiple models.
Additionally, it is not clear whether there is a significant benefit to using a multimodal Laplace approximation over a single mode Laplace approximation (i.e., whether generalisation improves).
-Another limitation of the Laplace approximation is that it is not scalable to non-trivial problems for the full-rank Laplace approximation without performing crude approximations.
-This is because the full-rank Laplace approximation requires the computation of the posterior precision, which is a \(D \times D\) matrix, where \(D\) is the number of parameters.
+Another limitation of the Laplace approximation is that it is not scalable to non-trivial problems for the full Laplace approximation without performing crude approximations.
+This is because the full Laplace approximation requires the computation of the posterior precision, which is a \(D \times D\) matrix, where \(D\) is the number of parameters.
Thus, in order to scale to large models and large datasets without approximating the posterior precision with, e.g., a diagonal matrix, we need to find a method to perform the Laplace approximation without instantiating the posterior precision.
During training, we typically only need to train the model in a deterministic way (to find the MAP), and then we can use the Laplace approximation to sample from the approximate posterior post-hoc.
Thus, for inference, we need to determine a method for sampling from the Gaussian approximate posterior without explicitly computing the posterior precision.
