commit 02e043e7c86d815899acf77cc56644a8db81c614
parent e7323f91ee6e7973b207accd011ea4d7e93c2d5b
Author: Silas Brack <s174433@student.dtu.dk>
Date: Thu, 9 Feb 2023 13:44:38 +0100
Small fixes in the abstract.
Diffstat:
1 file changed, 8 insertions(+), 7 deletions(-)
diff --git a/chapters/abstract.tex b/chapters/abstract.tex
@@ -4,21 +4,22 @@
% https://www.student.unsw.edu.au/writing-abstracts-honours-theses
Bayesian methods promise to provide a principled way to quantify uncertainty in neural networks
-This is important for many applications in machine learning, such as those involving safety-critical decisions which rely on risk assessment and interpretability.
+This is important for many applications in machine learning, such as those involving safety-critical decisions that rely on risk assessment and interpretability.
However, Bayesian inference is often computationally intractable, so approximate methods which compromise performance for practicality are used.
The Laplace approximation (LA) is one such method, though it itself typically relies on crude approximations to the posterior precision matrix to make it computationally feasible.
-When we do not factor the posterior precision into more practical approximations, the full posterior precision must be instantiated.
+When we do not factorise the posterior precision into more practical approximations, existing methods require that the full posterior precision be instantiated.
We call this the full Laplace approximation.
-For all but toy problems, since the full precision matrix scales quadratically with the number of parameters, the full posterior precision cannot be stored in memory.
+For all but toy problems, however, since the full precision matrix scales quadratically with the number of parameters, this matrix is too large to be stored in memory.
-In this work, we propose a method for computing the Laplace approximation using only Jacobian-vector products, which allows us to perform marginal training and posterior sampling using the full Laplace approximation without storing the entire posterior precision matrix.
+In this work, we propose a method for computing the Laplace approximation using only Jacobian-vector products.
+This allows us to perform marginal training and posterior sampling using the full Laplace approximation without storing the entire posterior precision matrix.
To accomplish this, we show that we can estimate the posterior precision's log-determinant and inverse square root using only Jacobian-vector products.
-To overcome the conditioning issues that arise during sampling, we propose several potential preconditioners which can be used to improve convergence of the inverse square root approximation.
-Next, we suggest a technique for evaluating the quality of the posterior samples without instantiating nor inverting the posterior precision based on results from traditional statistics.
+To overcome the conditioning issues that arise during sampling, we offer several potential preconditioners which can be used to improve convergence of the inverse square root approximation.
+Next, we give a technique for evaluating the quality of the posterior samples without instantiating nor inverting the posterior precision based on results from traditional statistics.
To implement these methods, we use JAX, a library for automatic differentiation which enables us to efficiently perform Jacobian-vector products without explicitly storing the entire posterior precision.
We perform the full Laplace approximation on both a sine function and MNIST using our method.
-This consists two steps, (a) training a neural network by maximising either its posterior probability or marginal likelihood and (b) sampling from this posterior.
+This consists of two steps, (a) training a neural network by maximising either its posterior probability or marginal likelihood and (b) sampling from this posterior.
Our approximate maximum marginal training procedure is able to learn a set of parameters which yields performance comparable to that of the maximum posterior and maximum likelihood training procedures.
By analysing quantile--quantile plots of our posterior samples and visualising these samples, we find that our approximate sampling method produces samples which are correctly distributed.
Our benchmarks of the performance of Jacobian-vector products estimate that our method yields a 10\,000x memory reduction over the conventional full Laplace approximation, since it does not store the quadratically-scaling full posterior precision matrix.
