commit 35cb369b3befee5f85202f97ab58ed19b1ff4ab7
parent e4c4b91d86ba0797578a3ef1faf5c6ec6ff04954
Author: Silas Brack <s174433@student.dtu.dk>
Date: Thu, 9 Feb 2023 23:53:35 +0100
Moreeeeee
Diffstat:
9 files changed, 29 insertions(+), 72 deletions(-)
diff --git a/chapters/ablation_experiments.tex b/chapters/ablation_experiments.tex
@@ -103,7 +103,6 @@ These experiments were run on a single NVIDIA A100 GPU.
\include{tables/mnist_ablation.tex}
\end{margintable}
-% RESULTS, DISCUSSION
Analysis of the p-values in \cref{tab:mnist-ablation} suggests that sampling is successful for \(Q \geq 10\) with a significance threshold of \(0.05\).
However, significance testing can be sensitive to the number of samples used, and, for large numbers of samples, these tests can excessively reject the null hypothesis.
In our case, the number of samples is generally fixed as a function of the number of parameters in the model and the amount of memory available.
diff --git a/chapters/abstract.tex b/chapters/abstract.tex
@@ -1,9 +1,7 @@
\chapter*{Abstract}
\addcontentsline{toc}{chapter}{Abstract}
-% https://www.student.unsw.edu.au/writing-abstracts-honours-theses
-
-Bayesian methods promise to provide a principled way to quantify uncertainty in neural networks
+Bayesian methods promise to provide a principled way to quantify uncertainty in neural networks.
This is important for many applications in machine learning, such as those involving safety-critical decisions that rely on risk assessment and interpretability.
However, Bayesian inference is often computationally intractable, so approximate methods which compromise performance for practicality are used.
The Laplace approximation (LA) is one such method, though it itself typically relies on crude approximations to the posterior precision matrix to make it computationally feasible.
@@ -19,7 +17,7 @@ Next, we give a technique for evaluating the quality of the posterior samples wi
To implement these methods, we use JAX, a library for automatic differentiation which enables us to efficiently perform Jacobian-vector products without explicitly storing the entire posterior precision.
We perform the full Laplace approximation on both a sine function and MNIST using our method.
-This consists of two steps, (a) training a neural network by maximising either its posterior probability or marginal likelihood and (b) sampling from this posterior.
+This consists of two steps: (a) training a neural network by maximising either its posterior probability or marginal likelihood and (b) sampling from this posterior.
Our approximate maximum marginal training procedure is able to learn a set of parameters which yields performance comparable to that of the maximum posterior and maximum likelihood training procedures.
By analysing quantile--quantile plots of our posterior samples and visualising these samples, we find that our approximate sampling method produces samples which are correctly distributed.
Our benchmarks of the performance of Jacobian-vector products estimate that our method yields a 10\,000x memory reduction over the conventional full Laplace approximation, since it does not store the quadratically-scaling full posterior precision matrix.
@@ -28,34 +26,24 @@ This is the first method for performing the Laplace approximation while only acc
\chapter*{Resum\'e}
\addcontentsline{toc}{chapter}{Resum\'e}
-Bayesianske metoder lover at give en principiel m{\aa}de at kvantificere usikkerheden i neurale netv{\ae}rk p{\aa} Dette er
-vigtigt for mange anvendelser inden for maskinindl{\ae}ring, f.eks. for sikkerhedskritiske beslutninger, der er afh{\ae}ngige af
-p{\aa} risikovurdering og fortolkelighed. Bayesiansk inferens er imidlertid ofte beregningsm{\ae}ssigt uh{\aa}ndterbar, s{\aa}
-anvendes der tiln{\ae}rmede metoder, som g{\aa}r p{\aa} kompromis med ydeevnen for at opn{\aa} praktiske fordele. Laplace-approximation
-(LA) er en s{\aa}dan metode, selv om den i sig selv typisk er baseret p{\aa} grove tiln{\ae}rmelser til den efterf{\o}lgende pr{\ae}cision
-matrix for at g{\o}re den beregningsm{\ae}ssigt gennemf{\o}rlig. N{\aa}r vi ikke faktoriserer den efterf{\o}lgende pr{\ae}cision i mere
-praktiske tiln{\ae}rmelser, kr{\ae}ver de eksisterende metoder, at den fulde posteriore pr{\ae}cision skal instantieres. Vi kalder
-dette den fulde Laplace-approximation. For alle problemer undtagen leget{\o}jsproblemer er det imidlertid s{\aa}dan, at den fulde pr{\ae}cisionsmatrix skalerer
-kvadratisk med antallet af parametre, er denne matrix for stor til at blive lagret i hukommelsen.
-
-I dette arbejde foresl{\aa}r vi en metode til beregning af Laplace-approximationen ved hj{\ae}lp af kun Jacobian-vektor
-produkter. Dette giver os mulighed for at udf{\o}re marginal tr{\ae}ning og posterior sampling ved hj{\ae}lp af den fulde Laplace
-approksimation uden at lagre hele den efterf{\o}lgende pr{\ae}cisionsmatrix. For at opn{\aa} dette viser vi, at vi kan
-estimere den posteriore pr{\ae}cisionens log-determinant og den inverse kvadratrod ved hj{\ae}lp af kun jacobian-vektorprodukter.
-For at overvinde de konditioneringsproblemer, der opst{\aa}r under pr{\o}vetagning, tilbyder vi flere potentielle pr{\ae}konditioneringssystemer
-som kan bruges til at forbedre konvergensen af den inverse kvadratrodsapproksimation. Dern{\ae}st giver vi en teknik
-til at evaluere kvaliteten af de efterf{\o}lgende stikpr{\o}ver uden at instantiere eller invertere den efterf{\o}lgende pr{\ae}cision
-baseret p{\aa} resultater fra traditionel statistik. Til at gennemf{\o}re disse metoder anvender vi JAX, et bibliotek til automatisk
-differentiering, som g{\o}r det muligt for os at udf{\o}re Jacobian-vektorprodukter effektivt uden eksplicit at lagre den
-hele den posteriore pr{\ae}cision.
-
-Vi udf{\o}rer den fulde Laplace-approximation p{\aa} b{\aa}de en sinusfunktion og MNIST ved hj{\ae}lp af vores metode. Denne
-best{\aa}r af to trin, (a) tr{\ae}ning af et neuralt netv{\ae}rk ved at maksimere enten dets posterior sandsynlighed eller marginale
-sandsynlighed og (b) udtagning af pr{\o}ver fra denne posterior. Vores tiln{\ae}rmede maksimale marginale tr{\ae}ningsprocedure er
-i stand til at l{\ae}re et s{\ae}t parametre, der giver en pr{\ae}station, som er sammenlignelig med den maksimale posterior- og
-maksimal sandsynlighed. Ved at analysere kvantil-kvantil-plots af vores posteriorpr{\o}ver
-og visualisering af disse pr{\o}ver finder vi, at vores tiln{\ae}rmede pr{\o}veudtagningsmetode giver pr{\o}ver, som
-er korrekt fordelt. Vores benchmarks af Jacobian-vektorprodukternes ydeevne vurderer, at vores
-metode giver en 10 000x hukommelsesreduktion i forhold til den konventionelle fulde Laplace-approximation, da den ikke
-ikke lagrer den kvadratisk skalerende matrix med fuld pr{\ae}cision for den efterf{\o}lgende tid. Dette er den f{\o}rste metode til at udf{\o}re den
-Laplace-approximation, samtidig med at der kun er implicit adgang til den efterf{\o}lgende pr{\ae}cision.
+Bayesianske metoder lover at give en principiel m{\aa}de at kvantificere usikkerheden i neurale netv{\ae}rk p{\aa}.
+Dette er vigtigt for mange anvendelser inden for maskinl{\ae}ring, f.eks. i forbindelse med sikkerhedskritiske beslutninger, der er afh{\ae}ngige af risikovurdering og fortolkningsmuligheder.
+Bayesiansk inferens er imidlertid ofte beregningsm{\ae}ssigt uh{\aa}ndterbar, s{\aa} der anvendes tiln{\ae}rmede metoder, som g{\aa}r p{\aa} kompromis med ydeevnen for at opn{\aa} praktisk anvendelighed.
+Laplace-approximationen (LA) er en s{\aa}dan metode, selv om den typisk er afh{\ae}ngig af grove tiln{\ae}rmelser af den efterfd{\o}lgende pr{\ae}cisionsmatrix for at gd{\o}re den beregningsm{\ae}ssigt gennemfd{\o}rlig.
+N{\aa}r vi ikke faktoriserer den efterfd{\o}lgende pr{\ae}cision i mere praktiske tiln{\ae}rmelser, kr{\ae}ver de eksisterende metoder, at den fulde efterfd{\o}lgende pr{\ae}cision skal instantieres.
+Vi kalder dette den fulde Laplace-approximation.
+Da den fulde pr{\ae}cisionsmatrix imidlertid for alle problemer undtagen legetd{\o}jsproblemer skalerer kvadratisk med antallet af parametre, er denne matrix for stor til at blive lagret i hukommelsen.
+
+I dette arbejde foresl{\aa}r vi en metode til beregning af Laplace-approksimationen ved hj{\ae}lp af Jacobian-vektorprodukter alene.
+Dette giver os mulighed for at udfd{\o}re marginal tr{\ae}ning og efterfd{\o}lgende stikprd{\o}veudtagning ved hj{\ae}lp af den fulde Laplace-approximation uden at lagre hele den efterfd{\o}lgende pr{\ae}cisionsmatrix.
+For at opn{\aa} dette viser vi, at vi kan estimere den posteriore pr{\ae}cisionens log-determinant og den inverse kvadratrod ved hj{\ae}lp af kun jacobianiske vektorprodukter.
+For at overvinde de konditioneringsproblemer, der opst{\aa}r under sampling, tilbyder vi flere potentielle pr{\ae}konditioneringsv{\ae}rktd{\o}jer, som kan anvendes til at forbedre konvergensen af den inverse kvadratrodsapproximation.
+Dern{\ae}st giver vi en teknik til at evaluere kvaliteten af de efterfd{\o}lgende stikprd{\o}ver uden at instantiere eller invertere den efterfd{\o}lgende pr{\ae}cision baseret p{\aa} resultater fra traditionel statistik.
+Til at gennemfd{\o}re disse metoder anvender vi JAX, et bibliotek til automatisk differentiering, som gd{\o}r det muligt at udfd{\o}re jacobian-vektorprodukter effektivt uden eksplicit lagring af hele den efterfd{\o}lgende pr{\ae}cision.
+
+Vi udfd{\o}rer den fulde Laplace-approximation p{\aa} b{\aa}de en sinusfunktion og MNIST ved hj{\ae}lp af vores metode.
+Denne best{\aa}r af to trin: (a) tr{\ae}ning af et neuralt netv{\ae}rk ved at maksimere enten dets posterior sandsynlighed eller marginale sandsynlighed og (b) prd{\o}veudtagning fra denne posterior.
+Vores tiln{\ae}rmede maksimale marginale tr{\ae}ningsprocedure er i stand til at l{\ae}re et s{\ae}t parametre, som giver en ydeevne, der er sammenlignelig med den maksimale posterior- og maksimale sandsynlighedstr{\ae}ningsprocedure.
+Ved at analysere quantile--quantile-plots af vores posteriorprd{\o}ver og visualisere disse prd{\o}ver finder vi, at vores tiln{\ae}rmede prd{\o}veudtagningsmetode giver prd{\o}ver, der er korrekt fordelt.
+Vores benchmarks af Jacobian-vektorprodukternes ydeevne ansl{\aa}r, at vores metode giver en 10\ 000x hukommelsesreduktion i forhold til den konventionelle fulde Laplace-approximation, da den ikke lagrer den kvadratisk skalerende matrix med fuld pr{\ae}cision for den efterfd{\o}lgende periode.
+Dette er den fd{\o}rste metode til at udfd{\o}re Laplace-approximationen, hvor der kun er implicit adgang til den efterfd{\o}lgende pr{\ae}cision.
diff --git a/chapters/conclusion.tex b/chapters/conclusion.tex
@@ -6,7 +6,7 @@
In this project, we aimed to develop a scalable full Laplace approximation for Bayesian neural networks.
In doing so, we developed a novel method for approximately sampling from the full Laplace posterior and methods for preconditioning the Hessian of the negative log-posterior, which involved attempting to better understand the spectrum of the generalised Gauss-Newton Laplace posterior precision.
-To test the quality of our sampling procedure, we showed how to evaluate the quality of the samples it produces while only accessing the samples themselves and the desired posterior precision, enabling us and to choose a suitable preconditioner for it.
+To test the quality of our sampling procedure, we showed how to evaluate the quality of the samples it produces while only accessing the samples themselves and the desired posterior precision, enabling us to choose a suitable preconditioner for it.
This evaluation method showed that, with an appropriate preconditioner, we can reliably produce samples which are Gaussian-distributed with the correct mean and covariance, as desired.
As such, we have shown that our approximate sampling method is a valid (albeit slow) method for sampling from the full Laplace posterior.
@@ -18,15 +18,12 @@ However, we also found that the marginal training procedure to optimise the hype
Finally, we showed that it is possible to perform the full Laplace approximation with marginal likelihood training, and sample from its posterior.
We demonstrated on a simple sine wave regression problem and an MNIST classification problem that our method can be used to train the full Laplace approximation and yield predictive performance which rivals deterministic training.
-However, we found that the sampling procedure is too slow too sample outside of a post-hoc setting, and may become a significant bottleneck for inference even under post-hoc Laplace on large-scale problems.
+However, we found that the sampling procedure is too slow to sample outside of a post-hoc setting, and may become a significant bottleneck for inference even under post-hoc Laplace on large-scale problems.
Overall, while we have developed novel methods for training and inference in the full Laplace approximation, we have yet to demonstrate its improved performance over the diagonal Laplace approximation nor its practical application to large-scale problems.
\section[Outlook and Future Developments]{Outlook and Future Developments}[Future Work]
-% OUR ISSUES
-
-% What were our limitations?
As mentioned in the previous section, we have not yet demonstrated the practicality of our method for full Laplace approximation, particularly for large-scale problems.
Hyperparameter selection typically requires performing a grid search over a range of hyperparameters, which is computationally expensive.
As such, the practicality of our method depends on the ability to perform marginal training to optimise hyperparameters via backpropagation.
@@ -36,18 +33,13 @@ This is closely related to the fact that we do not have a solid understanding of
This point is also related to the structure of the Hessian in the Laplace approximation more generally.
As such, there is still much work to be done to understand the methods we have developed and to improve them.
-% Also diagonal raw values maybe, compare the two?
-% Diagonal lets you know exactly which parameters contribute to the largest eigenvalues
Since we have not yet been able to train hyperparameters on the log-marginal likelihood, we have not yet been able to perform the full Laplace pipeline, which includes training the model parameters and hyperparameters and sampling from the posterior.
Because of this, we have not yet run experiments which attempt to demonstrate that the full Laplace approximation shows improved predictive performance or uncertainty calibration when compared to the diagonal or Kronecker-factored Laplace approximations.
Furthermore, the full Laplace approximation should be compared to other methods for Bayesian inference, such as variational approximations, deep ensembles, Monte Carlo dropout, and Markov chain Monte Carlo---both in terms of performance and runtime.
-% RELATED TO ACTUAL USEFULNESS OF THE METHOD
Furthermore, the feasibility of the full Laplace approximation must be demonstrated on a large-scale problem.
It is not clear how our method for the full Laplace approximation will scale with the number of parameters, data points, and model outputs.
This regards not only the actual predictive performance and uncertainty estimation of the method, but also the post-hoc inference time for the approximate posterior.
As such, important experiments must be performed to determine the scalability of the full Laplace approximation.
This includes experiments on a large-scale problem, such as a large-scale image classification problem on a dataset with hundreds of thousands to millions of rows, with significantly higher resolution than the MNIST dataset, and more than 10 classes, such as in the case of zero-shot learning.
These experiments would thus also require models with many parameters, such as a ResNet.
-
-% WHERE THE PROBLEM SHOULD BE EXPLORED NEXT. NOT SURE THERES MUCH FOR ME TO SAY
diff --git a/chapters/experiments.tex b/chapters/experiments.tex
@@ -92,20 +92,6 @@ We can see these results with \(\alpha = 0.1\) in \cref{fig:chisq-mnist}.
\cref{fig:chisq-mnist} shows that sampling succeeds for values of \(\alpha\) as low as 0.1, a value which is much smaller (and more ill-conditioned) than the value of \(\alpha\) we would likely use in practice.
As such, we can conclude that our approximate sampling method would be able to sample from the posterior distribution for a wide range of values of \(\alpha\) on MNIST.
-% We also show the spectrum of the posterior precision matrix for the MNIST problem in \cref{fig:spectrum-mnist} for a CNN with a \(3 \times 3\) filter and a \(5 \times 5\) filter.
-% The spectrum of the posterior precision matrix for the \(3 \times 3\) filter has two distinct clusters, with the first cluster being over an order of magnitude larger than the second.
-% The first cluster (which dominates the spectrum) consists of nine eigenvalues, and so the precision has an effective rank close to 9.
-% For the \(5 \times 5\) filter, we observe a hyper-exponential distribution of eigenvalues in this spectrum, with the spectrum appearing smooth (i.e., not having any distinct clusters).
-% This distribution of eigenvalues suggests that the effective rank of the precision is over 50, rather than 25 which would be plausible when considering that a \(3 \times 3\) filter has rank 9.
-% We do not yet understand this phenomenon and do not have a good explanation for why the posterior precision matrix for the \(5 \times 5\) filter has such a large effective rank and a smooth spectrum, unlike for the \(3 \times 3\) filter.
-
-% \begin{marginfigure}
-% \centering
-% \includegraphics{mnist_spectrum_3vs5.pdf}
-% \caption[Spectrum of the posterior precision on MNIST.]{Spectrum of the Laplace posterior precision on MNIST with a CNN. For the \(3 \times 3\) filter, there are two distinct clusters in the spectrum, with the first cluster being over an order of magnitude larger than the second. This first cluster consists of nine eigenvalues, and so the precision has an effective rank close to nine. For the \(5 \times 5\) filter, we observe a hyper-exponential distribution of eigenvalues in this spectrum, which does not have any distinct clusters. This suggests that the effective rank of the precision is over 50.}
-% \label{fig:spectrum-mnist}
-% \end{marginfigure}
-
\section{Discussion}
The choice of prior precision \(\alpha\) is non-trivial.
diff --git a/chapters/introduction.tex b/chapters/introduction.tex
@@ -4,18 +4,11 @@
\section{Bayesian Deep Learning}
-% Introduce Bayesian inference and its applications in machine learning.
-% General advantages Bayesian neural networks have over other traditional deterministic neural networks (and disadvantages).
-% Specific applications where these advantages are important and why they're important.
-% Specific applications of Laplace approximation in Bayesian neural networks.
-% Why this problem is important.
-
Bayesian methods allow for estimates of uncertainty which enable more efficient usage of data (e.g., via active learning) and avoid overfitting.
They accomplish this by using Bayes' theorem to model a distribution on the model parameters, which is then used to make predictions.
From here we can sample from this distribution to make predictions, thereby yielding uncertainty estimates, thus enabling the assessment of model predictive confidence.
In doing so, they allow for uncertainty estimates, which is important for many applications in machine learning, such as those involving safety-critical decisions which rely on understanding risk, where traditional deterministic neural networks are overconfident or fail to provide uncertainty estimates at all.
The estimation of model confidence is particularly important in domains where silent failures can have catastrophic consequences~\cite{jospin2022hands}, such as autonomous driving~\cite{rao2018deep}, medical diagnosis~\cite{ker2017deep}, and financial markets~\cite{cavalcante2016computational}.
-% The improved interpretability, however, is very valuable in the context of deep learning, where models are black-boxes which are difficult to interpret.
These methods have been successfully applied to a wide range of classical problems in statistics, though attempts to apply them to deep learning have had limited success.
For deep learning, the Bayesian approach is often limited by the computational cost of Bayesian inference.
This is because Bayesian inference requires computing the posterior distribution of the model parameters given the data, which, due to the nonlinear complexity of the neural network architecture, is often intractable.
diff --git a/chapters/laplace.tex b/chapters/laplace.tex
@@ -268,7 +268,7 @@ For mean square error (MSE) loss (e.g. for regression), we obtain the simple exp
\bm H_i \approx & {}\, (\bm y_i - f(\bm x_i, \bm \theta))^2 = 2 \identity,
\\ \bm \Lambda \approx&{}\, 2 \sum_{i=1}^N \bm J_i\T \bm J_i + \alpha \identity.
\end{align}
-If the Hessian of the loss function with respect to the model outputs \(\bm H_i\) is positive semidefinite, the GGN approximation is positive semidefinite, since each term in the sum \(\bm J_i\T \bm H_i \bm J_i\) is positive semidefinite.\sidenote{This is equivalent to the loss function being a convex function of the model parameters \(\bm \theta\).}
+If the Hessian of the loss function with respect to the model outputs \(\bm H_i\) is positive semidefinite, the GGN approximation is positive semidefinite, since each term in the sum \(\bm J_i\T \bm H_i \bm J_i\) is positive semidefinite.\sidenote{This is equivalent to the loss function being a convex function of the model outputs.}
Furthermore, this is the case for most common loss functions, such as MSE loss and cross-entropy loss.
For the case of the normal prior, the GGN approximation of the posterior precision will be positive definite, since the eigenvalues of \(\alpha \identity\) are all \(\alpha > 0\) and the eigenvalues of \(\sum_{i=1}^N \bm J_i\T \bm H_i \bm J_i\) are all greater than or equal to zero.
diff --git a/chapters/sampling.tex b/chapters/sampling.tex
@@ -182,7 +182,7 @@ The complete elliptic integral of the first kind is given by
\begin{align}
\mathcal{K}(p) = \int_0^{\pi/2} \left( 1 - m \sin(t)^2 \right)^{-1/2}\,dt,
\end{align}
-where \(p = 1 - m\), and is defined in the domain \(0 < p \leq 1\).\sidenote{If \(p > 1\), then what should be computed is instead the identity \(\mathcal{K}(p) = \mathcal{K}(1/p) / \sqrt{p}\).}
+where \(p = 1 - m\), and is defined in the domain \(0 < p \leq 1\).\sidenote{If \(p > 1\), then what we compute is instead the identity \(\mathcal{K}(p) = \mathcal{K}(1/p) / \sqrt{p}\).}
This integral is approximated by
\begin{align}
\mathcal{K}(p) \approx P(p) - \log(p) Q(p),
diff --git a/chapters/training.tex b/chapters/training.tex
@@ -186,7 +186,7 @@ For regression, the loss function which has a Bayesian interpretation as the neg
For negative log-posterior training, the same scaling can be applied with a Gaussian prior by dividing the prior precision by \(N\).
However, for the log-marginal likelihood, scaling the terms is no longer possible, since scaling the loss by a constant will not affect the gradients of the log-determinant term.
This means that it is important to compute the negative log-likelihood correctly as the SSE, since the gradients of the log-marginal likelihood are not invariant to scaling the log-marginal by a constant.
-To compute the posterior precision using a mini-batch, we thus have to scale the likelihood term by \(N / B\) to account for the fact that the likelihood is computed on a mini-batch of size \(B\) instead of the full dataset of size \(N\), yielding \(N / B \sum_{i=1}^B \bm J_i \bm H_i \bm J_i + \alpha \identity\).
+To compute the posterior precision using a mini-batch, we thus have to scale the likelihood term by \(N / B\) to account for the likelihood being computed on a mini-batch of size \(B\) instead of the full dataset of size \(N\), yielding \(N / B \sum_{i=1}^B \bm J_i \bm H_i \bm J_i + \alpha \identity\).
The log-determinant of this mini-batched precision is equivalent to
\begin{align}
\log\det \left( \frac{N}{B} \textstyle{\sum_{i=1}^B} \bm J_i \bm H_i \bm J_i + \alpha \identity \right)
diff --git a/main.tex b/main.tex
@@ -183,9 +183,8 @@
\dedication{
% \blackout
- % {It’s an incredibly exciting thing, this one, meaningless life of yours.}\\
{You don't have to have a dream.}\\
- \flushright -- Tim Minchin
+ \flushright---Tim Minchin
}
%----------------------------------------------------------------------------------------
