commit 6928956a1bbe0622e950d05dca05b411b3784443
parent bb9ebecedce12fe23a97324a09c0224fe92ebe21
Author: Silas Brack <s174433@student.dtu.dk>
Date: Thu, 26 Jan 2023 15:35:20 +0100
More goods.
Diffstat:
14 files changed, 181 insertions(+), 72 deletions(-)
diff --git a/chapters/abstract.tex b/chapters/abstract.tex
@@ -9,7 +9,11 @@
% What? What did the researcher find out? What were the key results?
% Why? Why are these results important? What is their significance?
-Introduce Bayesian inference and its use in machine learning.
+Introduce Bayesian inference and its applications in machine learning.
+General advantages Bayesian neural networks have over other traditional deterministic neural networks (and disadvantages).
+Specific applications where these advantages are important and why they're important.
+Specific applications of Laplace approximation in Bayesian neural networks.
+Why this problem is importnat.
Bayesian inference is used to ..., but requires an approximation to the posterior to make it computationally tractable.
The Laplace is one such approximation which assumes a normal distribution for the posterior and uses the Hessian to approximate the posterior precision.
diff --git a/chapters/conclusion.tex b/chapters/conclusion.tex
@@ -2,7 +2,7 @@
\chapter{Conclusion}
\labch{conclusion}
-\section{Summary and key results}
+\section{Summary and Key Results}
What results did we get? What did we learn? What did we propose?
Remember to actually include some results and a quantitative look at the results that were obtained.
@@ -11,9 +11,25 @@ How did we get these results? What methods did we use?
How did our findings contribute to the field? What changes?
-\section{Outlook and future developments}
+\section[Outlook and Future Developments]{Outlook and Future Developments}[Future Work]
+
+% OUR ISSUES
What were our limitations?
-Which avenues are most promising for future research?
-Either in order to solve the problem this paper tables or to solve the next problem.
+Fix marginal training, specifically hyperparameter optimisation
+Fix convolution performance in marginal
+Rank estimation
+
+Understand Hessian structure and spectrum
+Also diagonal raw values maybe, compare the two?
+Diagonal lets you know exactly which parameters contribute to the largest eigenvalues
+
+% RELATED TO ACTUAL USEFULNESS OF THE METHOD
+
+Perform a large-scale experiment.
+Propose one: resnet? dataset with millions of rows? (maybe not millions, but a lot)
+
+% WHERE THE PROBLEM SHOULD BE EXPLORED NEXT
+
+
diff --git a/chapters/experiments.tex b/chapters/experiments.tex
@@ -40,7 +40,7 @@ To compute the inverse for the efficient implicit HVP and GVP functions, we use
% RESULTS
-for values of 100k parameters or more, the manual computation failed, for both the inverse and the vector product (because it's the explicit \(N \times N\) matrix instantiation that fails)
+for values of 100k parameters or more, the manual computation failed, for both the inverse and the vector product (because the explicit \(N \times N\) matrix instantiation fails)
The largest number of parameters tested is 100M.
Even the implicit methods failed for 1B parameters, although this failure occurred in the instantiation of a single random vector of that size (and so would fail on this hardware for any method).
Notably, however, these implicit Hessian-vector product methods allowed for computation of matrix products and solves for problems up to 1000 times greater than the explicit method.
@@ -70,17 +70,44 @@ This is in line with the theory, as noted in \cref{sec:practical-ggn} (SHOULD IT
\section{Sampling Ablation}
+\begin{marginfigure}
+ \centering
+ \includegraphics{mnist_ablation.pdf}
+ \caption[Ablation over quadrature points for MNIST.]{Ablation over quadrature points for MNIST.}
+ \label{fig:mnist-ablation}
+\end{marginfigure}
+
Contour integral quadrature (CIQ) converges to the exact solution for \(\lim_{Q \to \infty} \bm s^{\mathrm{CIQ}}_{Q} = \bm K^{-1/2} \bm v\).
-However, CIQ is typically extremely close to the exact solution for low values of \(Q\).\sidenote{\textcite{pleiss2020fast} find that \(Q=20\) is approximately sufficient for most problems.}
+However, CIQ is typically close to the exact solution for low values of \(Q\).\sidenote{\textcite{pleiss2020fast} find that \(Q=20\) is approximately sufficient for most problems.}
We then investigated the number of quadrature points \(Q\) required to determine an adequate approximate solution for the Laplace approximation.
-Need to define a metric which determines if sampling was successful.
-Root-mean-square error of my percentiles vs the real chi-sq percentiles? Feels a bit weird.
-As we vary \(q \in [1, \ldots, 20]\), how does performance vary?
-What metrics can we use?
+We vary \(q \in \{1, \ldots, 20\}\) to determine the number of quadrature points required for the Laplace approximation.
+We then compute the Mahalanobis distance between the samples and the true posterior mean, and compare this to the Chi-squared distribution with \(D\) degrees of freedom.
+We then compute the p-value of the Kolmogorov-Smirnov test for each value of \(q\) by comparing the empirical CDF of the Mahalanobis distances to the theoretical \(\chi^2_D\) CDF.
+This p-value is then used to determine whether the null hypothesis that the samples are multivariate normal can be rejected based on some significance threshold \(\alpha\).
+Sample evaluation is explained in more detail in \cref{sec:sampling-evaluation}.
+The results of this experiment are shown in \cref{tab:mnist-ablation,fig:mnist-ablation}.
+\begin{margintable}
+ \centering
+ \caption[Ablation over quadrature points for MNIST.]{Sampling time and p-values for Kolmogorov-Smirnov with varying quadrature points \(q\) on MNIST with prior precision \(\alpha = 0.1\). Lower p-value indicates a higher probability of rejecting normality.}
+ \label{tab:mnist-ablation}
+ \include{tables/mnist_ablation.tex}
+\end{margintable}
+
+% RESULTS
+
+Based on \cref{fig:mnist-ablation}, sampling appears to be successful (based on a significance threshold of \(\alpha = 0.05\)) for \(q \in \{10, \ldots, 20\}\).
+
+% DISCUSSION
+However, inspection of CHISQ PLOTS suggests that there is not a significant difference between these values and \(q = 5\).
+In general, significance testing can be sensitive to the number of samples used, and, for large numbers of samples, these tests can excessively reject the null hypothesis.
+However, the number of samples is generally fixed as a function of the number of parameters in the model and the amount of memory available.
+As such, it is important to select an appropriate significance threshold for the test based on the number of posterior samples that are required.
+For 200 posterior samples, a more appropriate significance threshold of \(\alpha = 0.01\) or lower would reject sampling for \(q \leq 3\), since the cases where sampling completely fails have p-values that are below single floating point precision.
+Furthermore, the plots in CHISQ PLOTS suggest that the performance of the sampling method is effectively equal for \(q \in \{5, \ldots, 20\}\).
\section[Sampling for the Sine Function]{Sampling for the Sine Function}[The Sine Function]
@@ -102,4 +129,9 @@ The choice of \(\alpha\) is a non-trivial one
\section[Sampling for MNIST]{Sampling for MNIST}[MNIST]
-
+\begin{marginfigure}
+ \centering
+ \includegraphics{eigenvalues_mnist_55k_0.1_old.pdf}
+ \caption[Spectrum of the posterior precision on MNIST.]{Hyperexponential distribution of eigenvalues of Laplace posterior precision on MNIST with a convolutional NN with 17k parameters and a 3 by 3 filter.}
+ \label{fig:spectrum-mnist}
+\end{marginfigure}
diff --git a/chapters/introduction.tex b/chapters/introduction.tex
@@ -11,6 +11,12 @@ These methods have been successfully applied to a wide range of classical proble
% are particularly useful in the context of deep learning,
% where the models are often black-boxes and the data is often scarce.
+Introduce Bayesian inference and its applications in machine learning.
+General advantages Bayesian neural networks have over other traditional deterministic neural networks (and disadvantages).
+Specific applications where these advantages are important and why they're important.
+Specific applications of Laplace approximation in Bayesian neural networks.
+Why this problem is importnat.
+
Typically, the Bayesian approach to deep learning is to use a prior distribution over the model parameters and then compute the posterior distribution of the model parameters given the data, where the posterior is maximised to obtain the optimal model parameters for the available data.
This posterior distribution can then be sampled from to make predictions.
The optimisation can, depending on the choices of likelihood and prior, often be done analytically or numerically.
@@ -28,20 +34,11 @@ Currently, approximate Bayesian methods are either expensive to compute (Markov
% Beyond these, there are also domain-specific models which aim to provide
As such, there is demand for a method which exhibits the same computational cost as the optimization algorithms for deterministic neural networks while providing accurate posterior approximations and working out-of-the-box for any given architecture.
-% Laplace approximations are simple, yet prin-
-% cipled, posterior approximations suitable for
-% Bayesian modeling. Sadly these approximations
-% rely on access to the Hessian matrix of the loss
-% with respect to the model parameters. For large
-% models, it is even intractable to store this matrix,
-% and crude approximations (e.g. only storing the
-% Hessian diagonal) are called upon. We investigate
-% storing the Hessian implicitly as means to avoid
-% the fuzz.
-
The Laplace approximation~\sidecite{daxberger2021laplace,laplace1774memoire} is a simple yet theoretically well-supported posterior approximation suitable for Bayesian modeling.
-In the Laplace approximation, to compute the posterior distribution of the model parameters, we need to compute the Hessian of the loss with respect to the model parameters.
+The Laplace approximation has been applied in, for example, the prediction of earthquake hypocenters~\sidecite{izzatullah2022laplace}.
+The primary advantages of the Laplace approximation are that it is simple to implement, effective at out-of-distribution detection~\sidecite{miani2022laplacian} and can be performed post-hoc on a pre-trained model.
+In the Laplace approximation, to compute the posterior distribution of the model parameters, we need to compute the Hessian of the loss with respect to the model parameters.
Typically, this is done by storing the Hessian matrix explicitly and computing the inverse of this matrix.
Since the Hessian matrix is a \(D \times D\) square matrix with \(D^2\) elements, where \(D\) is the number of parameters, this is intractable for large models, which can contain millions or billions of parameters, thus limiting the applicability of full-rank Laplace outside of toy problems.
To overcome this, we can use a crude approximation, such as only storing the Hessian matrix's diagonal~\sidecite{lecun1989optimal,denker1990transforming}, its Kronecker factorisation~\sidecite{heskes2000natural,martens2015optimizing,botev2017practical}, and other low-rank approximations.%~\sidecite{}.
@@ -66,11 +63,11 @@ Since, to sample from the Laplace posterior distribution we need to compute the
% For large models, it is even intractable to store this matrix, and crude approximations (e.g. only storing the Hessian diagonal) are called upon.
% We investigate storing the Hessian implicitly as means to avoid the fuzz.
-\section{JAX and XLA}
+% \section{JAX and XLA}
-JAX~\sidecite{jax2018github} is a Python library for GPU-accelerated automatic differentiation and compilation of computational graphs into efficient machine code.
-To perform this compilation, it uses XLA, a compiler for linear algebra~\sidecite{sabne2020xla}.
-JAX enables us to efficiently compute Hessian-vector products without explicitly storing the Hessian matrix by compiling the Hessian-vector product computation using XLA.
+JAX~\sidecite{jax2018github} is a Python library for GPU-accelerated automatic differentiation and compilation of computational graphs into efficient machine code using XLA~\sidecite{sabne2020xla}.
+% To perform this compilation, it uses XLA, a compiler for linear algebra~\sidecite{sabne2020xla}.
+JAX enables us to efficiently compute Hessian-vector products without explicitly storing the Hessian matrix by compiling the Hessian-vector product computation with XLA.
% JAX is designed to be a drop-in replacement for NumPy, and is compatible with most of the Python scientific stack. JAX is built on top of XLA, a domain-specific compiler for linear algebra. XLA is able to compile Python code into efficient machine code, and is used by Google to accelerate TensorFlow and PyTorch. JAX is able to compile Python code into efficient machine code, and is used by Google to accelerate TensorFlow and PyTorch.
% It is designed to be a drop-in replacement for NumPy, and is compatible with most of the Python scientific stack. JAX is built on top of XLA, a domain-specific compiler for linear algebra. XLA is able to compile Python code into efficient machine code, and is used by Google to accelerate TensorFlow and PyTorch. JAX is able to compile Python code into efficient machine code, and is used by Google to accelerate TensorFlow and PyTorch.
diff --git a/chapters/preface.tex b/chapters/preface.tex
@@ -17,6 +17,8 @@
This thesis is the product of a five-month-long span of work conducted in the Section for Cognitive Systems of DTU Compute, during which I helped develop some of the ideas in the ``Large-Scale Laplace'' project.
This project is, at the time of writing, still ongoing, and I hope that we will be able to publish the results of our work in the near future.
+ In this project, I developed and implemented the sampling procedure for the full Hessian Laplace approximation.
+ However, I also contributed to the derivation and implementation of the marginal training algorithm.
}
\begin{flushright}
diff --git a/chapters/sampling.tex b/chapters/sampling.tex
@@ -26,7 +26,15 @@ However, these methods are not suitable for large matrices, as they require the
There are also methods which can compute the inverse square root of a low-rank matrix that is constructed as the inner product of two rectangular matrices by performing the inverse square root on the outer product (which is low-dimensional for low-rank problems), such as the Woodbury matrix identity.
However, the precision matrix we are interested in is not itself low-rank but a sum of low-rank matrices, and thus this method is not suitable for our problem.
This is discussed in more detail in \cref{sec:laplace,sec:preconditioning}.
-As such, these are not adequate methods for computing the inverse square root using only matrix-vector products.
+% As such, these are not adequate methods for computing the inverse square root using only matrix-vector products.
+
+Furthermore, methods which do not require computation of the square root of the covariance matrix typically attempt to exploit properties such as a block structure or sparsity of the covariance matrix~\sidecite{vono2022high}.
+However, we do not have a good understanding of the structure of the Laplace covariance matrix, and thus these methods are not suitable for our problem.
+% Talk about what we do know about the structure? Wrt the sum of low-rank GGNs
+When sampling from high-dimensional Gaussian distributions, it is common to use Markov chain Monte Carlo (MCMC) sampling methods such as Gibbs sampling~\cite{vono2022high}.
+However, these methods are tricky to tune and tend to converge very slowly for high-dimensional problems.
+As such, we do not use these methods for sampling from the Laplace approximation, and instead sample using the reparametrisation trick.
+For this, we need a method to compute the inverse square root using only matrix-vector products.
\section[Contour Integral Quadrature]{Contour Integral Quadrature}[Contour Integral Quad]
@@ -73,7 +81,7 @@ it is then also trivial to compute the square root product by multiplying \(\bm
\begin{marginfigure}%[-7cm]
\centering
\input{figures/cauchy_integral.tex}
- \caption[Representation of integration circle for the Cauchy integral formula.]{Representation of integration circle \(\Gamma\) contained within the domain of \(f\) as defined according to the Cauchy integral formula.}
+ \caption[Integration circle for the Cauchy integral formula.]{Integration circle \(\Gamma\) contained within the domain of \(f\) as defined according to the Cauchy integral formula.}
\label{fig:cauchy-integral-formula}
\end{marginfigure}
@@ -388,12 +396,26 @@ It is worth noting the issues that arise from scaling the term in the \(\sum \bm
\end{enumerate}
\section{Sampling Evaluation}
+\label{sec:sampling-evaluation}
+
+Into blah blah blah blah
+Sampling is sensitive to the conditioning number of the posterior precision.
+As such, we need a way to evaluate the quality of the samples drawn from the approximate posterior to determine if sampling occurred correctly.
+This means evaluating whether the samples are drawn from a normal distribution with the correct mean and precision.
+
+What we would like to fundamentally test is whether the CIQ algorithm converged correctly and that the samples were thus correctly generated.
+However, for cases when sampling fails to converge, this is typically caused by ill-conditioning of the posterior precision matrix.
+Since we only have access to the precision matrix and computing the covariance involves inverting the precision matrix using, for example, the conjugate gradient method (which is itself sensitive to conditioning), we cannot directly evaluate the covariance of the samples.
+Instead, we require a method to evaluate the quality of the samples without access to the covariance matrix.
+\subsection[Mahalanobis Distance and the Chi-Square Distribution]{Mahalanobis Distance and the Chi-Square Distribution}[Mahalanobis Distance]
+
+As per the defition of the Chi-squared distribution, for \(D\)-dimensional standard normal samples \(\bm \varepsilon_0\), the sum of square deviations from the mean is distributed as a Chi-square distribution with \(D\) degrees of freedom, as per
\begin{align}
- \varepsilon_0\T \varepsilon_0 \sim \chi^2(D)
+ \bm \varepsilon_0\T \bm \varepsilon_0 \sim \chi^2(D)
\end{align}
-Samples \(\varepsilon\) which are normally distributed but do not have a mean of zero and a standard deviation of one can be normalised and their sum of square deviations be calculated by computing their Mahalanobis distances, which will be Chi-square distributed with \(D\) degrees of freedom, as per
+Samples \(\bm \varepsilon\) which are normally distributed but do not have a mean of zero and a standard deviation of one can be normalised and their sum of square deviations be calculated by computing their squared Mahalanobis distances, which will be Chi-square distributed with \(D\) degrees of freedom, as per
%
\begin{align}
\bm d ={} & \sqrt{(\bm \varepsilon - \bm \mu)\T \bm \Lambda (\bm \varepsilon - \bm \mu)}\label{eq:mahalanobis-distance}
@@ -405,6 +427,8 @@ For the Laplace approximation, this can easily be computed as $\bm \Lambda = - \
Since this precision matrix can be computed exactly, this calculation can be performed to evaluate the whether a set of samples \(\bm \varepsilon\) are drawn from a distribution \(\normal(\bm \mu, \bm \Sigma)\) while only accessing the precision matrix in the evaluation.
As such, this method can be used to evaluate whether the approximate normal samples obtained from CIQ are correctly distributed.
+\subsection{Kolmogorov-Smirnov}
+
\section{Limitations}
Scaling with data?
diff --git a/chapters/theory.tex b/chapters/theory.tex
@@ -1,5 +1,5 @@
\setchapterpreamble[u]{\margintoc}
-\chapter{Laplace}
+\chapter{The Laplace Approximation}
\labch{laplace}
% http://saul.cpsc.ucalgary.ca/pmwiki.php/GradTips/GradTipsLiteratureReview
@@ -266,6 +266,13 @@ The Levenberg-Marquardt heuristic~\sidecite{pascanu2013revisiting}
A GGN approximation is any approximation which fulfils \cref{eq:ggn-whole-hessian}.
\sidecite{kunstner2019limitations}
+\section{Large-Scale Laplace}
+
+
+\section{Limitations}
+
+A sentence or two on multi-modality.
+
\section[The Fisher Information Matrix]{The Fisher Information Matrix}[Fisher Information]
\label{sec:fisher-information}
% https://agustinus.kristia.de/techblog/2018/03/11/fisher-information/
@@ -365,7 +372,7 @@ Furthermore, the condition number of this matrix is given by the ratio of the la
% \E{}{\sqrt{X}} \approx \sqrt{\E{}{X}} \left( 1 - \frac{\Var{X}}{8 \E{}{X}^2} \right)
% \end{align}
-\section{Krylov Subspace Methods}
+\section[Krylov Subspace Methods]{Krylov Subspace Methods}[Krylov Methods]
A Krylov subspace of a square matrix \(\bm K \in \reals^{N \times N}\) and a vector \(\bm b \in \reals^{N}\) is a subspace spanned by the set of images of \(\bm b\) under the first \(j\) powers of \(\bm K\).
\begin{align}
diff --git a/chapters/training.tex b/chapters/training.tex
@@ -5,11 +5,6 @@
\section{Maximising the Evidence}
\label{sec:maximising-the-evidence}
-WHY DO WE WANT TO MAXIMISE THE EVIDENCE?
-(1). OVERFIT LESS (SHOULD FIND FLAT MINIMA)
-(2). OPTMISE HYPERPARAMETERS
-(3). PERFORM MODEL SELECTION
-
As seen in \cref{eq:bayes-theorem}, the posterior probability for a given model is equal to
\begin{align}
p(\bm \theta \given \bm x) = \frac{p(\bm x \given \bm \theta) p(\bm \theta)}{p(\bm x)},
@@ -24,12 +19,13 @@ For this reason, we typically approximate it using a variational approximation,
However, since we're integrating over the posterior, maximising the marginal likelihood will seek minima that are flat, since the posterior will be very broad in these regions.
This is a desirable property, since it means that the model will be less likely to overfit the data, both from a theoretical standpoint~\sidecite{hochreiter1997flat} and in practice~\sidecite{keskar2016large,jiang2019fantastic,maddox2019simple}.
+
Furthermore, \sidetextcite{fong2020marginal} show that maximising the marginal (and consequently the log-marginal) is equivalent to performing k-fold cross validation for all values \(k = 1, \ldots, \infty\) and choosing the model with the highest average posterior probability (across each of \(k\) folds and across all values of \(k\)).
% This is a very strong result, since it shows that the marginal likelihood is a very good proxy for the true cross-validation performance of a model. autogenerated
Because of this, it can be used for model selection, since it will choose the model that performs best on average across all permutations of the data.
-% Finally, the marginal likelihood can be used to optimise hyperparameters, since it can be used to compute the gradient of the log-marginal with respect to the hyperparameters.
-% This is useful, since it allows us to use gradient-based optimisation methods to find the hyperparameters that maximise the marginal likelihood.
+Finally, the marginal likelihood can be used to optimise hyperparameters, since it can be used to compute the gradient of the log-marginal with respect to the hyperparameters.
+This is useful, since it allows us to use gradient-based optimisation methods to find the hyperparameters that maximise the marginal likelihood.
For the Laplace approximation, we approximate the posterior as \(p(\bm \theta \given \bm x) \approx \normal(\bm \theta \given \bm \theta_{\textsc{map}}, \bm \Lambda^{-1})\), where the posterior precision \(\bm \Lambda = -\nabla^2_{\bm\theta} \mathcal L(\bm \theta_{\textsc{map}})\) is the Hessian of the negative log-posterior at the maximum a posteriori (MAP) estimate \(\bm \theta_{\textsc{map}}\).
We thus obtain the approximate Laplace posterior
@@ -63,28 +59,32 @@ and
\end{align}
where \(\alpha\) is the precision of the prior and \(\rho\) is the precision of the likelihood.
-Combining \cref{eq:laplace-log-marginal,eq:normal-likelihood,eq:normal-prior}, we obtain
+Combining \cref{eq:laplace-log-marginal,eq:normal-likelihood,eq:normal-prior}, we expand the Laplace approximation to the log-marginal likelihood as
\begin{align}\label{eq:expanded-log-marginal}
\log p(\bm x) \stackrel{\textsc{la}}{\approx}{} & - \frac{N O}{2} \log(2 \pi) + \frac{N O}{2} \log \rho - \frac{1}{2} \rho \sum_{i=1}^N \norm{\bm y_i - f_{\bm \theta}(\bm x_i)}^2
\\ & - \frac{D}{2} \log(2 \pi) + \frac{D}{2} \log \alpha - \frac{1}{2} \alpha \norm{\bm \theta}^2 \nonumber
- \\ & + \frac{D}{2} \log (2 \pi) - \frac{1}{2} \log \det \bm \Lambda \nonumber
-\end{align}
-
-The \texttt{laplace} Python library provides a function to compute the log-marginal likelihood of the Laplace approximation. The function is called \texttt{log\_marginal\_likelihood} and is given by
-\begin{align}
- \log p(\bm x) \mathrel{\stackrel{\textsc{la}}{\approx}}{} & \frac{N O}{2} \log \rho - \frac{1}{2} \rho \sum_{i=1}^N \norm{\bm y_i - f_{\bm \theta}(\bm x_i)}^2
- \\ & - \frac{1}{2} \left( \log \det \bm \Lambda - D \log \alpha + \alpha \norm{\bm \theta}^2 \right). \nonumber
+ \\ & + \frac{D}{2} \log (2 \pi) - \frac{1}{2} \log \det \bm \Lambda, \nonumber
\end{align}
-As compared to the exact log-marginal from \cref{eq:expanded-log-marginal}, this implementation is obtained by omitting the constant terms which do not depend on \(\bm \theta\), \(\alpha\), or \(\rho\).
-However, in doing so, the authors ignore the terms \(D / 2 \log(2 \pi)\)
-While these terms do not affect the optimisation problem, they are useful for comparing models with different numbers of parameters.
-
-\begin{tabular}{ll}
- Minimise & Maximise
- \\\addlinespace 1. \(\rho / 2 \sum_{i=1}^N \norm{\bm y_i - f_{\bm \theta}(\bm x_i)}^2\) & 1. \(N O / 2 \log \rho\)
- \\ 2. \(1 / 2 \log \det \bm \Lambda\) & 2. \(D \log \alpha\)
- \\ 3. \(\alpha / 2 \norm{\bm \theta}^2\)
-\end{tabular}
+where it can be seen that there are constant terms do not depend on \(\bm \theta\), \(\alpha\), or \(\rho\).
+While these terms are not necessary for the optimisation problem, we choose to include them.
+In particular, the terms \(D / 2 \log(2 \pi)\) are important for comparing models with different numbers of parameters.\sidenote{We note that these constant terms are not included in the computation of the \texttt{log\_marginal\_likelihood} function from the \texttt{laplace} Python library~\cite{daxberger2021laplace}. In spite of this, \textcite{immer2021scalable} use this library in their experiments, where they compare models with different numbers of parameters.}
+Furthermore, the other terms are important for interpreting the log-marginal as a likelihood.
+
+% \begin{tabular}{ll}
+% Minimise & Maximise
+% \\\addlinespace 1. \(\rho / 2 \sum_{i=1}^N \norm{\bm y_i - f_{\bm \theta}(\bm x_i)}^2\) & 1. \(N O / 2 \log \rho\)
+% \\ 2. \(1 / 2 \log \det \bm \Lambda\) & 2. \(D \log \alpha\)
+% \\ 3. \(\alpha / 2 \norm{\bm \theta}^2\)
+% \end{tabular}
+
+% The \texttt{laplace} Python library provides a function to compute the log-marginal likelihood of the Laplace approximation. The function is called \texttt{log\_marginal\_likelihood} and is given by
+% \begin{align}
+% \log p(\bm x) \mathrel{\stackrel{\textsc{la}}{\approx}}{} & \frac{N O}{2} \log \rho - \frac{1}{2} \rho \sum_{i=1}^N \norm{\bm y_i - f_{\bm \theta}(\bm x_i)}^2
+% \\ & - \frac{1}{2} \left( \log \det \bm \Lambda - D \log \alpha + \alpha \norm{\bm \theta}^2 \right). \nonumber
+% \end{align}
+% As compared to the exact log-marginal from \cref{eq:expanded-log-marginal}, this implementation is obtained by omitting the constant terms which do not depend on \(\bm \theta\), \(\alpha\), or \(\rho\).
+% However, in doing so, the authors ignore the terms \(D / 2 \log(2 \pi)\)
+% While these terms do not affect the optimisation problem, they are useful for comparing models with different numbers of parameters.
\section[The Determinant Lower Bound]{The Determinant Lower Bound}[The Lower Bound]
@@ -151,3 +151,9 @@ Furthermore, the trace of
Estimating the trace and square trace of the Hessian matrix is the most computationally expensive part of estimating the log-marginal under the Laplace approximation, since this involves computing the GGN-vector product.
+\subsection{Limitations}
+
+Training hyperparameters
+Speed for convolutions
+Scalability to huge networks hasn't been tested
+Rank estimation
diff --git a/figures/eigenvalues_mnist_55k_0.1.pdf b/figures/eigenvalues_mnist_55k_0.1.pdf
Binary files differ.
diff --git a/figures/eigenvalues_mnist_55k_0.1_old.pdf b/figures/eigenvalues_mnist_55k_0.1_old.pdf
Binary files differ.
diff --git a/figures/mnist_ablation.pdf b/figures/mnist_ablation.pdf
Binary files differ.
diff --git a/glossary.tex b/glossary.tex
@@ -10,6 +10,14 @@
name={GVP},
description={is the specific memory-efficient JAX implementation of the Generalised-Gauss-Newton-vector product}
}
+\newglossaryentry{pdf}{
+ name={PDF},
+ description={is the probability density function of a distribution}
+}
+\newglossaryentry{cdf}{
+ name={CDF},
+ description={is the cumulative density function of a distribution}
+}
% Glossary entries (used in text with e.g. \acrfull{fpsLabel} or \acrshort{fpsLabel})
% \newacronym[longplural={Frames per Second}]{fpsLabel}{FPS}{Frame per Second}
diff --git a/main.tex b/main.tex
@@ -101,7 +101,7 @@
\subject{Masters of Science in Engineering}
\title{Effortless Bayesian Deep Learning}
-\subtitle{Tapping into the Potential of Modern Optimizers}
+\subtitle{Tapping Into the Potential of Modern Optimizers}
\author[Silas Brack]{Silas Brack}
@@ -250,18 +250,6 @@
\input{chapters/conclusion.tex}
%----------------------------------------------------------------------------------------
-% APPENDIX
-%----------------------------------------------------------------------------------------
-
-\appendix % From here onwards, chapters are numbered with letters, as is the appendix convention
-
-\pagelayout{wide} % No margins
-\addpart{Appendix}
-\pagelayout{margin} % Restore margins
-
-\input{chapters/appendix.tex}
-
-%----------------------------------------------------------------------------------------
\backmatter % Denotes the end of the main document content
\setchapterstyle{plain} % Output plain chapters from this point onwards
@@ -275,6 +263,18 @@
\defbibnote{bibnote}{Here are the references in citation order.\par\bigskip} % Prepend this text to the bibliography
\printbibliography[heading=bibintoc, title=Bibliography, prenote=bibnote] % Add the bibliography heading to the ToC, set the title of the bibliography and output the bibliography note
+%----------------------------------------------------------------------------------------
+% APPENDIX
+%----------------------------------------------------------------------------------------
+
+\appendix % From here onwards, chapters are numbered with letters, as is the appendix convention
+
+\pagelayout{wide} % No margins
+\addpart{Appendix}
+\pagelayout{margin} % Restore margins
+
+\input{chapters/appendix.tex}
+
% %----------------------------------------------------------------------------------------
% % NOMENCLATURE
% %----------------------------------------------------------------------------------------
diff --git a/tables/mnist_ablation.tex b/tables/mnist_ablation.tex
@@ -0,0 +1,13 @@
+\begin{tabular}{rrr}
+ \toprule
+ \(q\) & p-value & Time [m] \\
+ \midrule
+ 1 & 0.0000 & 10.6 \\
+ 2 & 0.0000 & 18.8 \\
+ 3 & 0.0000 & 23.4 \\
+ 4 & 0.0001 & 23.4 \\
+ 5 & 0.0284 & 23.9 \\
+ 10 & 0.0542 & 24.4 \\
+ 20 & 0.0561 & 24.9 \\
+ \bottomrule
+\end{tabular}
