commit 66c7d491590b3f505f25078d7174bf1a0f4bb76d
parent cb24c7dcc378f15be0735100084c55ea7665adce
Author: Silas Brack <s174433@student.dtu.dk>
Date: Sun, 12 Feb 2023 13:04:21 +0100
Fixed formatting, maybe done acknowledgments.
Diffstat:
6 files changed, 64 insertions(+), 57 deletions(-)
diff --git a/chapters/abstract.tex b/chapters/abstract.tex
@@ -29,21 +29,21 @@ This is the first method for performing the Laplace approximation while only acc
Bayesianske metoder lover at give en principiel m{\aa}de at kvantificere usikkerheden i neurale netv{\ae}rk p{\aa}.
Dette er vigtigt for mange anvendelser inden for machine learning, f.eks. i forbindelse med sikkerhedskritiske beslutninger, der er afh{\ae}ngige af risikovurdering og fortolkningsmuligheder.
Bayesiansk inferens er imidlertid ofte beregningsm{\ae}ssigt uh{\aa}ndterbar, s{\aa} der anvendes tiln{\ae}rmede metoder, som g{\aa}r p{\aa} kompromis med ydeevnen for at opn{\aa} praktisk anvendelighed.
-Laplace-approximationen (LA) er en s{\aa}dan metode, selv om den typisk er afh{\ae}ngig af grove tiln{\ae}rmelser af den efterf{\o}lgende pr{\ae}cisionsmatrix for at g{\o}re den beregningsm{\ae}ssigt gennemf{\o}rlig.
+Laplace-approksimationen (LA) er en s{\aa}dan metode, selv om den typisk er afh{\ae}ngig af grove tiln{\ae}rmelser af den efterf{\o}lgende pr{\ae}cisionsmatrix for at g{\o}re den beregningsm{\ae}ssigt gennemf{\o}rlig.
N{\aa}r vi ikke faktoriserer den efterf{\o}lgende pr{\ae}cision i mere praktiske tiln{\ae}rmelser, kr{\ae}ver de eksisterende metoder, at den fulde efterf{\o}lgende pr{\ae}cision skal instantieres.
-Vi kalder dette den fulde Laplace-approximation.
+Vi kalder dette den fulde Laplace-approksimation.
Da den fulde pr{\ae}cisionsmatrix imidlertid for alle problemer undtagen leget{\o}jsproblemer skalerer kvadratisk med antallet af parametre, er denne matrix for stor til at blive lagret i hukommelsen.
-I dette arbejde foresl{\aa}r vi en metode til beregning af Laplace-approksimationen ved hj{\ae}lp af Jacobian-vektorprodukter alene.
-Dette giver os mulighed for at udf{\o}re marginal tr{\ae}ning og efterf{\o}lgende stikpr{\o}veudtagning ved hj{\ae}lp af den fulde Laplace-approximation uden at lagre hele den efterf{\o}lgende pr{\ae}cisionsmatrix.
+I dette studie foresl{\aa}r vi en metode til beregning af Laplace-approksimationen ved hj{\ae}lp af Jacobian-vektorprodukter alene.
+Dette giver os mulighed for at udf{\o}re marginal tr{\ae}ning og efterf{\o}lgende stikpr{\o}veudtagning ved hj{\ae}lp af den fulde Laplace-approksimation uden at lagre hele den efterf{\o}lgende pr{\ae}cisionsmatrix.
For at opn{\aa} dette viser vi, at vi kan estimere den posteriore pr{\ae}cisionens log-determinant og den inverse kvadratrod ved hj{\ae}lp af kun jacobianiske vektorprodukter.
-For at overvinde de konditioneringsproblemer, der opst{\aa}r under sampling, tilbyder vi flere potentielle pr{\ae}konditioneringsv{\ae}rkt{\o}jer, som kan anvendes til at forbedre konvergensen af den inverse kvadratrodsapproximation.
+For at overvinde de konditioneringsproblemer, der opst{\aa}r under sampling, tilbyder vi flere potentielle pr{\ae}konditioneringsv{\ae}rkt{\o}jer, som kan anvendes til at forbedre konvergensen af den inverse kvadratrodsapproksimation.
Dern{\ae}st giver vi en teknik til at evaluere kvaliteten af de efterf{\o}lgende stikpr{\o}ver uden at instantiere eller invertere den efterf{\o}lgende pr{\ae}cision baseret p{\aa} resultater fra traditionel statistik.
Til at gennemf{\o}re disse metoder anvender vi JAX, et bibliotek til automatisk differentiering, som g{\o}r det muligt at udf{\o}re jacobian-vektorprodukter effektivt uden eksplicit lagring af hele den efterf{\o}lgende pr{\ae}cision.
-Vi udf{\o}rer den fulde Laplace-approximation p{\aa} b{\aa}de en sinusfunktion og MNIST ved hj{\ae}lp af vores metode.
+Vi udf{\o}rer den fulde Laplace-approksimation p{\aa} b{\aa}de en sinusfunktion og MNIST ved hj{\ae}lp af vores metode.
Denne best{\aa}r af to trin: (a) tr{\ae}ning af et neuralt netv{\ae}rk ved at maksimere enten dets posterior sandsynlighed eller marginale sandsynlighed og (b) pr{\o}veudtagning fra denne posterior.
Vores tiln{\ae}rmede maksimale marginale tr{\ae}ningsprocedure er i stand til at l{\ae}re et s{\ae}t parametre, som giver en ydeevne, der er sammenlignelig med den maksimale posterior- og maksimale sandsynlighedstr{\ae}ningsprocedure.
Ved at analysere quantile--quantile-plots af vores posteriorpr{\o}ver og visualisere disse pr{\o}ver finder vi, at vores tiln{\ae}rmede pr{\o}veudtagningsmetode giver pr{\o}ver, der er korrekt fordelt.
-Vores benchmarks af Jacobian-vektorprodukternes ydeevne ansl{\aa}r, at vores metode giver en 10\,000x hukommelsesreduktion i forhold til den konventionelle fulde Laplace-approximation, da den ikke lagrer den kvadratisk skalerende matrix med fuld pr{\ae}cision for den efterf{\o}lgende periode.
-Dette er den f{\o}rste metode til at udf{\o}re Laplace-approximationen, hvor der kun er implicit adgang til den efterf{\o}lgende pr{\ae}cision.
+Vores benchmarks af Jacobian-vektorprodukternes ydeevne ansl{\aa}r, at vores metode giver en 10\,000x hukommelsesreduktion i forhold til den konventionelle fulde Laplace-approksimation, da den ikke lagrer den kvadratisk skalerende matrix med fuld pr{\ae}cision for den efterf{\o}lgende periode.
+Dette er den f{\o}rste metode til at udf{\o}re Laplace-approksimationen, hvor der kun er implicit adgang til den efterf{\o}lgende pr{\ae}cision.
diff --git a/chapters/acknowledgments.tex b/chapters/acknowledgments.tex
@@ -1,24 +1,32 @@
\chapter*{Acknowledgments}
\addcontentsline{toc}{chapter}{Acknowledgments} % Add the ackn to the table of contents as a chapter
-\blackout
-{
- To S{\o}ren's group---especially Marco---for always making me feel like a part of the group and that my contribution was valued.
+From the first day of the hackathon, I've felt like a part of the team.
+And this is thanks to everyone in S{\o}ren's group---especially Marco---who have always made me feel included and like my contribution was valued.
+Everyone in the group has been helpful and kind, and I'm glad I've had the opportunity to work with bright, funny people who are always up for a good chat about anything and everything, right down to the last chat about five-for-one deals on paintings.
+You guys made the experience what it was.
- To Frederik and Marco for giving me the chance to dip my toes in ``real'' research in the most undiluted way possible, with what I perceive to be all the highs and the lows which constitute it.
+I'd also like to emphasise my appreciation to Frederik and Marco for introducing me to research and helping me bounce back from my initial frustrations.
+I've learned a lot from you guys, and I've gotten a lot of great advice from you.
+I've learned a lot from everyone in the group, actually.
- To S{\o}ren, for always being up for a great chat, for giving me the opportunity to work on difficult and novel problems (like preconditioning), and for approaching everything with kindness.
- I don't think I could have chosen a better supervisor.
+It's hard to overstate S{\o}ren's impact on my master's thesis experience.
+Thanks for always being up for a great chat (it led to me doing this thesis in the first place), for giving me the opportunity to work on challenging problems, and for being kind.
+I always felt welcome to drop by so I could bounce ideas off of you; you somehow always managed to make me feel like I was making progress, even when I was stuck.
+I don't think I could have chosen a better supervisor.
- To J\'ulia, Brandon and Julia for the constant laughs and endless opportunities to relax on the couch when working was too much (memes surrounding me working on the couch aside) and Adam, with who I shared the experience of writing a thesis---highs and lows both.
+I'm also grateful to Julia, J\'ulia, Brandon, and \'Ad\'am (the person, not the optimiser) for the endless opportunities to sit on the couch and relax and (definitely) not do any work.
+% I really appreciate having had your support.
+Thanks for being great friends.
- To Jason and Kristi for always cultivating/nurturing in me a sense of curiosity and independence.
- And for never forgetting me in the supermarket.
+I couldn't have done any of this without Jason and Kristi nurturing in me a sense of curiosity, independence, and kindness.
+The three of us go way back; and it's hard to say if I would've come to Denmark without their encouragement.
+Thanks for never forgetting me in the supermarket.
- To Lorenza, for convincing me to go to the Hackathon, for being a great friend and for being the best person to have a chat with when I was feeling down.
+To Lorenza, for convincing me to follow through with the hackathon, for being a great friend and for being the best person to chat with when I was feeling overwhelmed.
+You've unwaveringly supported me through this whole process, and I'm glad I've had the chance to get to know you.
- And above all else, to Steampunk.
-}
+And above all else, to Steampunk.
\begin{flushright}
\textit{Silas Brack}
diff --git a/chapters/laplace.tex b/chapters/laplace.tex
@@ -5,14 +5,13 @@
In this chapter, we discuss training and inference in deep learning from a Bayesian perspective.
We introduce the Laplace approximation as a method for approximating the posterior distribution of a model's parameters given the data.
-We analyse the construction of the Laplace approximate posterior and discuss its properties and how it can be computed in practice.
+We discuss the construction of the Laplace posterior, its properties, and how it can be computed in practice.
\section{Optimisation in Deep Learning}[Optimisation]
\label{sec:optimisation-frequentist}
Suppose a neural network is a real-valued function \(f: \reals^N \times \reals^D \rightarrow \reals^O\) parametrised in \(\bm\theta \in \reals^D\) which maps an input \(\bm x \in \reals^N\) to an output \(f(\bm x, \bm\theta) \equiv f_{\bm\theta}(\bm x)\).
Our goal is to find the parameters \(\bm\theta^*\) which best model the observed data.
-
From a frequentist perspective, we can define a loss function on the model \(\mathcal{L}(\bm\theta): \reals^D \rightarrow \reals\) such that the optimal parameters \(\bm\theta^*\) minimise this loss function for the given model.
Often, we formulate the loss function as the negative log-likelihood of the data under the model, \(\mathcal{L}(\bm\theta) = -\log \lik\).
In this case, the goal of optimisation is to find the parameters \(\bm\theta^*\) which maximise the likelihood of the data.\sidenote{Maximising the likelihood is equivalent to minimising the negative log-likelihood, since the log function is strictly monotonically increasing.}
@@ -20,7 +19,6 @@ In this case, the goal of optimisation is to find the parameters \(\bm\theta^*\)
The likelihood function is problem-specific, and is defined as the probability density of the data given the parameters \(\bm\theta\).
Typically, the likelihood is chosen to be a Gaussian distribution with mean \(f_{\bm\theta}(\bm x)\) and precision \(\rho\) (which is frequently chosen to be equal to one) for regression problems, and a categorical distribution with logits \(f_{\bm\theta}(\bm x)\) for classification problems.
These two cases correspond to the mean-squared error (MSE) and cross-entropy (CE) negative log-likelihood loss functions, respectively.
-
Due to the assumption of independence between observations, we can factorise the likelihood and the negative log-likelihood loss as
\begin{align}\label{eq:negative-log-likelihood}
p(\bm y \given \bm \theta) ={} & \prod_i^N p(\bm y_i \given \bm \theta),
@@ -60,27 +58,26 @@ SGD helps avoid getting stuck in local minima, but, like gradient descent, does
\label{sec:adam}
Adam~\cite{kingma2014adam} is a popular alternative to SGD for training neural networks.
-It is an approximate second-order optimisation method, which means that it approximates the curvature of the loss function.
-It is a variant of SGD that uses the first and second moments of the gradient of the loss function to scale the learning rate of each parameter.
+It is an approximate second-order optimisation method, which means that it approximates the curvature of the loss.
+It is a variant of SGD that uses the first and second moments of the gradient of the loss to scale the learning rate of each parameter.
In Adam, the first moment of the gradient is approximated by a moving average of the gradient and the second moment of the gradient is approximated by a moving average of the squared gradient.
-The moving averages are weighted by the parameters \(\beta_1\) and \(\beta_2\) respectively.
+The moving averages are weighted by the parameters \(\beta_1\) and \(\beta_2\).
The learning rate is then scaled by the ratio of the first and second moments.
-The update rule for Adam is thus given by
+The update rule for Adam is thus
\begin{align}
\bm\theta_{t+1} ={} & \bm\theta_t - \eta_t \bm m_t \odot \left(\sqrt{\bm v_t} + \varepsilon\right)^{-1},
\\ \bm m_t ={}& (1 - \beta_1^t) \left[ \beta_1 \bm m_{t-1} + (1 - \beta_1) \nabla_{\bm\theta} \mathcal{L}(\bm\theta_t) \right],
\\ \bm v_t ={}& (1 - \beta_2^t) \left[ \beta_2 \bm v_{t-1} + (1 - \beta_2) \nabla_{\bm\theta} \mathcal{L}(\bm\theta_t) \odot \nabla_{\bm\theta} \mathcal{L}(\bm\theta_t) \right],
\end{align}
where \(\odot\) denotes the Hadamard product, \(\varepsilon\) is a small constant to prevent division by zero, and \(\bm m_t\) and \(\bm v_t\) are the first and second moments of the gradient respectively.
-
The momentum term~\cite{polyak1964some} takes inspiration from physics, where the momentum of an object is the product of its mass and velocity.
In Adam, the first moment of the gradient is used to compute the momentum of the parameters, increasing the convergence speed of the optimisation~\cite{sutskever2013importance}.
The second moment of the gradient is also known as the variance of the gradient.
Adam approximates the second moment by the diagonal of the Fisher information matrix where the loss and model function are assumed to be linear.
This is a common approximation in practice, as this approximation is easy to compute.
The Fisher information matrix is discussed further in \cref{sec:ggn}.
-The use of the second moment is inspired by optimisation methods such as Newton's method and the natural gradient~\cite{rattray1998natural}, which use the second derivative of the loss function to compute the optimal step size by scaling the step by the inverse of the curvature of the loss function.
-These optimisation methods all identify a single set of parameters as the optimal solution.
+The use of the second moment is inspired by optimisation methods such as Newton's method and the natural gradient~\cite{rattray1998natural}, which use the second derivative of the loss to compute the optimal step size by scaling the step by the inverse of the curvature of the loss.
+These optimisation methods identify a single set of parameters as the optimal solution.
In the next section, we will discuss how to learn a distribution over parameters instead.
\section{Maximum a Posteriori Estimation}[MAP Estimation]
@@ -95,15 +92,14 @@ It is obtained from Bayes' theorem, which allows for uncertainty quantification
\end{align}
where \(p(\bm \theta \given \bm y)\) is the posterior distribution over the parameters, \(p(\bm \theta)\) is the prior distribution over parameters, \(p(\bm y \given \bm \theta)\) is the likelihood of the data given the parameters, and \(p(\bm y)\) is the marginal likelihood of the data.
The marginal likelihood is obtained by marginalising over the parameters, i.e., \(p(\bm y) = \int p(\bm y \given \bm \theta)\,p(\bm \theta)\,d\bm \theta\).
-Due to this integral, the marginal likelihood is often intractable.
+Due to this integral, the marginal is often intractable.
However, since the posterior distribution is proportional to the product of the likelihood by the prior, we can obtain the unnormalised posterior from the likelihood and the prior\marginnote{Since the marginal likelihood is a normalisation constant and does not depend on the parameters, it does not affect the optimisation problem.}
\begin{align}\label{eq:unnormalised posterior}
p(\bm \theta \given \bm y) \propto p(\bm y \given \bm \theta)\,p(\bm \theta) \eqqcolon \tilde{p}(\bm \theta \given \bm y) .
\end{align}
Since we want to determine the parameters which maximise the posterior distribution, we can typically use the same optimisation techniques as for frequentist learning, but with the negative log-posterior loss instead of the negative log-likelihood.
Furthermore, since the unnormalised posterior from \cref{eq:unnormalised posterior} is proportional to the likelihood and the prior, the exact posterior itself need not be tractable.
-
-Similarly to the negative log-likelihood, we can factorise the negative log-posterior loss as per
+Similarly to the negative log-likelihood, we factorise the negative log-posterior
\begin{align}\label{eq:negative-log-posterior}
\begin{split}
\mathcal{L}(\bm\theta) = -\log p(\bm \theta \given \bm y) ={} & - \log p(\bm y \given \bm \theta) - \log p(\bm\theta)
@@ -116,10 +112,10 @@ Here the regularisation term is given by the prior distribution \(p(\bm\theta)\)
Due to the black-box nature of neural networks, it is often difficult to define a prior distribution over the parameters using previous knowledge.
Because of this, the prior is often chosen to be a Gaussian distribution with zero mean and a diagonal covariance matrix with some precision \(\alpha\), i.e., \(p(\bm\theta) = \normal(\bm\theta \given \bm 0, \alpha^{-1}\bm I)\).\sidenote{In the frequentist setting, this type of regularisation is known as weight decay or \(L_2\)-regularisation. Note that \(- 1 / N \log p(\bm y \given \bm \theta) - \gamma / 2 \norm{\bm \theta}^2\) is the typical formulation in this setting, where \(\gamma\) is the regularisation strength and is usually within the range of \(10^{-2}\) to \(10^{-4}\). In our setting, we are instead computing the log-posterior \(-\log p(\bm y \given \bm \theta) - \alpha / 2 \norm{\bm \theta}^2\), and so we must multiply the regularisation strength by \(N\) to obtain \(\alpha\).}
The likelihood term is defined as in the frequentist setting (see \cref{sec:optimisation-frequentist}).
-However, the posterior distribution itself is typically intractable, so approximations to the posterior are used instead.
+However, the posterior distribution itself is typically intractable, so we approximate the posterior instead.
-We use the posterior distribution to quantify the uncertainty in our model parameters.
-Often, all that is needed is a set of samples from the posterior distribution.
+We use the posterior to quantify the uncertainty in our model parameters.
+Often, all that is needed is a set of samples from the posterior.
From here, we can make predictions in a Bayesian setting by computing the posterior predictive distribution and using Monte Carlo sampling as
\begin{align}\label{eq:posterior predictive}
\begin{split}
diff --git a/chapters/preface.tex b/chapters/preface.tex
@@ -1,25 +1,27 @@
\chapter*{Preface}
\addcontentsline{toc}{chapter}{Preface} % Add the preface to the table of contents as a chapter
-\blackout
-{
- I've been studying for a pretty long time.
- I suppose that, because of this, there is some sense of anticipation, of the sensation that a culmination/apogee of a.
- Many people have had mixed experiences while writing their theses, though in general people seem to associate the process with a sense of accomplishment, since it is perceived to be the step in a long journey.
- difficulty
- I, not always being the biggest adherent to the idea of symbol-for-the-sake-of-symbol, wasn't sure what to expect of this experience
+I've been studying for a pretty long time.
+I suppose that, because of this, there is some sense of anticipation, of the sensation that this is the culmination of a long journey.
+Not always having been the biggest adherent to the idea of symbol-for-the-sake-of-symbol, I wasn't sure what to expect of this experience.
+Many people have had mixed experiences while writing their theses, though in general people seem to associate the process with a sense of accomplishment, since it is perceived to be the step in a long journey.
+However, while there have been
- I had a slow start to the thesis, since for the first month or so I just explored the software and the literature, and tried to get a feel for the problem.
- As I tried to generate some preliminary results, I encountered a few roadblocks.
- As it turned out, these problems were quite fundamental to the problem---but S{\o}ren had recently read a paper which addressed them and invited me to a Hackathon with his research group, where we would try to implement the methods from the paper.
- Initially, I was a bit hesitant to go, since I had never been to a Hackathon before, and I was worried that I would be out of my depth and that I would not be able to contribute.
- However, Lorenza helped me get over my fear of marginalisation and I decided to go, and I am glad that I did.
+I definitely had a slow start to the thesis (most people do), since for the first month or so I just explored the software and the literature, and tried to get a feel for the problem.
+While I tried to generate some preliminary results, I quickly encountered some roadblocks.
+As it turned out, these issues were quite fundamental to the problem---but S{\o}ren had found a paper which addressed them and invited me to a hackathon with his research group, where he was planning to try to implement the methods from the paper.
+Initially, I was a bit hesitant about going, since I had never been to a hackathon in a research group, and I was worried that I would be out of my depth and that I would not be able to contribute.
+However, Lorenza helped me get over my fear of marginalisation and I decided to go, and I am glad that I did.
- This thesis is the product of a five-month-long span of work conducted in the Section for Cognitive Systems of DTU Compute, during which I helped develop some of the ideas in the ``Large-Scale Laplace'' project.
- This project is, at the time of writing, still ongoing, and I hope that we will be able to publish the results of our work in the near future.
- In this project, I developed and implemented the entire sampling procedure for the full Hessian Laplace approximation.
- However, I also contributed to the derivation and implementation of the marginal training algorithm.
-}
+The Hackathon and the project that followed it were probably the most fun I've had studying.
+
+I've gotten the chance to work on some really novel problems, like preconditioning and numerical issues, and have enjoyed every minute of it.
+
+This thesis is the product of a five-month-long span of work conducted in the Section for Cognitive Systems of DTU Compute, during which I helped develop some of the ideas in the ``Large-Scale Laplace'' project.
+This project is, at the time of writing, still ongoing; hopefully we'll soon publish the results of our work.
+In this project, I developed and implemented the sampling procedure for the full Hessian Laplace approximation.
+This includes the implementation of the sampling algorithm (and all the mathematical machinery necessary to make it work), as well as the formulation, implementation and testing of preconditioners.
+I also contributed to parts of the derivation and implementation of the marginal training algorithm, though I would say the bulk of this was done by Hrittik, Johan and Pablo.
\begin{flushright}
\textit{Silas Brack}
diff --git a/chapters/sampling.tex b/chapters/sampling.tex
@@ -413,7 +413,7 @@ Assume that the pivoted Cholesky factorisation finds the \(K\) largest eigenvect
If the eigenspectrum of this factorisation is steep enough, then the largest eigenvalue \(s_1\) will be significantly larger than the smallest eigenvalue \(s_K\) (i.e., there will be a steep drop-off from the largest to the smallest non-zero eigenvalue).
Since the eigenvectors of each per-observation term \(\bm J_i\T \bm H_i \bm J_i\) will not be the same, when we sum over them, the eigenvalues of the sum will not be the sum of the eigenvalues of the individual Hessians.
However, some of the eigenvectors may be close to each other, and so the spectrum of the GGN sum may be approximately the sum of the GGN eigenvalues.
-However, the rank of the total Hessian will increase with the number of observations and so the number of non-zero eigenvalues consequently increase as well.
+Nevertheless, the rank of the total Hessian will increase with the number of observations and so the number of non-zero eigenvalues consequently increase as well.
These eigenvalues will be smaller than the summed eigenvalues (because the eigenvectors may not be contained in the same subspace), but they will be larger than the smallest eigenvalue of the per-observation Hessian.
This will then lead to a steeper spectrum and a larger condition number of the total Hessian.
Thus, when the batch size \(B\) is large, the condition number of the sum of GGN Hessians will increase and the preconditioner will become less effective.
@@ -430,9 +430,10 @@ Alternatively, we can approximate the diagonal of the GGN matrix as the element-
In conclusion, to find the optimal parameters for the pivoted Cholesky preconditioner (the number of observations \(B\) and the rank of the pivoted Cholesky factorisation \(K\)), there are some rules of thumb to consider.
Notably, the quality of the preconditioner depends largely on the ease of inverting the \(\identity + \alpha^{-1} \bm L \bm L\T\) matrix, which depends on its condition number.
In practice, increasing the rank of the pivoted Cholesky factorisation will increase the condition number of this matrix, since we are approximating more of the largest eigenvectors and so we will capture progressively smaller eigenvalues.
-I've found that it's most effective to keep \(K\) low, such that the \(K \times K\) outer product of the pivoted Cholesky factorisation matrix \(L\) approximates few enough of the largest eigenvectors that the eigenvalue of the largest is close to that of the smallest (meaning this outer product is well-conditioned).
+
+We have found that it is most effective to keep \(K\) low, such that the \(K \times K\) outer product of the pivoted Cholesky factorisation matrix \(L\) approximates few enough of the largest eigenvectors that the eigenvalue of the largest is close to that of the smallest (meaning this outer product is well-conditioned).
We thus need to choose \(K\) such that it is large enough to approximate enough large eigenvectors and small enough to ensure the condition number of the outer product is low.
-Furthermore, you want enough observations \(B\) to be included in the preconditioner such that it's possible to accurately approximate the eigenvectors of the problem---it may be optimal to use the whole dataset, though this may worsen the conditioning of the outer product.
+Furthermore, you want enough observations \(B\) to be included in the preconditioner such that it is possible to accurately approximate the eigenvectors of the problem---it may be optimal to use the whole dataset, though this may worsen the conditioning of the outer product.
Furthermore, the pivoted Cholesky decomposition is not implemented in JAX.
Since the pivoted Cholesky decomposition involves incrementally adding columns to the factorisation, and JAX is not friendly towards mutable variables, this preconditioner is not straightforward to implement.
diff --git a/main.tex b/main.tex
@@ -104,7 +104,7 @@
%----------------------------------------------------------------------------------------
% \titlehead{M.Sc. Thesis}
-\subject{Masters of Science in Engineering}
+\subject{Master of Science in Engineering}
\title{Effortless Bayesian Deep Learning}
\subtitle{Tapping Into the Potential of Modern Optimizers}
