commit c7b4e8ac2c6a6d10d11c26b2258e9ff22c06a38c
parent 66c7d491590b3f505f25078d7174bf1a0f4bb76d
Author: Silas Brack <s174433@student.dtu.dk>
Date: Sun, 12 Feb 2023 13:07:59 +0100
More small fixes.
Diffstat:
4 files changed, 4 insertions(+), 4 deletions(-)
diff --git a/chapters/experiments.tex b/chapters/experiments.tex
@@ -52,7 +52,7 @@ As per \cref{eq:laplace-log-marginal}, the Laplace-approximate negative log-marg
These terms are shown in \cref{fig:sin-training} (top), where the negative log-likelihood \(-\log p(\bm y \given \bm \theta)\) and negative log-marginal \(- \log p(\bm y)\) terms are both strongly smoothed by a moving average.
We can see that all three terms (and therefore also the negative log-marginal loss itself) decrease during training.
The negative log-likelihood term dominates the overall decrease in the negative log-marginal.
-Since we're optimising the prior precision hyperparameter \(\alpha\) with the marginal likelihood, we also plot its variation throughout training in \cref{fig:sin-training} (bottom).
+Since we are optimising the prior precision hyperparameter \(\alpha\) with the marginal likelihood, we also plot its variation throughout training in \cref{fig:sin-training} (bottom).
We see that \(\alpha\) decreases linearly during training.
In practice, if we keep training, we find that \(\alpha\) reaches a minimum value below 1.
As we showed in \cref{fig:sin}, this is a poor prior precision, since it leads to an unreasonably high posterior predictive uncertainty.
diff --git a/chapters/laplace.tex b/chapters/laplace.tex
@@ -361,7 +361,7 @@ However, this seems to greatly depend on the structure of the neural network (e.
\begin{marginfigure}
\centering
\includegraphics{mnist_spectrum_3vs5.pdf}
- \caption[Spectrum of the posterior precision on MNIST.]{Spectrum of the Laplace posterior precision on 100 MNIST observations with a CNN. Note the log scale and the fact that we're zoomed in on the largest eigenvalues, as can be seen from the eigenvalue indices on the x-axis. For the \(3 \times 3\) filter, there are two distinct clusters in the spectrum, with the first cluster being over an order of magnitude larger than the second. This first cluster consists of nine eigenvalues, and so the precision has an effective rank close to nine. For the \(5 \times 5\) filter, we observe a hyper-exponential distribution of eigenvalues in this spectrum, which does not have any distinct clusters. This suggests that the effective rank of the precision is over 50, which is still significantly lower than the number of observations \(N = 100\) times the number of outputs \(O = 10\).}
+ \caption[Spectrum of the posterior precision on MNIST.]{Spectrum of the Laplace posterior precision on 100 MNIST observations with a CNN. Note the log scale and the fact that we are zoomed in on the largest eigenvalues, as can be seen from the eigenvalue indices on the x-axis. For the \(3 \times 3\) filter, there are two distinct clusters in the spectrum, with the first cluster being over an order of magnitude larger than the second. This first cluster consists of nine eigenvalues, and so the precision has an effective rank close to nine. For the \(5 \times 5\) filter, we observe a hyper-exponential distribution of eigenvalues in this spectrum, which does not have any distinct clusters. This suggests that the effective rank of the precision is over 50, which is still significantly lower than the number of observations \(N = 100\) times the number of outputs \(O = 10\).}
\label{fig:spectrum-mnist}
\end{marginfigure}
diff --git a/chapters/sampling.tex b/chapters/sampling.tex
@@ -329,7 +329,7 @@ This would \emph{hopefully} have an eigenbasis that is close to that of \(\texts
However, scaling the precondititioner up to the size of the entire dataset will lead to a preconditioner which is ill-conditioned, making inversion via conjugate gradient difficult.
As mentioned before, each msMINRES iteration performs one matrix-vector product and one preconditioner-vector product.
-However, even though the preconditioner is constructed in nearly the same way as the sum of GGN matrices in the posterior precision, the preconditioner is significantly cheaper to compute. %, since we're only computing the GGN on a subset of the data.
+However, even though the preconditioner is constructed in nearly the same way as the sum of GGN matrices in the posterior precision, the preconditioner is significantly cheaper to compute.
This is because the preconditioner-vector product will cost \(B / N \cdot t_{\mathrm{GGN}}\), where \(t_{\mathrm{GGN}}\) is the computational cost of performing the GGN-vector product.\sidenote{This analysis assumes that the computation of the GGN-vector product scales linearly in the number of observations used in the GGN matrix. Mathematically speaking, this is a valid assumption, due to the number of elementary operations that are performed in the algorithm. However, the efficient JAX XLA-compiled implementation of Jacobian-vector products can scale differently to na\"ive implementations, so the validity of the assumption is less clear in our case.}
Since we will have \(N \gg B\), then the cost of computing the preconditioner will be negligible compared to that of the GGN-vector product.
diff --git a/chapters/training.tex b/chapters/training.tex
@@ -14,7 +14,7 @@ As seen in \cref{eq:bayes-theorem}, the posterior probability for a given model
\begin{align}
p(\bm \theta \given \bm y) = \frac{p(\bm y \given \bm \theta)\,p(\bm \theta)}{p(\bm y)},
\end{align}
-where \(p(\bm y)\) is known as the evidence. It's also known as the \emph{marginal likelihood}, as it's obtained by marginalising \(\bm \theta\) from the likelihood, as per
+where \(p(\bm y)\) is known as the evidence. It is also known as the \emph{marginal likelihood}, as it is obtained by marginalising \(\bm \theta\) from the likelihood as
\begin{align}
p(\bm y) ={} & \int_{\bm \theta} p(\bm y \given \bm \theta)\,p(\bm \theta)\,d\bm \theta.
\end{align}
