commit 86bca759bad5355fbe90ef17c716ac352c367225
parent d926ca597a95ff63857ecc405b16b83acb5ee430
Author: Silas Brack <s174433@student.dtu.dk>
Date: Thu, 2 Feb 2023 19:45:27 +0100
Fixed experiments.
Diffstat:
2 files changed, 6 insertions(+), 3 deletions(-)
diff --git a/chapters/appendix.tex b/chapters/appendix.tex
@@ -67,16 +67,15 @@
\centering
\includegraphics{hessian_profile_cpu.pdf}
\caption[Comparison of Hessian-vector product performance.]{Top: comparison of the performance for Hessian-vector, GGN-vector, and manual matrix-vector products. Bottom: comparison of the performance of inverse-vector products for the same methods. Implicit inverse performed via conjugate gradient for HVP and GVP. We can see that the explicit Hessian-vector product fails for problems with more than 10\,000 dimensions and that the GVP is faster than the HVP.}
- \label{fig:hessian-profile-cpu}
\end{subfigure}
\hspace{0.5cm}
\begin{subfigure}[h]{0.45\textwidth}
\centering
\includegraphics{hessian_profile_ratios_cpu.pdf}
\caption[Comparison of speedup from HVP to GVP.]{Comparison of the speedup from computation of the Hessian-vector product (HVP) versus the GGN-vector product (GVP) (top) and inverse HVP versus inverse GVP using the conjugate gradient method (bottom). Speedup is calculated as the ratio of the wall-clock time of the HVP to the GVP. We can see that the two methods have the same time complexity (up to a linear factor).}
- \label{fig:hessian-profile-ratios-cpu}
\end{subfigure}
\caption{Performance comparison of different Hessian-vector products on CPU.}
+ \label{fig:hessian-profile-cpu}
\end{figure*}
\begin{figure*}
diff --git a/chapters/boring_experiments.tex b/chapters/boring_experiments.tex
@@ -37,6 +37,7 @@ We then measure the total wall-clock time for computing this Hessian product by
Furthermore, since we are looking to compute some variation of the matrix inverse of \(\bm K\), and inverting a high-dimensional matrix can also be very expensive, we also compare the performance of the \emph{inverse} Hessian-vector product \(\bm K^{-1} \bm v\).
To compute the inverse for the efficient implicit HVP and GVP functions, we use the conjugate gradient method, which only requires the use of matrix-vector multiplications to compute this inverse product.
To compute the explicit inverse, we use the \(\texttt{scipy.linalg}\) package, which uses the LAPACK library to compute the inverse.
+These experiments were run on a single NVIDIA A100 GPU.
% RESULTS
The results for this experiment have been visualised in \cref{fig:hessian-profile}.
@@ -91,6 +92,7 @@ This p-value is then used to determine whether the null hypothesis that the samp
Sample evaluation is explained in more detail in \cref{sec:sampling-evaluation}.
The results of this experiment are shown in \cref{tab:mnist-ablation,fig:mnist-ablation}.
Quantile--quantile plots for each value of \(Q\) are shown in the appendix, in \cref{fig:ablation-chisq}.
+These experiments were run on a single NVIDIA A100 GPU.
\begin{margintable}
\centering
@@ -108,10 +110,12 @@ Inspection of \cref{fig:ablation-chisq} in the appendix suggests that there is n
Using less than 4 quadrature points, however, results in sampling failure, while using exactly 4 quadrature points results in a borderline success.
For 200 posterior samples, a more appropriate significance threshold of \(0.01\) or lower would reject sampling for \(Q \leq 4\), since the cases where sampling fails completely have p-values that are below single floating point precision (i.e., they are effectively zero).
-In taking as much as 20 to 30 minutes to compute 200 posterior samples for an MNIST model with 15\,000 parameters, the time required to compute the samples is not negligible.
+In taking as much as 20 to 30 minutes to compute 200 posterior samples for an MNIST model with 15\,k parameters, the time required to compute the samples is not negligible.
In practice, it is likely that only a small number of samples will be obtained (possibly around 5--15), since for each sample it is necessary to store a parameter vector of size \(D\).
However, the sampling procedure still takes over 20 minutes to compute 10 samples.
As such, the time required to compute the samples may be a significant factor in the overall time required to compute the Laplace approximation.
In particular, for models with a very large number of parameters, the time required to compute the samples may be prohibitive when compared when the near-instant time required to sample from the diagonal Laplace approximation.
Even if the sampling time scales sublinearly with the number of parameters, it would still prevent the application in methods such as online Laplace, which require the Laplace approximation to be computed at each iteration, from being used in practice.
However, this sampling time constraint is also significant for MCMC methods, which are the most common method for sampling from the posterior distribution of a BNN.
+
+\marginnote{Results for the experiments in this chapter run on CPU are shown in the appendix, in \cref{fig:hessian-profile-cpu}.}
