commit 4aec31e55061f3a0495d9d6afcc9cd4cbfa9a385
parent bcea18df74f0e878ec0c8b3654cdccc5e3e5e556
Author: Silas Brack <s174433@student.dtu.dk>
Date: Wed, 1 Feb 2023 22:12:39 +0100
Small fixes.
Diffstat:
6 files changed, 42 insertions(+), 90 deletions(-)
diff --git a/chapters/abstract.tex b/chapters/abstract.tex
@@ -3,90 +3,42 @@
% https://www.student.unsw.edu.au/writing-abstracts-honours-theses
-% What? What did the researcher do in their research?
-% Why? What were the reasons for doing the research? What questions was the researcher trying to answer?
-% How? How did the researcher go about finding out the answers? What methods did they use?
-% What? What did the researcher find out? What were the key results?
-% Why? Why are these results important? What is their significance?
-
-Introduce Bayesian inference and its applications in machine learning.
-General advantages Bayesian neural networks have over other traditional deterministic neural networks (and disadvantages).
-Specific applications where these advantages are important and why they're important.
-Specific applications of Laplace approximation in Bayesian neural networks.
-Why this problem is important.
-
-Bayesian inference is used to ..., but requires an approximation to the posterior to make it computationally tractable.
-The Laplace is one such approximation which assumes a normal distribution for the posterior and uses the Hessian to approximate the posterior precision.
-Often, strong assumptions are made about the Hessian's structure, but these assumptions are not always accurate.
-For example, the Hessian can be approximated by its diagonal or its Kronecker factorisation, but these approximations are not always accurate.
-However, full-Hessian Laplace depends on the full Hessian, which scales quadratically with the number of parameters.
-As such, it is not possible to instantiate and store the Hessian for large-scale problems.
-However, the Hessian matrix is only used to train the evidence and at inference, where it is used to sample from the approximate posterior.
-In the computation of the evidence, we compute the log-determinant of the Hessian, and in the computation of the posterior, we compute the inverse square root of the Hessian to sample from the approximate posterior using the reparameterisation trick.
-
-However, it is possible to compute Hessian-vector products without storing the entire Hessian.
-It is possible to efficiently perform these Hessian-vector products using JAX, a library for automatic differentiation.
-We propose a method for computing the Hessian's log-determinant and inverse square root using only Hessian-vector products.
-This enables us to perform training and inference using the full-Hessian Laplace approximation without storing the entire Hessian.
-
-We show that
-
-% Skin is the largest and most accessible organ in the human body. It is an open window
-% over the overall health of patients, manifesting clues of both systemic and abnormal
-% aspects of their organism. The accessibility of skin makes it extremely attractive for
-% non-invasive sensing of various analytes.
-
-% Diabetes mellitus is a chronic disease that currently affects 537 million adults worldwide.
-% Standard self-monitoring of blood glucose involves invasive practices such as skin-pricking
-% for blood extraction. While microneedle-based devices for continuous glucose monitoring (CGM) have revolutionised the field, disruption of the skin barrier still represents
-% a deterrent for many diabetic patients. Non-invasive optical techniques are therefore
-% expected to lead the next technological paradigm shift within the CGM field.
-
-% The objective of this study is to establish a reliable experimental platform to be used to
-% investigate the interaction of terahertz (THz) light with human skin. This consists of
-% in vitro skin models representing all the essential features of human skin that interact
-% with THz light. THz photon energies cover the energy range of hydrogen bonds forming
-% between water molecules with each other and with proteins and carbohydrates. This
-% feature can be leveraged to sense analytes in the skin matrix via THz radiation.
-% Three different systems are tested in this study via a THz reflection system. First, the
-% data analysis framework is validated on liquid glucose solutions. Next, high water content gelatin hydrogels are prepared to simulate the skin’s collagen mesh and interlying
-% interstitial fluid. Finally, gelatin hydrogel sheets are used to probe the penetration depth
-% of a THz beam in an in vitro skin model.
-
-% The results suggest that a THz beam can penetrate around 140 µm into a 96\% water
-% hydrogel and around 160 µm into a 92\% water hydrogel. Furthermore, thermally cross-linked gelatin hydrogels are deemed unsuitable for THz characterisation of their water
-% content. For future work, chemically cross-linked gelatin hydrogels are recommended
-% instead, as they will also be more thermally and mechanically stable. This will allow
-% further engineering of the skin models, increasing their complexity to involve more biological elements. This will in turn enable a more comprehensive characterisation of
-% human skin in the THz regime.
+Bayesian methods have been used in machine learning for decades, but have recently seen an increased interest in the domain of deep learning, since they promise to provide a principled way to quantify uncertainty in neural networks.
+These methods rely on Bayesian inference to learn the posterior distribution of the model parameters given the data.
+In doing so, they allow for uncertainty estimates, which is important for many applications in machine learning, such as those involving safety-critical decisions which rely on risk assessment and interpretability.
+However, Bayesian inference is often computationally intractable, and as such, approximate methods which compromise performance for feasibility are used.
+The Laplace approximation is one such method which allows for post-hoc estimation of the posterior distribution by assuming a normal distribution for the posterior and using the Hessian to approximate the posterior precision.
+Additionally, since the Laplace approximation deduces uncertainty rather than learning it, it has many advantages over other Bayesian methods which attempt to learn variance.
+Due to its construction, the Laplace approximation enables training by maximising the marginal likelihood, which facilitates model selection and hyperparameter optimisation.
+However, the Laplace approximation typically relies on crude approximations to the posterior precision to make it computationally feasible.
+% Often, strong assumptions are made about the Hessian's structure, but these assumptions are not always accurate.
+Conventional full Laplace uses the full posterior precision, which cannot be instantiated for all but toy problems, since it scales quadratically with the number of parameters.
+In this work, we propose a method for performing the Laplace approximation using only Hessian-vector products, which enables us to perform marginal training and inference using the full Laplace approximation without storing the entire Hessian.
+To accomplish this, we compute the Hessian's log-determinant and inverse square root using only Hessian-vector products.
+We suggest a technique for evaluating the quality of posterior samples without instantiating the Hessian based on results from traditional statistics.
+To implement this, we use JAX, a library for automatic differentiation which enables us to efficiently perform Hessian-vector products without explicitly storing the entire Hessian.
+We show that our method is able to perform the full Laplace approximation on both a sine function and MNIST and that training the marginal likelihood using our method is feasible and produces meaningful results.
+This is the first method for performing the Laplace approximation while only accessing the Hessian implicitly.
+Future work will focus on carrying out hyperparameter optimisation and large-scale experiments using this method.
+MEH, I THINK I SPEND TOO MUCH TIME ON BACKGROUND AND INTRO.
\chapter*{Resum\'e}
\addcontentsline{toc}{chapter}{Resum\'e}
-% \lipsum[1-3]
-
-Huden er det største og mest tilgængelige organ i menneskekroppen.
-Det er et åbent vindue over patienternes generelle helbred og viser spor af både systemiske og unormale aspekter af deres organisme.
-Hudens tilgængelighed gør den yderst attraktiv for ikke-invasiv aftastning af forskellige analytter.
-
-Diabetes mellitus er en kronisk sygdom, som i øjeblikket rammer 537 millioner voksne på verdensplan.
-Standard selvkontrol af blodglukose indebærer invasive metoder som f.eks. hudprikning til udtagning af blod.
-Mens mikronålsbaserede anordninger til kontinuerlig glukoseovervågning (CGM) har revolutioneret området, udgør forstyrrelse af hudbarrieren stadig en en afskrækkende faktor for mange diabetespatienter.
-Ikke-invasive optiske teknikker er derfor forventes derfor at føre an i det næste teknologiske paradigmeskift inden for CGM-området.
-
-Formålet med denne undersøgelse er at etablere en pålidelig eksperimentel platform, der skal bruges til at at undersøge interaktionen mellem terahertz-lys (THz) og menneskelig hud.
-Denne består af in vitro hudmodeller, der repræsenterer alle de væsentlige egenskaber ved den menneskelige hud, som interagerer med THz-lys.
-THz-fotonenergierne dækker energiområdet for hydrogenbindinger, der danner mellem vandmolekyler med hinanden og med proteiner og kulhydrater.
-Dette funktion kan udnyttes til at registrere analytter i hudmatrixen via THz-stråling.
-
-I denne undersøgelse afprøves tre forskellige systemer via et THz-refleksionssystem.
-For det første anvendes dataanalyserammen valideres på flydende glukoseopløsninger.
-Dernæst blev der anvendt høj vandkontent gelatinehydrogeler fremstilles for at simulere hudens kollagennet og de mellemliggende interstitielle væske.
-Endelig anvendes gelatinehydrogelplader til at undersøge indtrængningsdybden af en THz-stråle i en in vitro hudmodel.
-
-Resultaterne tyder på, at en THz-stråle kan trænge omkring 140 µm ind i en 96 \% vand
-hydrogel og ca. 160 µm i en hydrogel med 92 \% vand.
-Endvidere kan termisk krydsende linkede gelatinehydrogeler anses for uegnede til THz-karakterisering af deres vand indhold.
-Til fremtidigt arbejde anbefales kemisk tværbundne gelatinehydrogeler.
-I stedet, da de også vil være mere termisk og mekanisk stabile.
-Dette vil gøre det muligt yderligere udvikling af hudmodellerne og øge deres kompleksitet til at omfatte flere bi- og ologiske elementer. Dette vil igen muliggøre en mere omfattende karakterisering af menneskehud i THz-regimet.
+Bayesianske metoder har v{\ae}ret anvendt inden for maskinl{\ae}ring i {\aa}rtier, men har for nylig oplevet en {\o}get interesse inden for dyb l{\ae}ring, da de lover at give en principiel m{\aa}de at kvantificere usikkerheden i neurale netv{\ae}rk p{\aa}.
+Disse metoder er baseret p{\aa} bayesiansk inferens til at l{\ae}re den efterf{\o}lgende fordeling af modelparametrene i betragtning af dataene.
+Derved giver de mulighed for usikkerhedsestimater, hvilket er vigtigt for mange anvendelser inden for maskinl{\ae}ring, f.eks. anvendelser, der vedr{\o}rer sikkerhedskritiske beslutninger, som er afh{\ae}ngige af risikovurdering og fortolkningsmuligheder.
+Bayesiansk inferens er imidlertid ofte beregningsm{\ae}ssigt uoverkommelig, og derfor anvendes tiln{\ae}rmede metoder, som g{\aa}r p{\aa} kompromis med ydeevnen for at opn{\aa} gennemf{\o}rlighed.
+Laplace-approximationen er en af disse metoder, som giver mulighed for post-hoc-estimation af den efterf{\o}lgende fordeling ved at antage en normalfordeling for den efterf{\o}lgende fordeling og anvende Hessian til at tiln{\ae}rme den efterf{\o}lgende pr{\ae}cision.
+Da Laplace-approksimationen desuden udleder usikkerheden i stedet for at l{\ae}re den, har den mange fordele i forhold til andre bayesianske metoder, som fors{\o}ger at l{\ae}re variansen.
+P{\aa} grund af sin konstruktion g{\o}r Laplace-approksimationen det muligt at tr{\ae}ne ved at maksimere den marginale sandsynlighed, hvilket letter modelvalg og optimering af hyperparametre.
+Laplace-approksimationen er imidlertid typisk afh{\ae}ngig af grove approksimationer af den efterf{\o}lgende pr{\ae}cision for at g{\o}re den computerm{\ae}ssigt gennemf{\o}rlig.
+% Ofte g{\o}res der st{\ae}rke antagelser om Hessians struktur, men disse antagelser er ikke altid n{\o}jagtige.
+Konventionel fuld Laplace anvender den fulde posteriore pr{\ae}cision, som ikke kan instantieres for alle undtagen leget{\o}jsproblemer, da den skalerer kvadratisk med antallet af parametre.
+I dette arbejde foresl{\aa}r vi en metode til at udf{\o}re Laplace-approximationen ved kun at bruge Hessian-vektorprodukter, hvilket g{\o}r det muligt at udf{\o}re marginal tr{\ae}ning og inferens ved hj{\ae}lp af den fulde Laplace-approximation uden at lagre hele Hessian.
+For at opn{\aa} dette beregner vi Hessians log-determinant og den omvendte kvadratrod ved kun at bruge Hessian-vektorprodukter.
+Vi foresl{\aa}r en teknik til at evaluere kvaliteten af efterf{\o}lgende pr{\o}ver uden at instantiere Hessian'en baseret p{\aa} resultater fra traditionel statistik.
+Til at gennemf{\o}re dette bruger vi JAX, et bibliotek til automatisk differentiering, som g{\o}r det muligt for os at udf{\o}re Hessian-vektorprodukter effektivt uden eksplicit at lagre hele Hessian.
+Vi viser, at vores metode er i stand til at udf{\o}re den fulde Laplace-approximation p{\aa} b{\aa}de en sinusfunktion og MNIST, og at tr{\ae}ning af den marginale sandsynlighed ved hj{\ae}lp af vores metode er gennemf{\o}rlig og giver meningsfulde resultater.
+Dette er den f{\o}rste metode til at udf{\o}re Laplace-approximationen, samtidig med at der kun er implicit adgang til Hessian.
+Det fremtidige arbejde vil fokusere p{\aa} optimering af hyperparametre og eksperimenter i stor skala ved hj{\ae}lp af denne metode.
diff --git a/chapters/experiments.tex b/chapters/experiments.tex
@@ -132,7 +132,7 @@ The choice of \(\alpha\) is a non-trivial one
\begin{figure}
\centering
\includegraphics{chisq_sin.pdf}
- \caption[Chi-squared sample plot for sine curve.]{Chi-squared sample plot for testing normality of Laplace approximation samples for a neural network modelling \(f(x) = \sin(x)\). The sample Mahalanobis distances are chi-squared distributed, then the samples themselves are multivariate normal with correct (fixed) covariance. As can be seen by looking at the histogram and the q-q plot, samples are multivariate normal.}
+ \caption[Chi-squared sample plot for sine curve.]{Chi-squared sample plot for visualising normality of 1000 posterior samples from the full Laplace approximation for a fully connected neural network modelling \(f(x) = \sin(x)\). Since the sample square Mahalanobis distances appear chi-squared distributed, the samples themselves are multivariate normal with correct (fixed) covariance. As can be seen by looking at the histogram and the Q--Q plot, samples are multivariate normal.}
\label{fig:chisq-sin}
\end{figure}
diff --git a/chapters/introduction.tex b/chapters/introduction.tex
@@ -41,7 +41,7 @@ This last point is important, since it also means that the Laplace approximation
In the Laplace approximation, to compute the posterior distribution of the model parameters, we need to compute the Hessian of the loss with respect to the model parameters.
Typically, this is done by storing the Hessian matrix explicitly and computing the inverse of this matrix.
-Since the Hessian matrix is a \(D \times D\) square matrix with \(D^2\) elements, where \(D\) is the number of parameters, this is intractable for large models, which can contain millions or billions of parameters, thus limiting the applicability of full-rank Laplace outside of toy problems.
+Since the Hessian matrix is a \(D \times D\) square matrix with \(D^2\) elements, where \(D\) is the number of parameters, this is intractable for large models, which can contain millions or billions of parameters, thus limiting the applicability of full-rank Laplace outside of toy problems of a few thousand parameters.
To overcome this, we can use a crude approximation, such as only storing the Hessian matrix's diagonal~\cite{lecun1989optimal,denker1990transforming}, its Kronecker factorisation~\cite{heskes2000natural,martens2015optimizing,botev2017practical}, and other low-rank approximations.%~\cite{}.
Many of these Hessian approximations have successfully been applied to the Laplace approximation~\cite{ritter2018scalable,ritter2018online}, though are limited by the quality of the approximation.
To overcome this, we propose a method which performs the full-rank Laplace approximation without requiring the Hessian to be stored explicitly.
diff --git a/chapters/laplace.tex b/chapters/laplace.tex
@@ -271,7 +271,7 @@ We then get the simple expression for the GGN approximation of the Hessian for M
\end{align*}
If the Hessian of the loss function with respect to the model outputs \(\bm H_i\) is positive semidefinite, the GGN approximation is positive semidefinite, since each term in the sum \(\bm J_i\T \bm H_i \bm J_i\) is positive semidefinite.
Furthermore, this is the case for most common loss functions, such as MSE loss and cross-entropy loss.
-For the case of the normal prior, the GGN approximation of the posterior precision will be positive definite, since the eigenvalues of \(\alpha \identity\) are alld \(\alpha > 0\) and the eigenvalues of \(\sum_{i=1}^N \bm J_i\T \bm H_i \bm J_i\) are all greater than or equal to zero.
+For the case of the normal prior, the GGN approximation of the posterior precision will be positive definite, since the eigenvalues of \(\alpha \identity\) are all \(\alpha > 0\) and the eigenvalues of \(\sum_{i=1}^N \bm J_i\T \bm H_i \bm J_i\) are all greater than or equal to zero.
% FISHER INFORMATION MATRIX
It is also interesting to note that, for our definition of the loss, the GGN approximation of the Hessian is equivalent to the Fisher information matrix, which is a measure of the curvature of the log-likelihood function with respect to the model parameters.
diff --git a/chapters/training.tex b/chapters/training.tex
@@ -34,7 +34,7 @@ We thus obtain the approximate Laplace posterior
\\ & + \frac{1}{2} (\bm \theta - \bm \theta_{\textsc{map}}) \nabla^2_{\bm \theta} \log p(\bm \theta_{\textsc{map}} \given \bm y) (\bm \theta - \bm \theta_{\textsc{map}})\nonumber
\\={}& \log p(\bm \theta_{\textsc{map}} \given \bm y) - \frac{1}{2} (\bm \theta - \bm \theta_{\textsc{map}}) \bm \Lambda (\bm \theta - \bm \theta_{\textsc{map}}).\nonumber
\end{align}
-The approximate Laplace marginal is then computed as
+The approximate Laplace marginal is then computed as\marginnote{MEH, this page feels a bit too equation-heavy.}
\begin{align}\label{eq:laplace-marginal}
p(\bm y) ={} \stackrel{\textsc{la}}{\approx}{} & \int_{\bm \theta} p(\bm \theta_{\textsc{map}} \given \bm y) \exp\left( -\frac{1}{2} (\bm \theta - \bm \theta_{\textsc{map}}) \bm \Lambda (\bm \theta - \bm \theta_{\textsc{map}}) \right)\,d \bm \theta
\\ ={}& p(\bm \theta_{\textsc{map}} \given \bm y) \int_{\bm \theta} \exp\left( -\frac{1}{2} (\bm \theta - \bm \theta_{\textsc{map}}) \bm \Lambda (\bm \theta - \bm \theta_{\textsc{map}})\right)\,d \bm \theta\nonumber
@@ -153,7 +153,7 @@ Similarly, we compute \(\mu_2\) as
\mu_2 =\Tr\left(\left( {\textstyle {\textstyle \sum_{i=1}^N}} \bm J_i\T \bm H_i \bm J_i \right)^2\right) + 2 \alpha \Tr\left( {\textstyle \sum_{i=1}^N} \bm J_i\T \bm H_i \bm J_i \right) + K \alpha^2
\\ \Tr\left(\left( {\textstyle {\textstyle \sum_{i=1}^N}} \bm J_i\T \bm H_i \bm J_i \right)^2\right) = \E{}{\bm \varepsilon_0\T \left( {\textstyle \sum_{i=1}^N} \bm J_i\T \bm H_i \bm J_i \right) \left( {\textstyle \sum_{i=1}^N} \bm J_i\T \bm H_i \bm J_i \right)\T \bm \varepsilon_0}. \nonumber
\end{align}
-We then apply a Monte Carlo approximation of the expectations by sampling \(S\) samples from \(\bm \varepsilon_0\sim \normal(\bm 0, \identity)\) to compute the traces.
+We then apply a Monte Carlo approximation of the expectation by sampling \(S\) samples from \(\bm \varepsilon_0\sim \normal(\bm 0, \identity)\) to compute the traces.
Fortunately, this means that we only need to compute one GGN-vector product \(\textstyle{\sum_{i=1}^N} \bm J_i\T \bm H_i \bm J_i \cdot \bm \varepsilon_0\) to estimate the \(\mu_1\) and \(\mu_2\) terms in \cref{eq:laplace-log-determinant-bound}.
Furthermore, we have that the trace can be rearranged and simplified to
\begin{align}\label{eq:trace-rearrangement}
diff --git a/main.tex b/main.tex
@@ -202,8 +202,8 @@
% \input{chapters/acknowledgments.tex}
% \index{acknowledgments}
-% \input{chapters/abstract.tex}
-% \index{abstract}
+\input{chapters/abstract.tex}
+\index{abstract}
%----------------------------------------------------------------------------------------
% TABLE OF CONTENTS & LIST OF FIGURES/TABLES
