commit 8853e7f71e4244ae5b25f8c7f3c413440226792e
parent cb6de7a253628739abb0dc2ddefa6eb5af819b60
Author: Silas Brack <s174433@student.dtu.dk>
Date: Wed, 25 Jan 2023 22:31:45 +0100
More writing.
Diffstat:
10 files changed, 256 insertions(+), 131 deletions(-)
diff --git a/chapters/abstract.tex b/chapters/abstract.tex
@@ -1,64 +1,88 @@
\chapter*{Abstract}
\addcontentsline{toc}{chapter}{Abstract}
-% \lipsum[1-3]
-
-Skin is the largest and most accessible organ in the human body. It is an open window
-over the overall health of patients, manifesting clues of both systemic and abnormal
-aspects of their organism. The accessibility of skin makes it extremely attractive for
-non-invasive sensing of various analytes.
-
-Diabetes mellitus is a chronic disease that currently affects 537 million adults worldwide.
-Standard self-monitoring of blood glucose involves invasive practices such as skin-pricking
-for blood extraction. While microneedle-based devices for continuous glucose monitoring (CGM) have revolutionised the field, disruption of the skin barrier still represents
-a deterrent for many diabetic patients. Non-invasive optical techniques are therefore
-expected to lead the next technological paradigm shift within the CGM field.
-
-The objective of this study is to establish a reliable experimental platform to be used to
-investigate the interaction of terahertz (THz) light with human skin. This consists of
-in vitro skin models representing all the essential features of human skin that interact
-with THz light. THz photon energies cover the energy range of hydrogen bonds forming
-between water molecules with each other and with proteins and carbohydrates. This
-feature can be leveraged to sense analytes in the skin matrix via THz radiation.
-Three different systems are tested in this study via a THz reflection system. First, the
-data analysis framework is validated on liquid glucose solutions. Next, high water content gelatin hydrogels are prepared to simulate the skin’s collagen mesh and interlying
-interstitial fluid. Finally, gelatin hydrogel sheets are used to probe the penetration depth
-of a THz beam in an in vitro skin model.
-
-The results suggest that a THz beam can penetrate around 140 µm into a 96\% water
-hydrogel and around 160 µm into a 92\% water hydrogel. Furthermore, thermally cross-linked gelatin hydrogels are deemed unsuitable for THz characterisation of their water
-content. For future work, chemically cross-linked gelatin hydrogels are recommended
-instead, as they will also be more thermally and mechanically stable. This will allow
-further engineering of the skin models, increasing their complexity to involve more biological elements. This will in turn enable a more comprehensive characterisation of
-human skin in the THz regime.
-
-\chapter*{Resum\'e}
-\addcontentsline{toc}{chapter}{Resum\'e}
-
-% \lipsum[1-3]
-
-Huden er det største og mest tilgængelige organ i menneskekroppen.
-Det er et åbent vindue over patienternes generelle helbred og viser spor af både systemiske og unormale aspekter af deres organisme.
-Hudens tilgængelighed gør den yderst attraktiv for ikke-invasiv aftastning af forskellige analytter.
-
-Diabetes mellitus er en kronisk sygdom, som i øjeblikket rammer 537 millioner voksne på verdensplan.
-Standard selvkontrol af blodglukose indebærer invasive metoder som f.eks. hudprikning til udtagning af blod.
-Mens mikronålsbaserede anordninger til kontinuerlig glukoseovervågning (CGM) har revolutioneret området, udgør forstyrrelse af hudbarrieren stadig en en afskrækkende faktor for mange diabetespatienter.
-Ikke-invasive optiske teknikker er derfor forventes derfor at føre an i det næste teknologiske paradigmeskift inden for CGM-området.
-
-Formålet med denne undersøgelse er at etablere en pålidelig eksperimentel platform, der skal bruges til at at undersøge interaktionen mellem terahertz-lys (THz) og menneskelig hud.
-Denne består af in vitro hudmodeller, der repræsenterer alle de væsentlige egenskaber ved den menneskelige hud, som interagerer med THz-lys.
-THz-fotonenergierne dækker energiområdet for hydrogenbindinger, der danner mellem vandmolekyler med hinanden og med proteiner og kulhydrater.
-Dette funktion kan udnyttes til at registrere analytter i hudmatrixen via THz-stråling.
-
-I denne undersøgelse afprøves tre forskellige systemer via et THz-refleksionssystem.
-For det første anvendes dataanalyserammen valideres på flydende glukoseopløsninger.
-Dernæst blev der anvendt høj vandkontent gelatinehydrogeler fremstilles for at simulere hudens kollagennet og de mellemliggende interstitielle væske.
-Endelig anvendes gelatinehydrogelplader til at undersøge indtrængningsdybden af en THz-stråle i en in vitro hudmodel.
-
-Resultaterne tyder på, at en THz-stråle kan trænge omkring 140 µm ind i en 96 \% vand
-hydrogel og ca. 160 µm i en hydrogel med 92 \% vand.
-Endvidere kan termisk krydsende linkede gelatinehydrogeler anses for uegnede til THz-karakterisering af deres vand indhold.
-Til fremtidigt arbejde anbefales kemisk tværbundne gelatinehydrogeler.
-I stedet, da de også vil være mere termisk og mekanisk stabile.
-Dette vil gøre det muligt yderligere udvikling af hudmodellerne og øge deres kompleksitet til at omfatte flere bi- og ologiske elementer. Dette vil igen muliggøre en mere omfattende karakterisering af menneskehud i THz-regimet.
+% https://www.student.unsw.edu.au/writing-abstracts-honours-theses
+
+% What? What did the researcher do in their research?
+% Why? What were the reasons for doing the research? What questions was the researcher trying to answer?
+% How? How did the researcher go about finding out the answers? What methods did they use?
+% What? What did the researcher find out? What were the key results?
+% Why? Why are these results important? What is their significance?
+
+Introduce Bayesian inference and its use in machine learning.
+
+Bayesian inference is used to ..., but requires an approximation to the posterior to make it computationally tractable.
+The Laplace is one such approximation which assumes a normal distribution for the posterior and uses the Hessian to approximate the posterior precision.
+Often, strong assumptions are made about the Hessian's structure, but these assumptions are not always accurate.
+For example, the Hessian can be approximated by its diagonal or its Kronecker factorisation, but these approximations are not always accurate.
+However, full-Hessian Laplace depends on the full Hessian, which scales quadratically with the number of parameters.
+As such, it is not possible to instantiate and store the Hessian for large-scale problems.
+However, the Hessian matrix is only used to train the evidence and at inference, where it is used to sample from the approximate posterior.
+In the computation of the evidence, we compute the log-determinant of the Hessian, and in the computation of the posterior, we compute the inverse square root of the Hessian to sample from the approximate posterior using the reparameterisation trick.
+
+However, it is possible to compute Hessian-vector products without storing the entire Hessian.
+It is possible to efficiently perform these Hessian-vector products using JAX, a library for automatic differentiation.
+We propose a method for computing the Hessian's log-determinant and inverse square root using only Hessian-vector products.
+This enables us to perform training and inference using the full-Hessian Laplace approximation without storing the entire Hessian.
+
+We show that
+
+% Skin is the largest and most accessible organ in the human body. It is an open window
+% over the overall health of patients, manifesting clues of both systemic and abnormal
+% aspects of their organism. The accessibility of skin makes it extremely attractive for
+% non-invasive sensing of various analytes.
+
+% Diabetes mellitus is a chronic disease that currently affects 537 million adults worldwide.
+% Standard self-monitoring of blood glucose involves invasive practices such as skin-pricking
+% for blood extraction. While microneedle-based devices for continuous glucose monitoring (CGM) have revolutionised the field, disruption of the skin barrier still represents
+% a deterrent for many diabetic patients. Non-invasive optical techniques are therefore
+% expected to lead the next technological paradigm shift within the CGM field.
+
+% The objective of this study is to establish a reliable experimental platform to be used to
+% investigate the interaction of terahertz (THz) light with human skin. This consists of
+% in vitro skin models representing all the essential features of human skin that interact
+% with THz light. THz photon energies cover the energy range of hydrogen bonds forming
+% between water molecules with each other and with proteins and carbohydrates. This
+% feature can be leveraged to sense analytes in the skin matrix via THz radiation.
+% Three different systems are tested in this study via a THz reflection system. First, the
+% data analysis framework is validated on liquid glucose solutions. Next, high water content gelatin hydrogels are prepared to simulate the skin’s collagen mesh and interlying
+% interstitial fluid. Finally, gelatin hydrogel sheets are used to probe the penetration depth
+% of a THz beam in an in vitro skin model.
+
+% The results suggest that a THz beam can penetrate around 140 µm into a 96\% water
+% hydrogel and around 160 µm into a 92\% water hydrogel. Furthermore, thermally cross-linked gelatin hydrogels are deemed unsuitable for THz characterisation of their water
+% content. For future work, chemically cross-linked gelatin hydrogels are recommended
+% instead, as they will also be more thermally and mechanically stable. This will allow
+% further engineering of the skin models, increasing their complexity to involve more biological elements. This will in turn enable a more comprehensive characterisation of
+% human skin in the THz regime.
+
+% \chapter*{Resum\'e}
+% \addcontentsline{toc}{chapter}{Resum\'e}
+
+% % \lipsum[1-3]
+
+% Huden er det største og mest tilgængelige organ i menneskekroppen.
+% Det er et åbent vindue over patienternes generelle helbred og viser spor af både systemiske og unormale aspekter af deres organisme.
+% Hudens tilgængelighed gør den yderst attraktiv for ikke-invasiv aftastning af forskellige analytter.
+
+% Diabetes mellitus er en kronisk sygdom, som i øjeblikket rammer 537 millioner voksne på verdensplan.
+% Standard selvkontrol af blodglukose indebærer invasive metoder som f.eks. hudprikning til udtagning af blod.
+% Mens mikronålsbaserede anordninger til kontinuerlig glukoseovervågning (CGM) har revolutioneret området, udgør forstyrrelse af hudbarrieren stadig en en afskrækkende faktor for mange diabetespatienter.
+% Ikke-invasive optiske teknikker er derfor forventes derfor at føre an i det næste teknologiske paradigmeskift inden for CGM-området.
+
+% Formålet med denne undersøgelse er at etablere en pålidelig eksperimentel platform, der skal bruges til at at undersøge interaktionen mellem terahertz-lys (THz) og menneskelig hud.
+% Denne består af in vitro hudmodeller, der repræsenterer alle de væsentlige egenskaber ved den menneskelige hud, som interagerer med THz-lys.
+% THz-fotonenergierne dækker energiområdet for hydrogenbindinger, der danner mellem vandmolekyler med hinanden og med proteiner og kulhydrater.
+% Dette funktion kan udnyttes til at registrere analytter i hudmatrixen via THz-stråling.
+
+% I denne undersøgelse afprøves tre forskellige systemer via et THz-refleksionssystem.
+% For det første anvendes dataanalyserammen valideres på flydende glukoseopløsninger.
+% Dernæst blev der anvendt høj vandkontent gelatinehydrogeler fremstilles for at simulere hudens kollagennet og de mellemliggende interstitielle væske.
+% Endelig anvendes gelatinehydrogelplader til at undersøge indtrængningsdybden af en THz-stråle i en in vitro hudmodel.
+
+% Resultaterne tyder på, at en THz-stråle kan trænge omkring 140 µm ind i en 96 \% vand
+% hydrogel og ca. 160 µm i en hydrogel med 92 \% vand.
+% Endvidere kan termisk krydsende linkede gelatinehydrogeler anses for uegnede til THz-karakterisering af deres vand indhold.
+% Til fremtidigt arbejde anbefales kemisk tværbundne gelatinehydrogeler.
+% I stedet, da de også vil være mere termisk og mekanisk stabile.
+% Dette vil gøre det muligt yderligere udvikling af hudmodellerne og øge deres kompleksitet til at omfatte flere bi- og ologiske elementer. Dette vil igen muliggøre en mere omfattende karakterisering af menneskehud i THz-regimet.
diff --git a/chapters/acknowledgments.tex b/chapters/acknowledgments.tex
@@ -1,24 +1,23 @@
\chapter*{Acknowledgments}
\addcontentsline{toc}{chapter}{Acknowledgments} % Add the ackn to the table of contents as a chapter
-\blackout
+% \blackout
{
- To S{\o}ren's group --- in particular, Marco --- for always making me feel like a part of the group, and like my contribution was valued.
+ To S{\o}ren's group --- especially Marco --- for always making me feel like a part of the group and that my contribution was valued.
- To Marco and Frederik for giving me the chance to dip my toes in ``real'' research in the rawest, realest way possible, with what I perceive to be all the highs and the lows which constitute it.
+ To Frederik and Marco for giving me the chance to dip my toes in ``real'' research in the most undiluted way possible, with what I perceive to be all the highs and the lows which constitute it.
- To S{\o}ren, for always being up for a great chat, for giving me the opportunity to work on difficult and novel problems (such as preconditioning), and for approaching everything with kindness.
+ To S{\o}ren, for always being up for a great chat, for giving me the opportunity to work on difficult and novel problems (like preconditioning), and for approaching everything with kindness.
I don't think I could have chosen a better supervisor.
- To J\'ulia, Brandon and Julia for the constant laughs and endless opportunities to relax on the couch when working was too much (memes surrounding me working on the couch aside)
- And to Adam, with who I shared the experience of writing a thesis --- highs and lows both.
+ To J\'ulia, Brandon and Julia for the constant laughs and endless opportunities to relax on the couch when working was too much (memes surrounding me working on the couch aside) and Adam, with who I shared the experience of writing a thesis --- highs and lows both.
- To Jason and Kristi for always cultivating/imparting/instilling/nurturing in me a sense of curiosity and independence
+ To Jason and Kristi for always cultivating/nurturing in me a sense of curiosity and independence.
And for never forgetting me in the supermarket.
- Lorenza
+ To Lorenza, for convincing me to go to the Hackathon, for being a great friend and for being the best person to have a chat with when I was feeling down.
- And above all else, \emph{to Steampunk}.
+ And above all else, to Steampunk.
}
\begin{flushright}
diff --git a/chapters/conclusion.tex b/chapters/conclusion.tex
@@ -2,4 +2,18 @@
\chapter{Conclusion}
\labch{conclusion}
+\section{Summary and key results}
+What results did we get? What did we learn? What did we propose?
+Remember to actually include some results and a quantitative look at the results that were obtained.
+
+How did we get these results? What methods did we use?
+
+How did our findings contribute to the field? What changes?
+
+\section{Outlook and future developments}
+
+What were our limitations?
+
+Which avenues are most promising for future research?
+Either in order to solve the problem this paper tables or to solve the next problem.
diff --git a/chapters/experiments.tex b/chapters/experiments.tex
@@ -2,7 +2,7 @@
\chapter{Experiments}
\labch{experiments}
-\section[The Advantage of Working With Hessian-Vector Products]{The Advantage of Working With Hessian-Vector Products}[Hessian-Vector Products]
+\section[Hessian-Vector Products]{Hessian-Vector Products}[Hessian-Vector Products]
In this project, we propose performing the Laplace approximation by only storing the computational graph for the implicit Hessian-vector product, thereby avoiding the explicit computation and storage of the quadratically-scaling Hessian matrix.
How significant of an effect does this have, though?
@@ -29,7 +29,7 @@ This experiment will then answer two questions:
\end{marginfigure}
We then compose the loss function \((g \circ f) (\bm x) := \sum^N_{i=1} (2 x_i + 0.5)^2\) as
-\(f(\bm x) = 2 \bm x + 0.5, g(\bm x) := \sum^N_{i=1} x_i^2\).
+\(f(\bm x) := 2 \bm x + 0.5, g(\bm x) := \sum^N_{i=1} x_i^2\).
This is equivalent to the sum-of-square-error loss \(g\) for a simple linear model \(f\).
Since \(f(x)\) is a linear function, the GGN matrix will exactly equal the actual Hessian.
We then initialise \(\bm x\) to a random vector and compute the matrix-vector product \(\bm K \bm v\) where \(\bm v\) is a random vector using three Hessian-vector products: the JAX efficient HVP and GVP functions and a manual Hessian instantiation and multiplication.
@@ -52,7 +52,7 @@ some analysis
comment on performance of manual products
comment on weird dip in inverse, possibly something to do with caching or something like that? like the size of your cache
-\begin{marginfigure}[-4cm]
+\begin{marginfigure}%[-4cm]
\centering
\includegraphics{hessian_profile_ratios.pdf}
\caption[Comparison of speedup from HVP to GVP.]{Comparison of the speedup from computation of the Hessian-vector product (HVP) versus the GGN-vector product (GVP) (top) and inverse HVP versus inverse GVP using the conjugate gradient method (bottom). Speedup is calculated as the ratio of the wall-clock time of the HVP to the GVP. It can be seen that the two methods have the same time complexity (up to a linear factor).}
diff --git a/chapters/introduction.tex b/chapters/introduction.tex
@@ -5,24 +5,27 @@
% \section{The Problem}
\section{Bayesian Deep Learning}
-Bayesian methods allow for estimates of uncertainty which enable more
-efficient usage of data (i.e., via active learning) and avoid
-overfitting.
-Furthermore, these uncertainty estimates improve model
-interpretability and assessment of model predictive confidence.
+Bayesian methods allow for estimates of uncertainty which enable more efficient usage of data (e.g., via active learning) and avoid overfitting.
+Furthermore, these uncertainty estimates improve model interpretability and assessment of model predictive confidence.
These methods have been successfully applied to a wide range of classical problems in statistics, though attempts to apply them to deep learning have had limited success.
-are particularly useful in the context of deep learning,
-where the models are often black-boxes and the data is often scarce.
+% are particularly useful in the context of deep learning,
+% where the models are often black-boxes and the data is often scarce.
+
+Typically, the Bayesian approach to deep learning is to use a prior distribution over the model parameters and then compute the posterior distribution of the model parameters given the data, where the posterior is maximised to obtain the optimal model parameters for the available data.
+This posterior distribution can then be sampled from to make predictions.
+The optimisation can, depending on the choices of likelihood and prior, often be done analytically or numerically.
+However, computing and sampling from the posterior is often difficult, and so approximate Bayesian methods are used instead.
+These approaches depend greatly on the choice of prior, which is non-trivial.
+Often, a prior is chosen which is simple and has a convenient analytical form, such as a Gaussian prior.
+However, even in this case, the choice of prior precision is still difficult, often being estimated by cross-validation.
+Instead, maximising the marginal likelihood allows for optimisation of hyperparameters in the likelihood and prior, which can be done using gradient-based optimisation algorithms.
+However, this is computationally infeasible, and so approximate Bayesian methods are used instead.
% \section{The State of Research in the Area}
\section{Current Methods}
-Currently, approximate Bayesian methods are either expensive to compute
-(Markov Chain Monte Carlo), are significantly more difficult to implement (such as variational inference), or simply perform poorly and are limited in
-their Bayesian interpretation (MC dropout).
-Gaussian processes
-Deep ensembles
-Beyond these, there are also domain-specific models which aim to provide
+Currently, approximate Bayesian methods are either expensive to compute (Markov Chain Monte Carlo), are significantly more difficult to implement (such as variational inference), or simply perform poorly and are limited in their Bayesian interpretation (MC dropout).
+% Beyond these, there are also domain-specific models which aim to provide
As such, there is demand for a method which exhibits the same computational cost as the optimization algorithms for deterministic neural networks while providing accurate posterior approximations and working out-of-the-box for any given architecture.
% Laplace approximations are simple, yet prin-
@@ -36,17 +39,15 @@ As such, there is demand for a method which exhibits the same computational cost
% storing the Hessian implicitly as means to avoid
% the fuzz.
-The Laplace approximation~\sidecite{daxberger2021laplace,laplace1774memoire,} is a simple, yet principled, posterior approximation suitable for Bayesian modeling.
+The Laplace approximation~\sidecite{daxberger2021laplace,laplace1774memoire} is a simple yet theoretically well-supported posterior approximation suitable for Bayesian modeling.
In the Laplace approximation, to compute the posterior distribution of the model parameters, we need to compute the Hessian of the loss with respect to the model parameters.
Typically, this is done by storing the Hessian matrix explicitly and computing the inverse of this matrix.
-Since the Hessian matrix is a \(D \times D\) square matrix with \(D^2\) elements, where \(D\) is the number of parameters, this is intractable for large models, which can contain millions or billions of parameters.
+Since the Hessian matrix is a \(D \times D\) square matrix with \(D^2\) elements, where \(D\) is the number of parameters, this is intractable for large models, which can contain millions or billions of parameters, thus limiting the applicability of full-rank Laplace outside of toy problems.
To overcome this, we can use a crude approximation, such as only storing the Hessian matrix's diagonal~\sidecite{lecun1989optimal,denker1990transforming}, its Kronecker factorisation~\sidecite{heskes2000natural,martens2015optimizing,botev2017practical}, and other low-rank approximations.%~\sidecite{}.
-All of these Hessian approximations have successfully been applied to the Laplace approximation~\sidecite{ritter2018scalable,ritter2018online}
+Many of these Hessian approximations have successfully been applied to the Laplace approximation~\sidecite{ritter2018scalable,ritter2018online}, though are limited by the quality of the approximation.
+To overcome this, we propose a method which performs the full-rank Laplace approximation without requiring the Hessian to be stored explicitly.
-% To address this, we propose a method which approximates the Hessian of the loss with respect to the model parameters without requiring the Hessian to be stored explicitly.
-
-% \section{My Solution}
\section{Large-Scale Laplace}
Large-Scale Laplace (LSL) is a method which approximates the Hessian of the loss with respect to the model parameters without requiring the Hessian to be stored explicitly.
@@ -73,21 +74,21 @@ JAX enables us to efficiently compute Hessian-vector products without explicitly
% JAX is designed to be a drop-in replacement for NumPy, and is compatible with most of the Python scientific stack. JAX is built on top of XLA, a domain-specific compiler for linear algebra. XLA is able to compile Python code into efficient machine code, and is used by Google to accelerate TensorFlow and PyTorch. JAX is able to compile Python code into efficient machine code, and is used by Google to accelerate TensorFlow and PyTorch.
% It is designed to be a drop-in replacement for NumPy, and is compatible with most of the Python scientific stack. JAX is built on top of XLA, a domain-specific compiler for linear algebra. XLA is able to compile Python code into efficient machine code, and is used by Google to accelerate TensorFlow and PyTorch. JAX is able to compile Python code into efficient machine code, and is used by Google to accelerate TensorFlow and PyTorch.
-\section{Research Objectives}
-
-This should be discussed in a bit more detail in a thesis, since there
-are certain objectives discussed in the project plan.
-In an article, a quick sentence which summarises the rest of the
-introduction and states succinctly exactly which problem will be approached
-and which approach will be made should suffice.
-
-In this project, we strive to develop an effortless Bayesian method.
-More specifically, this project investigates the connections between the
-Laplace approximation and the approximate second-order derivatives used
-in modern optimizers, such as Adam. We use a sampling-based training
-procedure, where a sample is first drawn from a Gaussian
-weight-posterior, a gradient step is performed on this sampled neural
-network, and the variance of the weight-posterior is updated with an
-approximate Hessian. Approximating the Hessian is the most time-consuming and painful-to-engineer step of this training procedure. In
-this project, we tap into the potential of modern machine learning
-frameworks to efficiently approximate the Hessian.
+% \section{Research Objectives}
+
+% This should be discussed in a bit more detail in a thesis, since there
+% are certain objectives discussed in the project plan.
+% In an article, a quick sentence which summarises the rest of the
+% introduction and states succinctly exactly which problem will be approached
+% and which approach will be made should suffice.
+
+% In this project, we strive to develop an effortless Bayesian method.
+% More specifically, this project investigates the connections between the
+% Laplace approximation and the approximate second-order derivatives used
+% in modern optimizers, such as Adam. We use a sampling-based training
+% procedure, where a sample is first drawn from a Gaussian
+% weight-posterior, a gradient step is performed on this sampled neural
+% network, and the variance of the weight-posterior is updated with an
+% approximate Hessian. Approximating the Hessian is the most time-consuming and painful-to-engineer step of this training procedure. In
+% this project, we tap into the potential of modern machine learning
+% frameworks to efficiently approximate the Hessian.
diff --git a/chapters/preface.tex b/chapters/preface.tex
@@ -1,15 +1,22 @@
\chapter*{Preface}
\addcontentsline{toc}{chapter}{Preface} % Add the preface to the table of contents as a chapter
-\blackout
+% \blackout
{
I've been studying for a pretty long time.
- I suppose that, because of this, there is some kind of expectation of anticipation, of the sensation that a culmination/apogee of a.
+ I suppose that, because of this, there is some sense of anticipation, of the sensation that a culmination/apogee of a.
+ Many people have had mixed experiences while writing their theses, though in general people seem to associate the process with a sense of accomplishment, since it is perceived to be the step in a long journey.
+ difficulty
I, not always being the biggest adherent to the idea of symbol-for-the-sake-of-symbol, wasn't sure what to expect of this experience
- I also took the meagre chance I saw to sneak in an en dash into the report
+ I had a slow start to the thesis, since for the first month or so I just explored the software and the literature, and tried to get a feel for the problem.
+ As I tried to generate some preliminary results, I encountered a few roadblocks.
+ As it turned out, these problems were quite fundamental to the problem --- but S{\o}ren had recently read a paper which addressed them and invited me to a Hackathon with his research group, where we would try to implement the methods from the paper.
+ Initially, I was a bit hesitant to go, since I had never been to a Hackathon before, and I was worried that I would be out of my depth and that I would not be able to contribute.
+ However, Lorenza helped me get over my fear of marginalisation and I decided to go, and I am glad that I did.
- The work in this thesis is the product of a five-month-long research project conducted in the Section for Cognitive Systems of DTU Compute, during which I helped develop some ideas in the paper ``Large-Scale Laplace''(cite).
+ This thesis is the product of a five-month-long span of work conducted in the Section for Cognitive Systems of DTU Compute, during which I helped develop some of the ideas in the ``Large-Scale Laplace'' project.
+ This project is, at the time of writing, still ongoing, and I hope that we will be able to publish the results of our work in the near future.
}
\begin{flushright}
diff --git a/chapters/sampling.tex b/chapters/sampling.tex
@@ -256,7 +256,7 @@ In CIQ, we will, in each iteration, perform one matrix-vector product \(\bm \Lam
As such, if the preconditioner-vector product is similarly or more expensive to compute compared to the matrix-vector product, then preconditioning will ....
However, an effective preconditioner will significantly improve the convergence speed of CIQ, and this effect can often outweigh the cost of computing the preconditioner.
-\subsection[Adam-esque fully linear GGN]{Adam-esque fully linear GGN}[Fully Linear GGN]
+\subsection{Fully Linear GGN}
\label{sec:adam-preconditioner}
If we calculate the GGN approximation of the Hessian by linearizing over the whole model and loss (as described in \cref{sec:practical-ggn}), we get
\begin{align}
@@ -267,14 +267,16 @@ If we calculate the GGN approximation of the Hessian by linearizing over the who
where \(\nabla\) is the gradient of the loss with regard to the model parameters.
DO I NEED TO PROVE THIS?
The problem with using this approximation is that the rank of \(\bm P\) is 1.
-Best-case scenario, we can reduce the highest eigenvalue of the Hessian.
+This means that, best-case scenario, we can reduce the highest eigenvalue of the Hessian.
This occurs when the gradient corresponds to the direction of the highest precision (empirically, it seems that \emph{this is not the case}).
This would then leave the second-highest eigenvalue unaffected.
If the rank of our precision matrix is greater than one, then this value seems likely to be similarly large to originally the largest eigenvalue.
Therefore, the conditioning number of our problem will not be significantly reduced.
As such, this preconditioner is not worth pursuing.
-\subsection{Sub-sampling the data}
+like adam
+
+\subsection{Sub-Sampling the Data}
We could just choose to calculate the Hessian over fewer points and invert it using the conjugate gradient method to find our preconditioner.
This would have the advantage of
\begin{align}
@@ -365,7 +367,7 @@ Furthermore, you want enough observations \(B\) to be included in the preconditi
Similarly to the pivoted Cholesky preconditioner, the randomly pivoted Cholesky preconditioner requires the diagonal of the GGN matrix in order for the decomposition to be close to the GGN.
We can thus approximate this term as we do in the pivoted Cholesky preconditioner (see \cref{sec:pivoted-cholesky}).
-\subsection[Other Laplace approximations]{Other Laplace approximations}[Other approximations]
+\subsection[Other Laplace Approximations]{Other Laplace Approximations}[Other approximations]
diff --git a/chapters/theory.tex b/chapters/theory.tex
@@ -7,7 +7,7 @@
\label{sec:optimisation-frequentist}
Suppose a neural network is a real-valued function \(f: \reals^n \times \reals^d \rightarrow \reals^o\) parameterised in \(\bm\theta\) which maps an input \(\bm x\) to an output \(f(\bm x; \bm\theta) \equiv f_{\bm\theta}(\bm x)\).
-Our goal is to find the optimal parameters \(\bm\theta^*\) which best model the observed data.
+Our goal is to find the parameters \(\bm\theta^*\) which best model the observed data.
From a frequentist perspective, we can define a loss function \(\mathcal{L}(\bm\theta): \reals^d \rightarrow \reals\) such that the optimal parameters \(\bm\theta^*\) minimise this loss function.
Often, we formulate the loss function as the negative log-likelihood of the data under the model, \(\mathcal{L}(\bm\theta) = -\log \lik\).
@@ -49,9 +49,10 @@ It can either be constant throughout training (\(\eta_t = \eta\)) or adaptive (i
\subsection[Natural Gradient Descent]{Natural Gradient Descent}[The Natural Gradient]
% https://agustinus.kristia.de/techblog/2018/03/14/natural-gradient/
-Natural Gradient Descent is an approximate second-order optimisation method. It has an interpretation as optimizing over a Riemannian manifold using an intrinsic distance metric, which implies the updates are invariant to transformations such as whitening. By using the positive semi-definite (PSD) Gauss-Newton matrix to approximate the (possibly negative definite) Hessian, NGD can often work better than exact second-order methods.
+Natural Gradient Descent is an approximate second-order optimisation method. It has an interpretation as optimizing over a Riemannian manifold using an intrinsic distance metric, which implies the updates are invariant to transformations such as whitening.
+By using the positive semi-definite (PSD) Gauss-Newton matrix to approximate the (possibly negative definite) Hessian, NGD can often work better than exact second-order methods.
-side{amari1998natural} introduced the natural gradient as a way to optimize a function \(f\) parameterised by \(\bm\theta\) by following the direction of the steepest descent in the Fisher information metric.
+\sidetextcite{rattray1998natural} introduced the natural gradient as a way to optimize a function \(f\) parameterised by \(\bm\theta\) by following the direction of the steepest descent in the Fisher information metric.
It can be interpreted as
% The algorithm can be seen in \cref{natural-gradient}.
%
diff --git a/chapters/training.tex b/chapters/training.tex
@@ -5,6 +5,11 @@
\section{Maximising the Evidence}
\label{sec:maximising-the-evidence}
+WHY DO WE WANT TO MAXIMISE THE EVIDENCE?
+(1). OVERFIT LESS (SHOULD FIND FLAT MINIMA)
+(2). OPTMISE HYPERPARAMETERS
+(3). PERFORM MODEL SELECTION
+
As seen in \cref{eq:bayes-theorem}, the posterior probability for a given model is equal to
\begin{align}
p(\bm \theta \given \bm x) = \frac{p(\bm x \given \bm \theta) p(\bm \theta)}{p(\bm x)},
@@ -14,6 +19,18 @@ where \(p(\bm x)\) is known as the evidence. It's also known as the \emph{margin
p(\bm x) ={} & \int_{\bm \theta} p(\bm \theta \given \bm x)\,d\bm \theta.
\end{align}
This term is typically intractable, since it involves integrating over the entire parameter space.
+For this reason, we typically approximate it using a variational approximation, such as the Laplace approximation, or by using a Monte Carlo approximation, such as the Markov chain Monte Carlo (MCMC) method. % autogenerated
+
+However, since we're integrating over the posterior, maximising the marginal likelihood will seek minima that are flat, since the posterior will be very broad in these regions.
+This is a desirable property, since it means that the model will be less likely to overfit the data, both from a theoretical standpoint~\sidecite{hochreiter1997flat} and in practice~\sidecite{keskar2016large,jiang2019fantastic,maddox2019simple}.
+
+Furthermore, \sidetextcite{fong2020marginal} show that maximising the marginal (and consequently the log-marginal) is equivalent to performing k-fold cross validation for all values \(k = 1, \ldots, \infty\) and choosing the model with the highest average posterior probability (across each of \(k\) folds and across all values of \(k\)).
+% This is a very strong result, since it shows that the marginal likelihood is a very good proxy for the true cross-validation performance of a model. autogenerated
+Because of this, it can be used for model selection, since it will choose the model that performs best on average across all permutations of the data.
+
+% Finally, the marginal likelihood can be used to optimise hyperparameters, since it can be used to compute the gradient of the log-marginal with respect to the hyperparameters.
+% This is useful, since it allows us to use gradient-based optimisation methods to find the hyperparameters that maximise the marginal likelihood.
+
For the Laplace approximation, we approximate the posterior as \(p(\bm \theta \given \bm x) \approx \normal(\bm \theta \given \bm \theta_{\textsc{map}}, \bm \Lambda^{-1})\), where the posterior precision \(\bm \Lambda = -\nabla^2_{\bm\theta} \mathcal L(\bm \theta_{\textsc{map}})\) is the Hessian of the negative log-posterior at the maximum a posteriori (MAP) estimate \(\bm \theta_{\textsc{map}}\).
We thus obtain the approximate Laplace posterior
\begin{align}\label{eq:laplace-posterior}
@@ -46,7 +63,7 @@ and
\end{align}
where \(\alpha\) is the precision of the prior and \(\rho\) is the precision of the likelihood.
-Combining \cref{eq:laplace-log-marginal}, \cref{eq:normal-likelihood}, and \cref{eq:normal-prior}, we obtain
+Combining \cref{eq:laplace-log-marginal,eq:normal-likelihood,eq:normal-prior}, we obtain
\begin{align}\label{eq:expanded-log-marginal}
\log p(\bm x) \stackrel{\textsc{la}}{\approx}{} & - \frac{N O}{2} \log(2 \pi) + \frac{N O}{2} \log \rho - \frac{1}{2} \rho \sum_{i=1}^N \norm{\bm y_i - f_{\bm \theta}(\bm x_i)}^2
\\ & - \frac{D}{2} \log(2 \pi) + \frac{D}{2} \log \alpha - \frac{1}{2} \alpha \norm{\bm \theta}^2 \nonumber
@@ -69,8 +86,6 @@ While these terms do not affect the optimisation problem, they are useful for co
\\ 3. \(\alpha / 2 \norm{\bm \theta}^2\)
\end{tabular}
-\sidetextcite{fong2020marginal} show that maximising the marginal (and consequently the log-marginal) is equivalent to performing k-fold cross validation for all values \(k = 1, \ldots, \infty\) and choosing the model with the highest average posterior probability (across each of \(k\) folds and across all values of \(k\)).
-
\section[The Determinant Lower Bound]{The Determinant Lower Bound}[The Lower Bound]
We compute an \emph{upper} bound on the log-determinant of the Hessian of the negative log-posterior \(\bm \Lambda \in \reals^{D \times D}\), i.e.,
@@ -96,12 +111,6 @@ Since \(\bm \Lambda\) is a low-rank matrix given by \(\sum_{i=1}^N \bm J_i\T \bm
\end{align}
where \(\lambda_1, \ldots, \lambda_O\) are the eigenvalues of \(\bm \Lambda\).
-Furthermore, the trace of
-\begin{align}
- \Tr(\bm \Lambda) ={} & \Tr({\textstyle{\sum_{i=1}^N}} \bm J_i\T \bm H_i \bm J_i) = \sum_{i=1}^N \Tr(\bm J_i\T \bm H_i \bm J_i)
- \\ ={}& \sum_{i=1}^N \Tr(\bm H_i \bm J_i \bm J_i\T)
-\end{align}
-
Since \(B_D(\mu_1, \mu_2, \beta)\) is an upper bound on the log-determinant of \(\bm \Lambda\), we can use this to compute a lower bound on the log-marginal likelihood, as per \cref{eq:expanded-log-marginal}.
This is effective because we want to maximise the log-marginal.
@@ -111,10 +120,34 @@ We then need to compute the trace of the Hessian matrix, as per \cref{eq:mu1-mu2
\subsection[Hutchinson's Trace Estimator]{Hutchinson's Trace Estimator}[The Trace Estimator]
+Let \(\bm \varepsilon_0 \sim \normal(\bm 0, \identity)\).
+Since \(\E{}{\bm \varepsilon_0 \bm \varepsilon_0\T} = \identity\), we compute the trace of the precision matrix as
+\begin{align}
+ \mu_1 ={} & \Tr(\bm \Lambda) = \Tr\left(\E{}{\bm \varepsilon_0 \bm \varepsilon_0\T} \bm \Lambda\right)
+ \\ ={}& \E{}{\Tr(\bm \varepsilon_0 \bm \varepsilon_0\T \bm \Lambda)} = \E{}{\Tr{\bm \varepsilon_0\T \bm \Lambda \bm \varepsilon_0}}
+ \\ ={}& \E{}{\bm \varepsilon_0\T \left( {\textstyle \sum_{i=1}^N} \bm J_i\T \bm H_i \bm J_i \right) \bm \varepsilon_0} + D \alpha.
+\end{align}
+Similarly, we compute the trace of the square precision matrix as
+\begin{align}
+ \Tr(\bm \Lambda^2) ={} & \Tr\left(\left( {\textstyle {\textstyle \sum_{i=1}^N}} \bm J_i\T \bm H_i \bm J_i \right)^2\right) + 2 \alpha \Tr\left( {\textstyle \sum_{i=1}^N} \bm J_i\T \bm H_i \bm J_i \right) + D \alpha^2
+ \\ ={} & \E{}{\bm \varepsilon_0\T \left( {\textstyle \sum_{i=1}^N} \bm J_i\T \bm H_i \bm J_i \right) \left( {\textstyle \sum_{i=1}^N} \bm J_i\T \bm H_i \bm J_i \right)\T \bm \varepsilon_0} + D \alpha.
+\end{align}
+Fortunately, this means that we only need to compute one GGN-vector product
+% \begin{align}
+
+% \end{align}
+
+Furthermore, the trace of
+\begin{align}
+ \Tr(\bm \Lambda) ={} & \Tr({\textstyle{\sum_{i=1}^N}} \bm J_i\T \bm H_i \bm J_i) = \sum_{i=1}^N \Tr(\bm J_i\T \bm H_i \bm J_i)
+ \\ ={}& \sum_{i=1}^N \Tr(\bm H_i \bm J_i \bm J_i\T)
+\end{align}
+
\subsection{Scaling}
\subsection{Computational Cost}
+Estimating the trace and square trace of the Hessian matrix is the most computationally expensive part of estimating the log-marginal under the Laplace approximation, since this involves computing the GGN-vector product.
diff --git a/main.bib b/main.bib
@@ -93,6 +93,17 @@
publisher = {MIT Press}
}
+@article{hochreiter1997flat,
+ title = {Flat minima},
+ author = {Hochreiter, Sepp and Schmidhuber, J{\"u}rgen},
+ journal = {Neural computation},
+ volume = {9},
+ number = {1},
+ pages = {1--42},
+ year = {1997},
+ publisher = {MIT Press One Rogers Street, Cambridge, MA 02142-1209, USA journals-info~…}
+}
+
@software{jax2018github,
author = {James Bradbury and Roy Frostig and Peter Hawkins and Matthew James Johnson and Chris Leary and Dougal Maclaurin and George Necula and Adam Paszke and Jake Vander{P}las and Skye Wanderman-{M}ilne and Qiao Zhang},
title = {{JAX}: composable transformations of {P}ython+{N}um{P}y programs},
@@ -101,6 +112,20 @@
year = {2018}
}
+@inproceedings{jiang2019fantastic,
+ title = {Fantastic Generalization Measures and Where to Find Them},
+ author = {Yiding Jiang and Behnam Neyshabur and Hossein Mobahi and Dilip Krishnan and Samy Bengio},
+ booktitle = iclr,
+ year = {2020}
+}
+
+@article{keskar2016large,
+ title = {On large-batch training for deep learning: Generalization gap and sharp minima},
+ author = {Keskar, Nitish Shirish and Mudigere, Dheevatsa and Nocedal, Jorge and Smelyanskiy, Mikhail and Tang, Ping Tak Peter},
+ journal = {arXiv preprint arXiv:1609.04836},
+ year = {2016}
+}
+
@inproceedings{kingma2014adam,
title = {Adam: A method for stochastic optimization},
author = {Kingma, Diederik P and Ba, Jimmy},
@@ -138,6 +163,14 @@
publisher = {Cambridge University Press}
}
+@article{maddox2019simple,
+ title = {A simple baseline for {B}ayesian uncertainty in deep learning},
+ author = {Maddox, Wesley J and Izmailov, Pavel and Garipov, Timur and Vetrov, Dmitry P and Wilson, Andrew Gordon},
+ journal = nips,
+ volume = {32},
+ year = {2019}
+}
+
@inproceedings{martens2015optimizing,
title = {Optimizing neural networks with {K}ronecker-factored approximate curvature},
author = {Martens, James and Grosse, Roger},
@@ -183,10 +216,21 @@
year = {2020}
}
+@article{rattray1998natural,
+ title = {Natural gradient descent for on-line learning},
+ author = {Rattray, Magnus and Saad, David and Amari, Shun-ichi},
+ journal = {Physical review letters},
+ volume = {81},
+ number = {24},
+ pages = {5461},
+ year = {1998},
+ publisher = {APS}
+}
+
@article{ritter2018online,
title = {Online structured laplace approximations for overcoming catastrophic forgetting},
author = {Ritter, Hippolyt and Botev, Aleksandar and Barber, David},
- journal = {Advances in Neural Information Processing Systems},
+ journal = nips,
volume = {31},
year = {2018}
}
