masters-thesis

README.md (2870B)

---

 Effortless Bayesian Deep Learning: Tapping into the Potential of Modern Optimizers
 ==============================
 
 Bayesian methods allow for estimates of uncertainty which enable more efficient usage of data (i.e., via active learning) and avoid overfitting.
 Furthermore, these uncertainty estimates improve model interpretability and assessment of model predictive confidence.
 
 Currently, approximate Bayesian methods are either expensive to compute (MCMC), are significantly more difficult to implement (such as variational inference), or simply perform poorly and are limited in their Bayesian interpretation (MC dropout).
 As such, there is demand for a method which exhibits the same computational cost as the optimization algorithms for deterministic neural networks while providing accurate posterior approximations and working out-of-the-box for any given architecture.
 
 In this project, we strive to develop an effortless Bayesian method.
 More specifically, this project investigates the connections between the Laplace approximation and the approximate second-order derivatives used in modern optimizers, such as Adam.
 We use a sampling-based training procedure, where a sample is first drawn from a Gaussian weight-posterior, a gradient step is performed on this sampled neural network, and the variance of the weight-posterior is updated with an approximate Hessian.
 Approximating the Hessian is the most time-consuming and painful-to-engineer step of this training procedure.
 In this project, we tap into the potential of modern machine learning frameworks to efficiently approximate the Hessian.
 
 # Setup
 
 This document is designed to be generated with `latexmk` and cleaned with `latexmk -C`.
 You can use either `pdflatex` or `xelatex` to compile the document, though I've used `pdflatex` throughout the process and can't guarantee that `xelatex` will work out of the box.
 If you want to use `xelatex`, you should update the `latexmkrc` file to use `xelatex` instead of `pdflatex`.
 I highly recommend using VSCode with the LaTeX Workshop extension to compile the document.
 I also lightly recommend using the LTeX extension and the stylecheck script to check for common errors, though this is more out of personal preference.
 If using the LaTeX Workshop extension, I would add this to your `settings.json` file:
 
 ```json
 {
     "latex-workshop.hover.preview.mathjax.extensions": ["boldsymbol"],
     "latex-workshop.hover.preview.newcommand.newcommandFile": "mathjax.tex",
     "latex-workshop.hover.preview.newcommand.parseTeXFile.enabled": true,
     "latex-workshop.latex.clean.method": "cleanCommand",
     "latex-workshop.latex.recipe.default": "latexmk (latexmkrc)",
     "latex-workshop.latex.tools": [
         {
             "name": "latexmk_rconly",
             "command": "latexmk",
             "args": [],
             "env": {}
         },
     ],
 }
 ```