masters-thesis

abstract.tex (6703B)

---

 \chapter*{Abstract}
 \addcontentsline{toc}{chapter}{Abstract}
 
 Bayesian methods promise to provide a principled way to quantify uncertainty in neural networks.
 This is important for many applications in machine learning, such as those involving safety-critical decisions that rely on risk assessment and interpretability.
 However, Bayesian inference is often computationally intractable, so approximate methods which compromise performance for practicality are used.
 The Laplace approximation (LA) is one such method, though it itself typically relies on crude approximations to the posterior precision matrix to make it computationally feasible.
 When we do not factorise the posterior precision into more practical approximations, existing methods require that the full posterior precision be instantiated.
 We call this the full Laplace approximation.
 For all but toy problems, however, since the full precision matrix scales quadratically with the number of parameters, this matrix is too large to be stored in memory.
 
 In this work, we propose a method for computing the Laplace approximation using only Jacobian-vector products.
 This allows us to perform marginal training and posterior sampling using the full Laplace approximation without storing the entire posterior precision matrix.
 To accomplish this, we show that we can estimate the posterior precision's log-determinant and inverse square root using only Jacobian-vector products.
 To overcome the conditioning issues that arise during sampling, we offer several potential preconditioners which can be used to improve convergence of the inverse square root approximation.
 Next, we give a technique for evaluating the quality of the posterior samples without instantiating nor inverting the posterior precision based on results from traditional statistics.
 To implement these methods, we use JAX, a library for automatic differentiation which enables us to efficiently perform Jacobian-vector products without explicitly storing the entire posterior precision.
 
 We perform the full Laplace approximation on both a sine function and MNIST using our method.
 This consists of two steps: (a) training a neural network by maximising either its posterior probability or marginal likelihood and (b) sampling from this posterior.
 Our approximate maximum marginal training procedure is able to learn a set of parameters which yields performance comparable to that of the maximum posterior and maximum likelihood training procedures.
 By analysing quantile--quantile plots of our posterior samples and visualising these samples, we find that our approximate sampling method produces samples which are correctly distributed.
 Our benchmarks of the performance of Jacobian-vector products estimate that our method yields a 10\,000x memory reduction over the conventional full Laplace approximation, since it does not store the quadratically-scaling full posterior precision matrix.
 This is the first method for performing the Laplace approximation while only accessing the posterior precision implicitly.
 
 \chapter*{Resum\'e}
 \addcontentsline{toc}{chapter}{Resum\'e}
 
 Bayesianske metoder lover at give en principiel m{\aa}de at kvantificere usikkerheden i neurale netv{\ae}rk p{\aa}.
 Dette er vigtigt for mange anvendelser inden for machine learning, f.eks. i forbindelse med sikkerhedskritiske beslutninger, der er afh{\ae}ngige af risikovurdering og fortolkningsmuligheder.
 Bayesiansk inferens er imidlertid ofte beregningsm{\ae}ssigt uh{\aa}ndterbar, s{\aa} der anvendes tiln{\ae}rmede metoder, som g{\aa}r p{\aa} kompromis med ydeevnen for at opn{\aa} praktisk anvendelighed.
 Laplace-approksimationen (LA) er en s{\aa}dan metode, selv om den typisk er afh{\ae}ngig af grove tiln{\ae}rmelser af den efterf{\o}lgende pr{\ae}cisionsmatrix for at g{\o}re den beregningsm{\ae}ssigt gennemf{\o}rlig.
 N{\aa}r vi ikke faktoriserer den efterf{\o}lgende pr{\ae}cision i mere praktiske tiln{\ae}rmelser, kr{\ae}ver de eksisterende metoder, at den fulde efterf{\o}lgende pr{\ae}cision skal instantieres.
 Vi kalder dette den fulde Laplace-approksimation.
 Da den fulde pr{\ae}cisionsmatrix imidlertid for alle problemer undtagen leget{\o}jsproblemer skalerer kvadratisk med antallet af parametre, er denne matrix for stor til at blive lagret i hukommelsen.
 
 I dette studie foresl{\aa}r vi en metode til beregning af Laplace-approksimationen ved hj{\ae}lp af Jacobian-vektorprodukter alene.
 Dette giver os mulighed for at udf{\o}re marginal tr{\ae}ning og efterf{\o}lgende stikpr{\o}veudtagning ved hj{\ae}lp af den fulde Laplace-approksimation uden at lagre hele den efterf{\o}lgende pr{\ae}cisionsmatrix.
 For at opn{\aa} dette viser vi, at vi kan estimere den posteriore pr{\ae}cisionens log-determinant og den inverse kvadratrod ved hj{\ae}lp af kun jacobianiske vektorprodukter.
 For at overvinde de konditioneringsproblemer, der opst{\aa}r under sampling, tilbyder vi flere potentielle pr{\ae}konditioneringsv{\ae}rkt{\o}jer, som kan anvendes til at forbedre konvergensen af den inverse kvadratrodsapproksimation.
 Dern{\ae}st giver vi en teknik til at evaluere kvaliteten af de efterf{\o}lgende stikpr{\o}ver uden at instantiere eller invertere den efterf{\o}lgende pr{\ae}cision baseret p{\aa} resultater fra traditionel statistik.
 Til at gennemf{\o}re disse metoder anvender vi JAX, et bibliotek til automatisk differentiering, som g{\o}r det muligt at udf{\o}re jacobian-vektorprodukter effektivt uden eksplicit lagring af hele den efterf{\o}lgende pr{\ae}cision.
 
 Vi udf{\o}rer den fulde Laplace-approksimation p{\aa} b{\aa}de en sinusfunktion og MNIST ved hj{\ae}lp af vores metode.
 Denne best{\aa}r af to trin: (a) tr{\ae}ning af et neuralt netv{\ae}rk ved at maksimere enten dets posterior sandsynlighed eller marginale sandsynlighed og (b) pr{\o}veudtagning fra denne posterior.
 Vores tiln{\ae}rmede maksimale marginale tr{\ae}ningsprocedure er i stand til at l{\ae}re et s{\ae}t parametre, som giver en ydeevne, der er sammenlignelig med den maksimale posterior- og maksimale sandsynlighedstr{\ae}ningsprocedure.
 Ved at analysere quantile--quantile-plots af vores posteriorpr{\o}ver og visualisere disse pr{\o}ver finder vi, at vores tiln{\ae}rmede pr{\o}veudtagningsmetode giver pr{\o}ver, der er korrekt fordelt.
 Vores benchmarks af Jacobian-vektorprodukternes ydeevne ansl{\aa}r, at vores metode giver en 10\,000x hukommelsesreduktion i forhold til den konventionelle fulde Laplace-approksimation, da den ikke lagrer den kvadratisk skalerende matrix med fuld pr{\ae}cision for den efterf{\o}lgende periode.
 Dette er den f{\o}rste metode til at udf{\o}re Laplace-approksimationen, hvor der kun er implicit adgang til den efterf{\o}lgende pr{\ae}cision.