We present a deep generative model of bilingual sentence pairs. The model generates source and target sentences jointly from a shared latent representation and is parameterised by neural networks. Efficient training is done by amortised variational inference and reparameterised gradients. Additionally, we discuss the statistical implications of joint modelling and propose an efficient approximation to maximum a posteriori decoding for fast test-time predictions. We demonstrate the effectiveness of our model in three scenarios: in-domain training, mixed-domain training, and learning from a mix of gold-standard and synthetic data. Our experiments show consistently that our joint formulation outperforms conditional modelling in all such scenarios.

NLP has seen a surge in neural network models in recent years. These models provide state-of-the-art performance on many supervised tasks. Unsupervised and semi-supervised learning has only been addressed scarcely, however. Deep Generative Models (DGMs) make it possible to integrate neural networks with probabilistic graphical models. Using DGMs one can easily design latent variable models that account for missing observations and thereby enable unsupervised and semi-supervised learning with neural networks. The method of choice for training these models is variational inference. This tutorial offers a general introduction to variational inference followed by a thorough and example-driven discussion of how to use variational methods for training DGMs. It provides both the mathematical background necessary for deriving the learning algorithms as well as practical implementation guidelines. Moreover, we discuss common pitfalls that one may encounter when using DGMs for NLP applications, such as the latent variable being ignored by the model, and discuss potential solutions from a theoretical and practical perspective. Importantly, the tutorial will cover models with continuous and discrete variables.

This tutorial is scheduled to be presented at

• ACL 2018 in Melbourne: July 2018
• University of Amsterdam: March 22-23 2018

and it has already been presented at

• Monash University on November 16 2017
• University of Melbourne on October 31 and November 2 2017
• Amazon (Berlin) on July 26-27 2017

This is a shortened version of a tutorial on variational inference and deep generative models that I have developed with Philip Schulz and which we will be presenting at ACL2018. For this occasion, I had to leave the part about discrete variables out, but perhaps the slides I would have used can still be useful.

Invited talk at Naver Labs (France). The original talk was divide into two session and I also presented models that I have developed with my students and collaborators.

Deep generative models (DGMs) are probabilistic models parametrised by neural networks (NNs). DGMs combine the power of NNs with the generality of the probabilistic learning framework allowing a modeller to be more explicit about her statistical assumptions. To unlock this power however one must consider efficient ways to approach probabilistic inference. Variational inference surfaced as the method of choice, howerver, efficient and effective VI for DGMs require low-variance gradient estimation for stochastic computation graphs (Kingma and Welling, 2013; Rezende et al, 2014; Titsias and Lazaro-Gredilla, 2014). In this talk I will present an overview of deep generative modelling, amortised variational inference, and the mathematics behind low-variance reparameterised gradients.

In this talk I will outline a few deep generative models applied to induction of word alignments. I propose a fully probabilistic account to unsupervised problems such as word embedding, alignment and segmentation. This is in contrast with deterministic representations obtained as a byproduct of training neural networks as fully supervised classifiers. If you care about NLP applications such as machine translation and if you are interested in the synergy between probabilistic graphical modelling and neural networks, then do come to my talk.

Invited lecture for NLP1 at UvA. I cover basic notions about directed graphical models and present IBM1 as an application of latent variable modelling. I also present maximum likelihood estimation via expectation maximisation for categorical distributions.

Guest lecture for Jelle Zuidema’s course on Unsupervised Language Learning at UvA. In this one hour talk, I present sampling methods and their role in performing MLE and/or posterior inference with special attention to the case of parsing.

Lecture at the Ninth Machine Translation Marathon in Trento.

In this one hour talk, I give an overview of decoding for statistical machine translation.

In this 3h tutorial, I will present word-alignment, phrase-based and hierarchical phrase-based statistical machine translation. The tutorial will also include a factored-based and syntax-based models.