# Stochastic approximation

- A Stochastic Approximation Method
- Acceleration of Stochastic Optimization by Averaging
- Introduction to Stochastic Approximation Algorithms
- Stochastic Optimization (Chapter 6 of Handbook of Computational Statistics)
- Slides by Li and Rowland

Batch size vs learning rate

# Bayesian optimisation

Tutorials and Workshops

- A Taxonomy of Global Optimization Methods Based on Response Surfaces
- A Tutorial on Bayesian Optimization of Expensive Cost Functions, with Application to Active User Modeling and Hierarchical Reinforcement Learning
- BayesOpt

Theses and papers

- Bayesian Gaussian Processes for Sequential Prediction, Optimisation and Quadrature
- Practical Bayesian Optimization of Machine Learning Algorithms
- Bayesian Optimization of Text Representations
- Bayesian Optimisation for Machine Translation
- Speed-Constrained Tuning for Statistical Machine Translation Using Bayesian Optimization

Code

# Evolutionary algorithms

- Simple Evolutionary Optimization Can Rival Stochastic Gradient Descent in Neural Networks
- Evolution Strategies as a Scalable Alternative to Reinforcement Learning
- Connection to VI
- NEAT
- Deep learning using genetic algorithms

# Misc

Papers

- Algorithms for Hyper-Parameter Optimization
- Learning to Learn without Gradient Descent by Gradient Descent
- Learning to learn by gradient descent by gradient descent

Code