Oregon State University

Corvallis, OR 97331

- Recommended Textbook: Deep Learning. Ian Goodfellow. Yoshua Bengio, Aaron Courville (referred to as Deep Learning Book). MIT Press. 2016
- Week 1 (Intro + ML/Optimization Refresher)
- Week 2 (Basic Neural Networks)
- Week 3 (Neural Networks Optimization 1 and Theoretical Implications)
- Week 4 (Convolutional Neural Networks)
- Week 5 (Other Deep Architectures and Recurrent Neural Models)
- Week 6 (Recurrent Neural Models)
- Week 7 (Neural Network Regularization)
- Week 8 (Neural Network Optimization)
- Week 9
- Week 10

Monday: Introduction to Deep Learning

About the Class

Wednesday: Introduction to Deep Learning + Optimization Primer 1

Reading: Chapter 2, The Elements of Statistical Learning. Hastie, Friedman, Tibshirani.

Chapter 5, Deep Learning Book.

Wednesday: Basic Feedforward Network

Reading: Chapter 4 and Chapter 6, Deep Learning Book

Monday: Neural Networks Optimization 1 (Stochastic mini-batch gradient descent, momentum, early stopping and weight decay)

Wednesday: Theoretical Implications + Convolutional Neural Networks

Suggested readings: Sections 8.1 - 8.3 and Chapter 9, Deep Learning Book

VGG Convolutional Networks Practical, great hands-on tutorial of CNN.

Best Practices for Convolutional Neural Networks Applied to Visual Document Analysis P. Simard, D. Steinkraus and J.C. Platt. ICDAR 2003. About elastic deformation.

Chapter 5, Pattern Recognition and Machine Learning. Christopher Bishop. Springer, 2006.

Monday: Convolutional Neural Networks

Wednesday: Convolutional Neural Networks (cont.) (Visualization of Convolutional Networks)

Suggested readings: Chapter 9, Deep Learning Book

K Chatfield, K Simonyan, A Vedaldi, A Zisserman. Return of the devil in the details: Delving deep into convolutional nets.
arXiv preprint arXiv:1405.3531

K. Simonyan, A. Zisserman. Very Deep Convolutional Networks for Large-Scale Image Recognition. ICLR 2015

Zeiler and Fergus. Visualizing and Understanding Convolutional Networks. ECCV 2014

K. Simonyan, A. Vedaldi and A. Zisserman. Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps, ICLR 2014.

J. Zhang, J. Lin, J. Brandt, X. Shen, S. Sclaroff. Top-Down Neural Attention by Excitation Backprop. ECCV 2016.

Monday: Quiz + Deep Learning Toolboxes

Wednesday: Deconvolution and Other Vision Problems + A brief Introduction to Other Neural Architectures

Suggested readings: I. Goodfellow. NIPS 2016 Tutorial: Generative Adversarial Networks. arXiv:1701:00160

I. Gulrajani, et al. Improved Training of Wasserstein GANs. NIPS 2017

Monday: Recurrent Neural Models

Wednesday: Project Proposal Presentations

Suggested readings: K. Greff et al. LSTM: A Search Space Odyssey . IEEE Transactions on Neural Networks and Learning Systems. 2017, 28(10):2222-2232.

Monday: Project Proposal Presentations + Recurrent Neural Networks (cont.)

Wednesday: Recurrent Neural Networks (cont.) + Neural Network Regularization

Suggested readings: A. Graves. Generating Sequences with Recurrent Neural Networks . arXiv:1308.0850.

Vinyals, Oriol, et al. Show and tell: A neural image caption generator. CVPR 2015.

Monday: Neural Network Regularization

Wednesday: Quiz + LSTM Training + Neural Network Optimization #2

Suggested readings: Sergey Ioffe, Christian Szegedy. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift ICML 2015.

Diederik Kingma, Jimmy Ba. Adam: A Method for Stochastic Optimization ICLR 2015.

Monday: ResNet and New Architectures

Wednesday: Deep Learning in NLP (Guest lecture from Dr. Liang Huang)

Suggested readings:
Slides for ResNet

ResNet paper

Slides for DenseNet

DenseNet Paper

Monday: Unsupervised Deep Learning

Wednesday: Deep Reinforcement Learning (Guest lecture from Dr. Alan Fern)