TCOM5970/ECE5973: Deep Learning
In a wide sense, deep learning is a study of machine learning models composed of multiple layers of representation (hence deep). In the last decade, there has been significant progress of using artificial neural networks for deep models. This leads to numerous breakthroughs on image, video, and natural language processing applications.
This course is aimed to be quite handson and should provide students with sufficient details for them to quickly apply to their own research. In particular, applications relating to computer vision and communications will be discussed.
There may be some math but we probably won't spend much time going into proofs. Instead, we may try to go through (not exhasively) some of the free libraries such as Caffe and Torch. And you are definitely encouraged to explore and leverage them for your course project.
N.B. Because we will “borrow” programming assignment from Stanford 231n, Python prerequiste is needed.
Textbook
Ian Goodfellow, Yoshua Bengio, and Aaron Courville, Deep Learning, MIT Press.
It is not required but is a very good reference.
Reference
Some Deep Learning Toolboxes and Libraries
Tensorflow: From Google, probably most popular package. Not quite optimized for single PC
Caffe2: From Facebook
Caffe: From Berkeley
Torch 7: From NYU, and used by Facebook/Twitter
Theano: From Bengio's group in Montreal
Keras: Highlevel layer on top of Theano/Tensorflow
Lasagne: Highlevel layer on top of Theano
matconvnet: From Oxford, kind of restricted
mxnet: From Amazon
Neon: From Intel
Deeplearning4j
Office Hours
There are no “regular” office hours. And you are welcome to come catch me anytime or contact me through emails.
Course Syllabus (Tentative)
Overview of machine learning
History of artificial neural networks
Perceptrons
Backpropagation algorithms
Regularization and dropout
Weight initialization
Optimization methods
Convolutional neural networks (CNN)
RCNN, faster RCNN
Weight visualization, Deep visualization, tSNE, deepdream
Recurrent neural networks
LSTM networks
Restricted boltzmann machines
Autoencoders
Deep belief networks
Projects
Your projects will be presented in class on May 5. You may include an additional written report if you would like to further clarify anything.
Grading
"Activities": 30%. Quizzes, paper review, presentations, etc.
Homework: 30%. Programming assignments.
Final Project: 40%.
Calendar
 Topics  Materials 
1/20  Overview, AI, machine learning and its types, artificial neural networks and its history  slides, Example: image classification 
1/27  Machine learning basics, linear regression, regularization, ridge regression, lasso, logistic regression, softmax classifier, loss function, optimization basics  slides, linear regression notebook 
2/2  Backpropagation, activation functions, weight initialization, dropout, batch normalization, more on optimization tricks, debugging optimizers, hyperparameter optimization  slides, optimizer comparison notebook 
2/9  Convolutional neural networks (CNNs)  slides 
2/16  Object localization and detection, CNNs and arts, fooling CNNs  slides 
2/23  Recurrent neural networks (RNNs), long shortterm memory (LSTM) networks  slides 
3/3  Echo state networks (ESNs), image captioning  slides, project explanation, pyESN: Python implementation of ESN, neuraltalk2 
3/10  NLP tasks, name entity recognition, recursive neural networks  slides 
3/23  Seq2seq models, neural machine translations, chatbots  slides 
3/30  Memory networks  slides 
4/7  Neural Turing machines  slides 
4/14  Restricted Boltzmann machines, deep belief networks  slides, a practical guide of training RBM 
4/21  Autoencoders, generative adversarial networks  slides 
4/28  Deep reinforcement learning  slides, epilogue, AlphaGo Zero

