CSCI 8955: Advanced Data Analytics: Statistical Learning and Optimization

Overview: The last decade has witnessed an increasing amount of data generated by modern applications, such as daily photos, videos, and medical image scans. There is a great need for techniques to learn from this data. In this course, we will discuss advanced topics in data analysis, with an emphasis on statistical learning and related optimization problems. In particular, we will cover learning methods (e.g., image regression, dictionary learning, random forests, and deep learning), numerical optimization approaches (e.g., the adjoint method, coordinate descent, and stochastic gradient descent), and their connections. The applications include prediction, classification, segmentation, and other tasks in image analysis. This course is targeted towards graduate students, and the lectures will be based on articles from journals and conferences in the field of computer vision and medical image analysis.

Announcement

Top 3 Best Course Projects

• Andrew King. Semantic Segmentation of Coral Reef Survey Images with Fully Convolutional Models.

• Vyom Shrivastava, Prajay Shetty, and Dong-Yu Yu. Facial Landmark Detection using Tasks-Constrained Deep Convolutional Network.

• Dharamendra Kumar, Rakesh Bhavsar, and Ayush Jaiswal. Scalable Object Detection and Model Comparison.

Innovative Course Project

• Jiankun Zhu and Jiahao Xu. Handwriting Recognition, Alternatives to CNNs.

Best Paper Presenter: Andrew King.

Course Information

• Class meetings: TR 02:00pm – 03:15pm, W 02:30 pm – 03:20 pm, @ Boyd 306, GSRC

• Instructor: Yi Hong (yihong -at- cs.uga.edu, office: Boyd 616)

• Office hours: T 3:30pm - 4:30pm, R 10am - 11am, or by appointment

• Course webpage: http://cs.uga.edu/~yihong/CSCI8955-Fall2017.html

Topics

• Image and shape regression: cross-sectional and longitudinal studies

• Sparse representation, dictionary learning, and low-rank approximation

• Random forests: classification forests and regression forests

• Deep learning: convolutional neural networks (CNNs) and recurrent neural networks (RNNs)

Prerequisites

No prior experience in computer vision or medical image analysis is required, while some exposure to image analysis, machine learning, or numerical computation is highly recommended.

Grading

This course will mix lectures with reading and discussion of related research papers and projects. Students should present their selected papers and work on a project related to the topics discussed in this course. In particular, the grading will be based on

• Project (40%), including proposal (5%), update (5%), presentation (15%), and write-up (15%)

• Paper presentation (25%)

• Paper summaries (20%)

• Participation (15%), including attendance (5%), study group (5%), and presenation evaluation (5%).

There is no exam.

Tentative Schedule

Group 1: Rajeswari, Ankita, Andrew, Christopher, and Sam

Group 2: Vyom, Dhara, Rakesh, Ayush, and Dongyu

Group 3: Abolfazl, Mehdi, Abbas, Moham, and Vamsi

Group 4: Jiankun, Pranay, Jiahao, Chirag, and Prajay

Reading List

Papers (in black) will be assigned on a first-come-first-serve basis. You may also propose a paper that is not listed, but you must get it approved.

Image and Shape Regression

Image registration

1. Zitova and Flusser, Image registration methods: a survey, Image and Vision Computing 2003.
2. Beg et al., Computing large deformation metric mappings via geodesic flows of diffeomorphisms, IJCV 2005.
3. Brown et al., Multi-image matching using multi-scale oriented patches, CVPR 2005.
4. Hart et al., An optimal control approach for deformable registration, CVPR Workshop 2009.

Geodesic regression

1. Niethammer et al., Geodesic regression for image time-series, MICCAI 2011.
2. Fletcher, T., Geodesic regression and the theory of least squares on Riemannian manifolds, IJCV 2013.
3. Singh et al., A vector momenta formulation of diffeomorphisms for improved geodesic regression and atlas construction, ISBI 2013.
4. Fishbaugh et al., Geodesic shape regression in the framework of currents, IPMI 2013.

Higher-order models

1. Hinkle et al., Intrinsic polynomials for regression on Riemannian manifolds, JMIV 2014.
2. Singh et al., Splines for diffeomorphisms, MedIA 2015.
3. Hong et al., Parametric regression on the Grassmannian, TPAMI 2016.

Longitudinal studies

1. Singh et al., A hierarchical geodesic model for diffeomorphic longitudinal shape analysis, IPMI 2013.
2. Durrleman et al., Toward a comprehensive framework for the spatiotemporal statistical analysis of longitudinal shape data, IJCV 2013.
3. Schiratti et al., A mixed effect model with time reparametrization for longitudinal univariate manifold-valued data, IPMI 2015.

Kernel regression

1. Davis et al., Population shape regression from random design data, ICCV 2007.
2. Banerjee et al., A nonlinear regression technique for manifold valued data with applications to medical image analysis, CVPR 2016.

Sparse Representation, Dictionary Learning, and Low-Rand Approximation

Sparse coding & Dictionary learning

1. Aharon et al., K-SVD: An algorithm for designing overcomplete dictionaries for sparse representation, TSP 2006.
2. Elad and Aharon, Image denoising via sparse and redundant representation over learned dictionaries, TIP 2006.
3. Mairal et al., Online dictionary learning for sparse coding, ICML 2009.
4. Wright et al., Robust face recognition via sparse representation, TPAMI 2009.
5. Yang et al., Image super-resolution via sparse representation, TIP 2010.
6. Wang et al. Semi-coupled dictionary learning with application to image super-resolution and photo-sketch synthesis, CVPR 2012.
7. Mairal et al. Task-driven dictionary learning, TPAMI 2012.
8. Gangeh et al., Supervised dictionary learning and sparse representation - a review, arXiv:1052.05928 2015.
9. Bao et al., Dictionary learning for sparse coding: Algorithms and convergence analysis, TPAMI 2016.
10. Yang and Yang, Top-Down Visual Saliency via Joint CRF and Dictionary Learning, TPAMI 2016.

Low rank approximation

1. Wright et al., Robust principal component analysis: Exact recovery of corrupted low-rank matrices by convex optimization, NIPS 2009.
2. Cabral et al., Matrix completion for weakly-supervised multi-label image classification, TPAMI 2012.
3. Zhang et al., Learning structured low-rank representation for image classification, CVPR 2013.
4. Liu et al., Low-rank atlas image analyzes in the presence of pathologies, TMI 2015.
5. Ge et al., Matrix Completion has No Spurious Local Minimum, NIPS 2016.

Random Forests

Local optimization

1. Fanello et al., Filter forests for learning data-dependent convolutional kernels, CVPR 2014.
2. Schulter et al., Fast and accurate image upscaling with super-resolution forests, CVPR 2015.
3. Dollar and Zitnick, Fast edge detection using structured forests, TPAMI 2015.

Global optimization

1. Schulter et al., Alternating decision forest, CVPR 2013.
2. Schulter et al., Alternating regression forests for object detection and pose estimation, ICCV 2013.
3. Ren et al., Global refinement of random forest, CVPR 2015.

Hybrid architecture

1. Kontschieder et al., Deep neural decision forests, ICCV 2015.
2. Xie et al., Aggregated Residual Transformations for Deep Neural Networks, arXiv:1611.05431 2016.
3. Loannou et al., Decision forests, convolutional networks and the models in-between, arXiv:1603.01250 2016.
4. Murthy et al., Deep Decision Network for Multi-Class Image Classification, CVPR 2016.

Understanding random forests

1. Denil et al., Narrowing the gap: Random forests in theory and in practice, ICML 2014.
2. Scornet et al., Consistency of random forests, The Annals of Statistics 2015.
3. Scornet et al., On the asymptotics of random forests, Journal of Multivariate Analysis 2016.

Deep Learning

Overview

1. LeCun et al., Deep learning, Nature 2015.
2. Schmidhuber et al., Deep learning in neural networks: An overview, Neural Networks, 2015.
3. Lipton et al., A critical review of recurrent neural networks for sequence learning, arXiv:1506.00019 2015.

Convolutional neural networks

1. Alex et al., ImageNet classification with deep convolutional neural networks, NIPS 2012.
2. Szegedy et al., Going deeper with convolutions, CVPR 2015.
3. Long et al., Fully convolutional networks for semantic segmentation, CVPR 2015.
4. He et al., Deep residual learning for image recognition, CVPR 2016.
5. Dai et al., Deformable Convolutional Networks, arXiv:1703.06211 2017.
6. Huang et al., Densely Connected Convolutional Networks, CVPR 2017.

Recurrent neural network

1. Gregor et al., DRAW: A recurrent neural network for image generation, arXiv:1502.04623 2015.
2. Srivastava et al., Unsupervised learning of video representations using LSTMs, ICML 2015.
3. Jain et al., Structural-RNN: Deep Learning on Spatio-Temporal Graphs, CVPR 2016.
4. Chung et al., Hierarchical Multiscale Recurrent Neural Networks, arXiv:1609.01704 2016.

Autoencoder

1. Vincent et al., Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion, JMLR 2010.
2. Le et al., Building high-level features using large scale unsupervised learning, ICML 2012.
3. Kingma et al., Auto-encoding variational Bayes, ICLR 2014.
4. Burda et al., Importance weighted autoencoders, ICLR 2016.

Deep structured models

1. Schwing and Urtasun, Fully connected deep structured networks, arXiv:1503.02351 2015.
2. Chen et al., Learning deep structured models, ICML 2015.
3. Zheng et al., Conditional random fields as recurrent neural networks, ICCV 2015.

Generative Adversarial Networks (GANs)

1. Goodfellow et al., Generative adversarial nets, NIPS 2014.
2. Radford et al., Unsupervised representation learning with deep convolutional generative adversarial networks, arXiv:1511.06434 2016.
3. Oord et al., Pixel Recurrent Neural Networks, ICML 2016.
4. Mao et al., Least squares generative adversarial networks, arXiv:1611.04076 2017.
5. Shrivastava et al., Learning from Simulated and Unsupervised Images through Adversarial Training, CVPR 2017.

Training deep neural networks

1. Srivastava et al., Dropout: A simple way to prevent neural networks from overfitting, JMLR 2014.
2. Ioffe et al., Batch normalization: Accelerating deep network training by reducing internal covariate shift, arXiv:1502.03167 2015.
3. Papernot et al., Semi-supervised Knowledge Transfer for Deep Learning from Private Training Data, ICLR 2017.
4. Goyal et al., Accurate, Large Minibatch SGD:Training ImageNet in 1 Hour, arXiv:1706.02677 2017.

Object Detection & Annotation

1. Girshick et al., Rich feature hierarchies for accurate object detection and semantic segmentation, CVPR 2014.
2. Girshick R., Fast R-CNN, ICCV 2015.
3. Ren et al., Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks, TPAMI 2017.
4. Redmon et al., You only look once: Unified, real-time object detection, CVPR 2016.
5. Liu et al., SSD: Single Shot Multibox Detector, ECCV 2016.
6. Redmon and Farhadi. YOLO9000: Better, Faster, Stronger, ICCV 2017.
7. Castrejon et al., Annotating Object Instances with a Polygon-RNN, ICCV 2017.

Deep Learning in Medical Image Analysis

1. Ronneberger et al., U-net: Convolutional Networks for Biomedical Image Segmentation, MICCAI 2015.
2. Shen et al., Multi-scale Convolutional Neural Networks for Lung Nodule Classification, IPMI 2015.
3. Milletari et al., V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation, arXiv:1606.04797 2016.
4. Albarqouni et al., AggNet: Deep Learning From Crowds for Mitosis Detection in Breast Cancer Histology Images, TMI 2016.
5. Setio et al., Pulmonary Nodule Detection in CT Images: False Positive Reduction Using Multi-View Convolutional Networks, TMI 2016.
6. Dou et al., Automatic Detection of Cerebral Microbleeds From MR Images via 3D Convolutional Neural Networks, TMI 2016.
7. Havaei et al., Brain tumor segmentation with Deep Neural Networks, Medical Image Analysis 2017.
8. Litjens et al., A Survey on Deep Learning in Medical Image Analysis, arXiv:1702.05747 2017.
9. Esteva et al., Dermatologist-level classification of skin cancer with deep neural networks, Nature 2017.

Understanding deep learning & Visualization

1. Szegedy et al., Intriguing properties of neural networks, arXiv:1312.6199 2013.
2. Goodfellow et al., Explaining and harnessing adversarial examples, arXiv preprint arXiv:1412.6572 2014.
3. Simonyan et al., Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps, ICLR workshop 2014.
4. Zeiler et al., Visualizing and understanding convolutional networks, ECCV 2014.
5. Nguyen et al., Deep neural networks are easily fooled: High confidence predictions for unrecognizable images, CVPR 2015.
6. Yosinski et al., Understanding neural networks through deep visualization, ICML 2015.
7. Zhang et al., Understanding deep learning requires rethinking generalization, ICLR 2017.

Reference Books and Resources

• Trevor Hastie, Robert Tibshirani, and Jerome Friedman. The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Second Edition, Springer.

• Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. In preparation for MIT Press. http://www.deeplearningbook.org.

• Julien Mairal, Francis Bach, and Jean Ponce. Sparse Modeling for Image and Vision Processing. Now Publishers, 2014.

• Antonio Criminisi, Jamie Shotton, and Ender Konukoglu. Decision Forests for Classification, Regression, Density Estimation, Manifold Learning and Semi-Supervised Learning. Microsoft Research technical report TR-2011-114.

• Antonio Criminisi and Jamie Shotton. Decision Forests for Computer Vision and Medical Image Analysis. Springer 2013.

• Madeleine Udell, Corinne Horn, Reza Zadeh, and Stephen Boyd. Generalized Low Rank Models. Foundations and Trends® in Machine Learning, 2016.

Useful Links

Dectionary Learning and Low-Rank Approximation

• SPAMS

• Dictionary Learning Tools for Matlab

• Semi-Coupled Dictionary Learning

• LRSLibrary

• Low-Rank Matrix Recovery and Completion

• Low-Rank Atlas-to-Image Registration

Random Forests

• Sherwood C and C# code library for decision forests

• Regression tree package by Pierre Geurts

• Scikit-learn

• CloudForest

Deep Learning

• Caffe

• Torch

• Theano

• Tensorflow

• Keras (High-level wrapper for Theano and Tensorflow)

Abbreviations

• Journals – IJCV: International Journal of Computer Vision,   TPAMI: IEEE Transactions on Pattern Analysis and Machine Intelligence,   TSP: IEEE Transactions on Signal Processing,   JMIV: Journal of Mathematical Imaging and Vision,   TIP: IEEE Transactions on Image Processing,   TMI: IEEE Transactions on Medical Imaging,   MedIA: Medical Image Analysis,   JMLR: Journal of Machine Learning Research.

• Conferences – MICCAI: Medical Image Computing and Computer Assisted Intervention,   IPMI: Information Processing in Medical Imaging,   ISBI: International Symposium on Biomedical Imaging,   ICML: International Conference on Machine Learning, CVPR: IEEE Conference on Computer Vision and Pattern Recognition,   ECCV: European conference on Computer Vision,   ICCV: International Conference on Computer Vision,   NIPS: Neural Information Processing Systems,   ICLR: International Conference on Learning Representations.

Disclaimer

The instructor reserves the right to make changes to the syllabus, including project due dates. These changes will be announced as early as possible.