AM261 Class Notes, Fall 2007

Lectures by Stuart Geman

transcribed by Mario Micheli

• Projects

• All Lectures, 1 to 40, in one file (20.7MB)

• Lecture 1, 09/05/07
  Tentative Syllabus

• Lecture 2, 09/07/07
  Gibbs Ensemble, Maximum Entropy Principle (MEP)

• Lecture 3, 09/10/07
  More on MEP, Large Deviations

• Lecture 4, 09/12/07
  Large Deviations Principle (LDP)

• Lecture 5, 09/14/07
  Properties of the Partition Function

• Lecture 6, 09/17/07
  Information Theory

• Lecture 7, 09/19/07
  Coding, Asymptotic Independence

• Lecture 8, 09/21/07
  Exchangeability, de Finetti's Theorem

• Lecture 9, 09/24/07
  Maxwell's distribution of velocities for ideal gases, Local Chaos

• Lecture 10, 09/26/07
  Statistics: Estimation, Consistency

• Lecture 11, 09/28/07
  Bias and Variance, Examples, Inspection Paradox
  Plus: a paper on the Inspection Paradox

• Lecture 12, 10/01/07
  Bias and Variance of Density Estimator, Data Likelihood

• Lecture 13, 10/03/07
  Maximum Likelihood Principle and Relative Entropy

• Lecture 14, 10/05/07
  Classification, Neyman-Pearson Lemma and ROC curve

• Lecture 15, 10/10/07
  Proof of Neyman-Pearson Lemma, Bayesian Classification

• Lecture 16, 10/12/07
  K Nearest Neighbor (KNN) classifier

• Lecture 17, 10/15/07
  Bayesian generalization of KNN, Bias and Variance

• Lecture 18, 10/17/07
  Support Vector Machines, Max-margin Linear Classifier
  Plus: a note on the Max-margin Linear Classifier

• Lecture 19, 10/19/07
  Max-margin Nonlinear Classifier, soft-margin, Wolfe Dual

• Lecture 20, 10/22/07
  Kernels, Mercer's Theorem

• Lecture 21, 10/24/07
  Radial Basis Functions

• Lecture 22, 10/26/07
  General RBFs; Bias, Variance, Consistency of SVM using RBFs

• Lecture 23, 10/29/07
  Examples of Bias-Variance tradeoff, Cross Validation for density estimation

• Lecture 24, 10/31/07
  Cross Validation in general, density estimation for Exponential r.v.

• Lecture 25, 11/02/07
  More on density Estimation, VC Dimension and Metric Entropy

• Lecture 26, 11/05/07
  A theorem by Vapnik, Human Learning, Introduction to Dependency Graphs

• Lecture 27, 11/07/07
  MRFs, Gibbs distributions, Hammersley and Clifford Theorem, Application to Markov Chains

• Lecture 28, 11/09/07
  Marginals and Posteriors, Hidden Markov Models

• Lecture 29, 11/12/07
  Computing on General Graphs, Dynamic Programming

• Lecture 30, 11/14/07
  Marginals and Normalizing Constants, Computational Complexity

• Lecture 31 (revised), 11/16/07
  Sampling, Forward-backwards algorithm for HMMs, Examples

• Lecture 32, 11/19/07
  Introduction to Markov Chain Monte Carlo (MCMC)

• Lecture 33, 11/26/07
  Equilibrium distribution, Ergodicity, Detailed Balance, Reversibility

• Lecture 34, 11/28/07
  Gibbs Sampler, Metropolis-Hastings algorithm

• Lecture 35, 11/30/07
  MCMC in practice: Sampling, Computing Expectations, and Simulated Annealing

• Lecture 36, 12/03/07
  More on Simulated Annealing, Image Processing

• Lecture 37, 12/05/07
  Numerical examples, Estimation, Exponential Families, Likelihood Function

• Lecture 38, 12/07/07
  Maximum Likelihood for Exponential Family: Gradient Descent, Convexity

• Lecture 39, 12/10/07
  Pseudo Likelihood, motivational example for EM (Expectation-Maximization)

• Lecture 40, 12/12/07
  MM and EM