Course Description:
Originally developed by C.E. Shannon in the 1940's for describing
bounds on information rates across telecommunication channels,
information and coding theory is now being employed in a large number
of disciplines for modeling and analysis of problems that are
statistical in nature. This course will provide a general
introduction to the field of information and coding theory. Main
course topics will include entropy, error correcting codes, source
coding, data compression. Of special interest will be the connection
of these ideas to problems in pattern recognition. The second half of
the course will include a number of projects relevant to Neuroscience,
Cognitive and Linguistic Sciences, and Computer Vision. Grades will
be based on projects, homework, a midterm and a final. Prerequisites:
Calculus. MATLAB or other computer experience will be helpful. Prior
exposure to probability theory/statistics will be helpful.
Course Outline I. Introduction II. Probability & Entropy III. Data Compression IV. Data Transmission V. Applications/Projects Text: Information Theory, Pattern Recognition and Neural Networks by David Mackay (1/17/00, or later preprint) - A bound copy of the essential chapters will available at a local copy center. Reserve Reading: (Sciences Library) Elements of Information Theory, Thomas M. Cover, Joy A. Thomas, Wiley, 1991. Foundations of Statistical Natural Language Processing, Christopher D. Manning and Hinrich Shutze, MIT Press, 1999.
Please register for AM194, Section II. (Section I is a different special topics course).
Entropy on the World Wide Web by Chris Hillman.
Information Theory, Pattern Recognition and Neural Networks
by David Mackay.