Seminars

View all Seminars  |  Download ICal for this event

Deep learning with information-theoretic insights

Series: Department Seminar

Speaker: Dr. Ashok Vardhan Makkuva

Date/Time: Mar 05 18:00:00

Location: Online Seminar

Abstract:
Ranging from arts to science, deep learning has achieved remarkable breakthroughs in recent years through ChatGPT (natural languages), AlphaGo (game playing), and AlphaFold (biology). At the same time, many of these deep models are susceptible to surprising failures like hallucinations and lack of arithmetic skills. With their ever growing prominence and ubiquity, a fundamental understanding of the success and pitfalls of these deep learning models is of thus paramount importance.

Along this theme, I will present my contributions in deep learning with insights from information theory and probability. Through our work Attention with Markov, I will demonstrate a principled framework for a systematic theoretical and empirical analysis of large language models (LLMs). In particular, our framework allows for a precise characterization of the interplay between the data-distributional properties, the model architecture, and the final model performance, which we believe to be first of its kind. In addition to interesting insights, our framework provides a new avenue for a principled study of LLMs with exciting open questions, which I will discuss in the end.
<br>
This is an online seminar. The meeting link is: Link

Speaker Bio:
Ashok is a postdoctoral researcher at EPFL with Michael Gastpar. He obtained his PhD in ECE from the University of Illinois at Urbana-Champaign in August 2022, with Pramod Viswanath and Sewoong Oh. He obtained his Masters in ECE with Yihong Wu also from UIUC in 2017. Earlier he graduated from IIT Bombay with a B.Tech. in EE and Minors in Mathematics working with Vivek Borkar. His research interests are in foundations of data science in topics including machine learning, information theory, optimization, and statistics. He is a recipient of Best Paper Award from ACM MobiHoc 2019. He is also a recipient of several graduate student awards and fellowships including Joan and Lalit Bahl Fellowship (twice), Sundaram Seshu International Student Fellowship, finalist for the Qualcomm Innovation Fellowship 2018.

Host Faculty: R Govindarajan