Seminars

View all Seminars  |  Download ICal for this event

Understanding self-supervised pre-training: Generalisation, expressivity and dynamics

Series: Department Seminar

Speaker: Prof. Debarghya Ghoshdastidar, TUM School of Computation, Information and Technology, Technical University of Munich, Germany

Date/Time: Mar 28 16:00:00

Location: CSA Auditorium, (Room No. 104, Ground Floor)

Abstract:
In this talk, I will briefly introduce self-supervised pretraining commonly used in foundation models for vision and tabular data. I will also introduce the key question related to statistical generalisation in foundation models: How do we guarantee statistical generalisation for different downstream prediction tasks given that the model is pre-trained with large amount of unlabelled (augmented) data? I will discuss some approaches for deriving generalisation error bounds, and highlight the issue with such results.

I will subsequently focus on the equivalence between self-supervised neural networks and kernel principal component analysis (PCA). This equivalence is based on two ideas: (i) optimal solution of self-supervised kernel models can be computed as a spectral embedding, and (ii) infinitely wide neural networks are equivalent to kernel models, characterised by the neural tangent kernel (NTK). Using the kernel connection of self-supervised learning, one can gain further insights into the role of augmented data and characterise the optimal augmentations. This talk is based on a series of works (arXiv: 2309.02028, 2403.08673, 2412.03486, 2411.11176).

Speaker Bio:
Prof. Ghoshdastidar conducts research in the theory of machine learning, artificial intelligence and network science. The main focus of his research is on the statistical understanding and interpretability of methods used in machine learning. His works provide new insights and algorithms for decision problems, involving complex data such as networks and preference relations, that arise in various fields including neuroscience, crowdsourcing and computer vision. Prof. Ghoshdastidar obtained a bachelor degree in electrical engineering in 2010 from Jadavpur University, India. He obtained a master degree in 2012 from the Indian Institute of Science, where he further completed his doctoral research in 2016. Subsequently, he joined the University of Tuebingen as a post-doctoral researcher and led a junior research group funded by the Baden-Wuerttemberg Foundation. In September 2019, he joined the Department of Informatics as a Tenure Track Professor for Theoretical Foundations of Artificial Intelligence.

Host Faculty: Prof. Ambedkar Dukkipati