BEGIN:VCALENDAR
VERSION:2.0
PRODID:-//project/author//NONSGML v1.0//EN
CALSCALE:GREGORIAN
BEGIN:VEVENT
DTEND:20230413T120000Z
UID:3d385daa22c45dddf66f86d32f0783b3-445
DTSTAMP:19700101T120016Z
DESCRIPTION:Data-free pruning of DNNs
URL;VALUE=URI:https://www.csa.iisc.ac.in/newweb/event/445/data-free-pruning-of-dnns/
SUMMARY:In this session there will be two talks on Data Free Pruning of DNNs,  an emerging theme in Model Compression.  
Each talk is based on a paper to be presented at ICLR 2023. 


Talk 1: 4:00-4:30

Tanay Narshana
Machine Learning Engineer 
Observe.AI


Title: DFPC: Data flow driven pruning of coupled channels without data.

Abstract:

Deep Learning models have now become ubiquitous. However, the hardware requirements to deploy SoTA models are increasing at a faster rate than what Moores law can deliver. This makes such models challenging to deploy. Model compression, particularly pruning, is one way to alleviate this problem. Modern, multi-branched neural network architectures often possess complex interconnections like residual connections between layers, which we call coupled channels (CCs). Most existing works are typically designed for pruning single-branch models like VGG-nets. While these methods yield accurate subnetworks, the improvements in inference times when applied to multi-branch networks are comparatively modest. These methods do not prune CCs, which we observe contribute significantly to inference time. For instance, layers with CCs as input or output take more than 66% of the inference time in ResNet-50. Structured pruning of CCs in these multi-branch networks is an under-researched problem. Moreover, pruning in the data-free regime, where data is not used for pruning, is gaining traction owing to privacy concerns and computational costs associated with fine-tuning. In this talk, we present our recently accepted work at ICLR 2023 on the problem of pruning CCs in the data-free regime. The efficacy of our methodology is demonstrated via empirical results. We achieve up to 1.66x improvements in inference time for ResNet-101 trained on CIFAR-10 with a 5% accuracy drop without fine-tuning. With access to the ImageNet training set, we achieve significant improvements over the data-free method and see an improvement of at least 47.1% in speedup for a 2.3% accuracy drop for ResNet-50 against our baselines.



Talk2: 4:30-5:00

Chaitanya Murti 
PhD Student
Robert Bosch Centre for Cyberphysical Systems
IISc

Title:  TVSPrune - Purning Nondescriminative filters Total Variation Separability of Intermediate Features

Abstract:

Achieving structured, data-free sparsity of deep neural networks (DNNs) remains an open area of research.  In this work, a solution to the problem of pruning filters with only access to the original data distribution, and without access to the original training set or loss function is proposed. The solution is based on the following hypothesis:well-trained models possess discriminative filters, and any non-discriminative filters can be pruned without impacting the predictive performance of the classifier. A new paradigm for pruning neural networks is proposed based on this hypothesis: distributional pruning, wherein access to the distributions that generated the original datasets is required. The discriminative ability of filters is formalised and quantified using the total variation (TV) distance between the class-conditional distributions of the filter outputs. Next, the LDIFF score is proposed. The LDIFF score is a heuristic to quantify the extent to which a layer possesses a mixture of discriminative and non-discriminative filters. The main contribution is a novel one-shot pruning algorithm, called TVSPrune, that identifies non-discriminative filters for pruning. This algorithm is extended to IterTVSPrune, wherein TVSPrune is applied iteratively, thereby enabling greater sparsification of a given model.
DTSTART:20230413T120000Z
END:VEVENT
END:VCALENDAR