Seminars

View all Seminars  |  Download ICal for this event

Data-free pruning of DNNs

Series: Department Seminar

Speaker: 1. Tanay Narshana Machine Learning Engineer Observe.AI 2. Chaitanya Murti PhD Student Robert Bosch Centre for Cyberphysical Systems IISc

Date/Time: Apr 13 16:00:00

Location: CSA Seminar Hall (Room No. 254, First Floor)

Faculty Advisor: Prof.Chiranjib Bhattacharyya

Abstract:
In this session there will be two talks on Data Free Pruning of DNNs, an emerging theme in Model Compression.
Each talk is based on a paper to be presented at ICLR 2023.


Talk 1: 4:00-4:30

Tanay Narshana
Machine Learning Engineer
Observe.AI


Title: DFPC: Data flow driven pruning of coupled channels without data.

Abstract:

Deep Learning models have now become ubiquitous. However, the hardware requirements to deploy SoTA models are increasing at a faster rate than what Moores law can deliver. This makes such models challenging to deploy. Model compression, particularly pruning, is one way to alleviate this problem. Modern, multi-branched neural network architectures often possess complex interconnections like residual connections between layers, which we call coupled channels (CCs). Most existing works are typically designed for pruning single-branch models like VGG-nets. While these methods yield accurate subnetworks, the improvements in inference times when applied to multi-branch networks are comparatively modest. These methods do not prune CCs, which we observe contribute significantly to inference time. For instance, layers with CCs as input or output take more than 66% of the inference time in ResNet-50. Structured pruning of CCs in these multi-branch networks is an under-researched problem. Moreover, pruning in the data-free regime, where data is not used for pruning, is gaining traction owing to privacy concerns and computational costs associated with fine-tuning. In this talk, we present our recently accepted work at ICLR 2023 on the problem of pruning CCs in the data-free regime. The efficacy of our methodology is demonstrated via empirical results. We achieve up to 1.66x improvements in inference time for ResNet-101 trained on CIFAR-10 with a 5% accuracy drop without fine-tuning. With access to the ImageNet training set, we achieve significant improvements over the data-free method and see an improvement of at least 47.1% in speedup for a 2.3% accuracy drop for ResNet-50 against our baselines.



Talk2: 4:30-5:00

Chaitanya Murti
PhD Student
Robert Bosch Centre for Cyberphysical Systems
IISc

Title: TVSPrune - Purning Nondescriminative filters Total Variation Separability of Intermediate Features

Abstract:

Achieving structured, data-free sparsity of deep neural networks (DNNs) remains an open area of research. In this work, a solution to the problem of pruning filters with only access to the original data distribution, and without access to the original training set or loss function is proposed. The solution is based on the following hypothesis:well-trained models possess discriminative filters, and any non-discriminative filters can be pruned without impacting the predictive performance of the classifier. A new paradigm for pruning neural networks is proposed based on this hypothesis: distributional pruning, wherein access to the distributions that generated the original datasets is required. The discriminative ability of filters is formalised and quantified using the total variation (TV) distance between the class-conditional distributions of the filter outputs. Next, the LDIFF score is proposed. The LDIFF score is a heuristic to quantify the extent to which a layer possesses a mixture of discriminative and non-discriminative filters. The main contribution is a novel one-shot pruning algorithm, called TVSPrune, that identifies non-discriminative filters for pruning. This algorithm is extended to IterTVSPrune, wherein TVSPrune is applied iteratively, thereby enabling greater sparsification of a given model.

Speaker Bio:
1. Tanay is a Machine Learning Engineer at Observe.AI, working with topic models and LLMs. He recently graduated from the Department of CSA, IISc, with a masters degree, advised by Prof. Chiranjib Bhattacharyya. Before that, he graduated with a bachelors degree from IIT Ropar. 2. Chaitanya Murti is currently a PhD candidate at the Robert Bosch Centre for Cyberphysical Systems of the Indian Institute of Science. Previously, he completed his Bachelor of Science and Master of Science degrees from the Illinois Institute of Technology.

Host Faculty: Prof. Chiranjib Bhattacharyya