Introduction

NeuroSurgeon is a python toolkit built to enable deep learning researchers to easily uncover and manipulate subnetworks within trained models. NeuroSurgeon provides a simple API to inject differentiable binary masks techniques into linear, attention, and convolution layers in BERT, GPT, ResNet, and ViT-style models within Huggingface Transformers. Differentiable masking has several use cases, including:

• Pruning to uncover functional subnetworks

  • Relevant Papers:

    • Csordás et al. 2021

    • Lepori et al. 2023

    • Panigrahi et al. 2023

• Subnetwork Probing

  • Relevant Papers:

    • Cao et al. 2021

    • Conmy et al. 2023

• Training with L0 Regularization

  • Relevant Papers:

    • Louizos et al. 2018

    • Savarese et al. 2020

Installation

NeuroSurgeon is tested in python3.9, and requires several packages, such as PyTorch and the Huggingface Transformers libary. Install using pip: pip install NeuroSurgeon

Support

This is a research library, designed to be extended and improved as the field advances. Please report bugs, issues, and feature requests here!