Deep Learning

Generalisable Agents for Neural Network Optimisation

We train RL agents (an agent per layer) on simple supervised settings to learn to improve optimisation by taking actions, like adapting the learning rate, and then generalise to more resource-intensive supervised learning tasks.

Kale-ab Tessera, Callum Tilbury, Sasha Abramowitz, Ruan de Kock, Omayma Mahjoub, Benjamin Rosman, Sara Hooker, Arnu Pretorius

Generalisable Agents for Neural Network Optimisation

Deep Learning Indaba Practicals 2022

A collection of high-quality practicals covering a variety of modern machine learning techniques.

Just-in-Time Sparsity: Learning Dynamic Sparsity Schedules

Sparse neural networks have various benefits such as better efficiency and faster training and inference times, while maintaining the …

Kale-ab Tessera, Chiratidzo Matowe, Arnu Pretorius, Benjamin Rosman, Sara Hooker

Just-in-Time Sparsity: Learning Dynamic Sparsity Schedules

Keep the Gradients Flowing: Using Gradient Flow to Study Sparse Network Optimization

We use gradient flow to study sparse network optimization.

Kale-ab Tessera, Sara Hooker, Benjamin Rosman

Keep the Gradients Flowing: Using Gradient Flow to Study Sparse Network Optimization

On Sparsity In Deep Learning: The Benefits And Pitfalls Of Sparse Neural Networks And How To Learn Their Architectures

Overparameterization in deep learning has led to many breakthroughs in the field. However, overparameterized models also have …

Kale-ab Tessera