Debugging using Orthogonal Gradient Descent

In this report we consider the following problem: Given a trained model that is partially faulty, can we correct its behaviour without having to train the model from scratch? In other words, can we ``debug" neural networks similar to how we address bugs in our mathematical models and standard computer code. We base our approach on the hypothesis that debugging can be treated as a two-task continual learning problem. In particular, we employ a modified version of a continual learning algorithm called Orthogonal Gradient Descent (OGD) to demonstrate, via two simple experiments on the MNIST dataset, that we can in-fact $\mathit{\text{unlearn}}$ the undesirable behaviour while retaining the general performance of the model, and we can additionally $\mathit{\text{relearn}}$ the appropriate behaviour, both without having to train the model from scratch.