Subspace Distillation for Continual Learning

An ultimate objective in continual learning is to preserve knowledge learned in preceding tasks while learning new tasks. To mitigate forgetting prior knowledge, we propose a novel knowledge distillation technique that takes into the account the manifold structure of the latent/output space of a neural network in learning novel tasks. To achieve this, we propose to approximate the data manifold up-to its first order, hence benefiting from linear subspaces to model the structure and maintain the knowledge of a neural network while learning novel concepts. We demonstrate that the modeling with subspaces provides several intriguing properties, including robustness to noise and therefore effective for mitigating Catastrophic Forgetting in continual learning. We also discuss and show how our proposed method can be adopted to address both classification and segmentation problems. Empirically, we observe that our proposed method outperforms various continual learning methods on several challenging datasets including Pascal VOC, and Tiny-Imagenet. Furthermore, we show how the proposed method can be seamlessly combined with existing learning approaches to improve their performances. The codes of this article will be available at https://github.com/csiro-robotics/SDCL.


Introduction
Continual Learning (CL) is the process of robust, efficient and gradual learning in non-stationary environments.A fundamental aspect of intelligence is the capability of incrementally learning from sequential experiences.Equipping neural networks with CL capability requires the model to preserve its previously learned experiences while acquiring novel knowledge.Neural networks, trained in an offline mode, are currently the method of choice in a wide spectrum of problems in AI and machine learning.The underlying assumption here is that the model has the knowledge about all the decisions it should take in the future apriori.For example, all classes a model will encounter in future are known in an image classification task.Furthermore, in offline training, data used for training the model in future steps should be i.i.d, otherwise internal representations learned by the model are hardly useful.
In this paper, our focus is to design a mechanism that enables neural network model to learn continually in a dynamic environment.One may wonder is it advantageous for a model to learn sequentially like humans?Continual learning techniques will endow our machines to learn potentially over a lifetime, as does a human.Furthermore, having visual understanding and semantic segmentation in mind, continual adaptation to a changing target specification enables the model to learn a diverse, and growing set of classes.This aspect of continual learning is commonly considered as a necessity towards human-level artificial general intelligence [1].Also, we note that continual learning

Input Image Segmentation Mask
Step: t-1 Step: t Step: t+1 Classes (e.g.. table and chair) from the old task at step t and future task at step t + 1 are collapsed into background at current task t. methods could offer profound advantages for models even in stationary settings, by enabling them to improve their efficacy without the need to train from scratch upon availability of new data.
In a continual learning setting, current Deep Neural Networks (DNNs) exhibits a drastic fall in the overall performance when the model is trained on a series of tasks.Precisely, in absence of samples from old tasks, the performance degrades on previously encountered tasks after the model is trained on a novel tasks.In other words, the knowledge from previous tasks gets overwritten thoroughly and DNNs forget previously learned tasks abruptly once information relevant to novel tasks is presented [2].This phenomenon of forgetting prior tasks because of the changes on critical weights related to previously observed tasks is often referred to as Catastrophic Forgetting [3,4,5,6,7].Therefore, to design DNNs for Continual Learning, one needs to address Catastrophic Forgetting.
Another problem of interest in this paper is Continual Semantic Segmentation (CSS).Semantic segmentation [8,9,10] is the task of assigning a category label such as "person" or "vehicle" to every single pixel of an image.In class-incremental CSS problem, a model is sequentially exposed to learn a set of novel classes.At the end of each training step, the CSS model is supposed to classify a pixel with all the seen classes until current task for evaluation.Aside from catastrophic forgetting, in CSS, we need to tackle another fundamental problem, namely Background Shift [11].
In a conventional semantic segmentation setup, all object categories are predefined, and the class "background" encapsulates all other object categories that are not relevant to the problem at hand.In contrast and in CSS, at each learning step, the class background merely corresponds to categories that do not belong to any of the classes at the current step (see Fig. 1).As a result, the class background contains not only pixels from unseen and future classes but also pixels from previously seen and old classes.This setting can be considered as a dense prediction task with noisy labels as the future unseen classes or old seen ones are grouped under a super-class named background.If certain measures are not taken, the background shift could exacerbate the catastrophic forgetting even further.
A common way of addressing Catastrophic Forgetting and Background Shift is to distill the knowledge from old model to current one (see Fig. 3).Distillation methods such as LwF [12], PODnet [13]), often match the output/latent representation of a network, and hence ensuring that the prior knowledge remains unchanged in current model and the performance remains consistent on old tasks.
In this paper, we introduce a structured form of knowledge distillation [14] that is suitable for both Continual Learning with class-incremental setting and Continual Semantic Segmentation (CSS) [15,11].Precisely, we propose to distill the structure of the feature space in the intermediate layers of neural network to preserve the previously learned knowledge in the current model.Our proposed method encodes the structure via low-dimensional subspaces.Subspaces have been used in a broad range of problems in computer vision to model the data manifold locally.Subspaces are robust to perturbation, and can be computed for high-dimensional data easily, hence has been employed .We see a significant improvement of IOU when Subspace Distillation is added with output distillation based methods: ILT [15], and MiB [11] and pseudo-labeling based feature distillation method: PLOP [16] methods.
with success for adapting neural networks [17,18].Therefore, to mitigate catastrophic forgetting in CL, we propose to maintain geometric structure of the feature space through encoding subspaces between models from sequential learning steps.Our approach starts with decomposing extracted feature map in the intermediate layers of deep neural network followed by constructing structures of it through selecting prominent subspaces that approximate the data manifold to the first-order.This enables us to impose constraint to maintain similar subspace structures between old and new models.In our method, we formulate the constraints using the geometry of Grassmannian [19], and propose to minimize the distance between corresponding feature subspaces of old and current model.
As a motivating example, to examine the ability of subspace distillation in improving catastrophic forgetting by preserving prior knowledge, we evaluate the performance of subspace distillation by adding it to existing state-ofthe-art CSS methods, e.g.. ILT [15], MiB [11], and PLOP [16] and report the result in Fig. 2. The result shows that the average IOU for all of the three methods improves significantly on the Pascal VOC dataset.Furthermore, our proposed subspace distillation algorithm can be merged with other distillation techniques seamlessly and provides them with complementary constraints to enforce structural similarities.
Overall, our contributions in this paper are as follows: • We propose a robust feature distillation strategy, namely Subspace Distillation (SD) to tackle catastrophic forgetting in CL through applying constraint on maintaining similar feature structure between old and new model.
• We present a generalized end-to-end continual learning framework using our proposed Subspace Distillation (SD) strategy in presence of a small subset of already observed samples from past tasks.
• Our proposed Subspace Distillation (SD) strategy requires backpropagation through Singular Value Decomposition (SVD) method as it relies on SVD to compute the basis of subspaces.We show how this can be done in closed form solution by using partial derivative.
• Our proposed approach outperforms state-of-the-art continual learning methods on MNIST, CIFAR10 and Tiny-Imagenet datasets with varying memory size.
• We also show that a significant improvement can be achieved by combining subspace distillation strategy with existing methods for CSS on Pascal VOC dataset for different short and long task settings.

Related Work
In this section we discuss related works in class-incremental learning and continual semantic segmentation as the targeted application.

Continual Learning
A variety of methods has been proposed to alleviate catastrophic forgetting for class-incremental classification problems [4,12,13,20,21,22].These methods for continual learning are classified primarily into three categories, such as regularization, dynamic architecture and memory replay based methods [23].Regularization based methods preserve already learned information by imposing constraints on the update of weight [24,25,4,26], intermediate feature representation [13,27,28], prediction [12,29], gradient [30,31], or combination thereof.Li et al. [12] in Learning without Forgetting (LwF) introduced knowledge distillation strategy in the output layer to minimize the dissimilarity between old task and new one.The significance of each synapses is measured to penalize update on most influential synapses and a surrogate loss function to estimate loss for old tasks is used with modified cost function in SI [26].Hou et al. in [27] proposed cosine normalization to combat catastrophic forgetting and facilitate the seamless integration of new and prior knowledge within a continual learning model.By adjusting the magnitudes of the model's weights, cosine normalization provides precise control over the influence of novel tasks.This mechanism ensures that the model does not prioritize the new task at the expense of previously learned tasks, thereby preserving valuable knowledge while accommodating novel knowledge.This normalization approach helps to mitigate the negative impact of catastrophic forgetting and enhances the model's ability to generalize across multiple tasks.Cheraghian et al. in [32] introduced a semantic-aware distillation loss for fewshot class incremental learning that takes into account the semantic structure of the data.The distillation strategy utilizes semantic embeddings associated with each class to guide the distillation process.The integration of semantic information fosters the preservation of prior knowledge in the continual learning (CL) model by facilitating learning not only from the representations of the previous model but also from the semantic relationships among the classes.During the incremental learning process, PODNet in [13] used a Pooled Outputs Distillation (POD) mechanism to transfer knowledge from the previously learned tasks to the current task.Specifically, the outputs of the intermediate layers of the previous model are pooled and distilled into the corresponding layer of current model through minimizing the discrepancies using Euclidean distance of L2-normalized features.Mathematically, POD can be expressed as 2 , where f old i (x) and f new i (x) are the pooled output of the i th spatial position for input x using old and new model respectively.One of the limitations of PODNet is the absence of an explicit mechanism to effectively preserve the underlying latent structure, as it primarily relies on imposing constraints on the pooled features.Consequently, the presence of outliers or noisy attributes can potentially compromise the effectiveness of distillation in PODNet.In contrast, our proposed subspace distillation approach addresses this concern by imposing constraints on the low-dimensional subspaces derived from the latent features of both the old and current models.This not only enhances robustness to noise but also ensures the preservation of latent structure in continual learning scenarios.Dynamic architecture based methods allocate new neurons to adapt to novel task.Andrei et al. [33] introduced progressive network where old knowledge remains unchanged by keeping previously trained model frozen and a novel sub-network with fixed resources is allocated to learn new knowledge.Yoon et al. [22] proposed dynamically extendable network (DEN) that learns a compact representation by selective training and expanding neural network capacity by optimal number of units when new task arrives.Recently, Douillard et al. in [34] proposed first transformer based architecture where dynamic expansion of task specific token is used.Memory-based methods partially stores the previous data and train model by replaying stored old data together with new data [29,35].Rebuffi et al. [29] introduced iCARL method where herding based sample selection was used to keep a small portion of previous dataset in the memory and replayed interleaved with new samples.Many recent approaches have extended iCARL to bias correction problem for classifier [36], a metric learning model for imbalance dataset [27], a memory sampling method [37,38,39].Javed et al. [36] introduced a dynamic threshold moving method to address the classifier bias generated by the knowledge distillation approach in iCARL.Hou et al. [27] in LUCIR proposed to combine inter-class separation constraint, old classes geometric structure preserving constraint and cosine normalization to tackle imbalance dataset problem.Aljundi et al. [37] claimed that memory sampling method is crucial, and accuse random memory sample selection for sub-optimal performance on old tasks.In [37] replay memory sampling is defined as a constrained optimization problem and formulated as solid angle minimization problem to maximize the diversity in the replay memory.While in iCARL, the memory constitutes samples randomly chosen based on the cluster centers, in [38] and [39], diverse sample selection mechanisms, including most interfered sample retrieval and global distribution matching based sampling are utilized.
To address the security concern, few recent approaches employed generative model to produce samples belonging to old classes [40] instead of storing subset of old samples in memory.Deep Generative Replay (DGR) [40] proposed a dual-model architecture, one for generating pseudo samples and another for solving tasks by replaying pseudo samples together with new samples.To reduce the memory footprint of storing real samples, compressed feature have been stored in [41,42].REMIND [41] used product quantization method to quantize latent representation and stored indices in memory that were used later to decode the representation for reply.However, because of incremental update in model, stored latent representation also requires adaptation.To fit the stored representation into current latent space, Iscen et al. [42] employed a multi-layer perceptron to map corresponding old and new feature map generated for images of current task.
Recently, Dark Experience Replay (DER++) [43] proposed to store both logit, and label for corresponding sample and used the reservoir sampling strategy to select samples from data stream for memory buffer.In DER++, knowledge distillation is performed by mapping output logit from current model with corresponding memory logit.

Continual Semantic Segmentation
In recent time, a growing number of works have emerged into continual semantic segmentation.In the literature, CSS methods primarily fall into two major categories: (i) regularization and (ii) Replay based model.
Following the success of regularization methods in CL, several works have proposed mechanism for controlling update of neuron weights to mitigate catastrophic forgetting in CSS [15,11,16,44].ILT [15] investigates a regularization-based technique by freezing weights of the encoder network after learning the first task and applying knowledge distillations for the upcoming tasks.ILT is also equipped with the masked cross-entropy loss and masks on the output of the current model to consider only the seen classes.However, this approach doesn't resolve the background shift problem of CSS properly.Since the background in CSS may include previously seen objects, MiB [11] proposed to preserve the knowledge of an old model with knowledge distillation.However, the difference compared to standard knowledge distillation is that the probabilities of the background and the novel classes are appended such that the non-overlapping prediction of a novel class as the output of a current model can be aligned with the output of a previous model.As knowledge distillation shows a promising direction for CSS, PLOP [16] introduced pseudo-labelling to tackle the background shift problem and adapt Pooled Out Distillation (POD) [13] to preserve the previously learned knowledge.Since POD is developed for a classification problem in classical continual learning settings with reliance on global statistics, thus PLOP uses a multi-scale version of POD to integrate local and global statistics at different intermediate layers.SDR [44] leveraged contrastive learning together with novel sparsity constraint and prototype matching strategy to efficiently learn novel tasks and mitigate forgetting of prior knowledge.In SDR, to organize geometric structure of feature representation, clusters of data are described using prototypes that are forced to be closer in consecutive learning steps and far apart from one another by using prototype matching and repulsive force.Additional sparsity constraint imposed on feature representation of same classes helped to construct well-separated and tight cluster as well as create space for accommodating novel classes.Replay based methods tackle the catastrophic forgetting in CSS by either storing real images from past tasks [45] or generating synthetic data of prior classes [46].Cha et al. in SSUL [45] proposed to extract future classes from background that are defined as unknown classes to facilitate learning novel classes.SSUL used a subset of old samples for the first time in the literature of CSS to improve stability and plasticity of model.Additionally, freezing weights of encoder and old classifiers with binary cross entropy loss, and pseudo-labeling techniques helped to improve catastrophic forgetting in [45].Maracani et al. [46] employed generative model to produce samples from previously seen tasks together with newly proposed background impainting method for pseudo-labeling to tackle background shift and catastrophic forgetting.

Preliminaries
In continual learning, a model needs to learn from a series of tasks.Each learning task is represented by its training set, often samples from novel classes or concepts.Let T = {T 1 , T 2 , • • • , T T } be a sequence of T tasks.In the setup we are interested in this work, every task comprises of where X i ∈ X denotes a training image of size W × H and y i ∈ Y is the corresponding class belonging to current task t.We also maintain a fixed size memory M to retain a small subset of samples from old tasks to better tackle the catastrophic forgetting.The goal of our knowledge distillation based approach is to apply constraints on updating weights of model so that the model generates similar latent representation and prediction in future tasks (i.e., t, t + 1, There is a subtle difference in setup when CL is considered for semantic segmentation.For the problem of continual semantic segmentation, the training set consists of input samples and their corresponding segmentation mask.We denote the training set of CSS by where Y i ∈ Y is the corresponding segmentation mask for the input image X i .Each pixel of the image belongs to a set of classes given by the current task C t .Previously observed classes (i.e., C t−1 ) and future classes (e.g., C t+1 ) are all labeled as the background class c bg for task t.In both continual semantic segmentation and classification tasks, at step t, the model should be able to predict all the observed classes, C 1:t , throughout the learning experience.

Evaluating an Incremental Learning Model
Class-Incremental Learning The performance of continual learning methods are measured using the average task accuracy [25] in experiments.Average accuracy is computed by average performance across all the previously observed and current tasks after training on current task t and is defined as: where Acc t,i is the accuracy of task i after learning task t. .

Continual Semantic Segmentation
Intersection Over Union (IOU) [47] metric is commonly used to evaluate the robustness towards catastrophic forgetting, in other words stability, and the ability of learning new class (i.e., plasticity) of CSS methods.IOU is computed at the end of learning all task at step t for (i) initial set of classes C 1 at first task, (ii) incrementally leaned classes C 2:t , and (iii) all classes C 1:t .IOU is defined as follows where T P, FP and FN refers to true-positive, false-positive and false-negative, respectively. .

Regularization with Knowledge Distillation
Distilling knowledge from old model to the current model has shown promises to mitigate catastrophic forgetting of neural network in a CL setting [12,13].Here, the old model with the knowledge of already observed tasks acts as a teacher model and the purpose is to distill knowledge from teacher model to student model (i.e., current model) such that prediction of these two models matches for previously seen tasks samples.Knowledge distillation is often performed on the probability space to relate the temperature smoothed probability distribution of old to the new model [12,11].Feature distillation on the other hand is performed on the feature space extracted from the intermediate layers of neural network to match corresponding local or global statistics of feature maps between old and new model [44,15,13,16].
Assume that the extracted feature and the prediction using teacher model from step t − 1 for input X are f t−1 and ŷt−1 , respectively.Similarly, let f t and ŷt be the feature and prediction respectively for the same input using the student model at step t.The knowledge distillation is performed between teacher and student model by minimizing the following loss function Feature distillation strategy applies constraint on the similarity between old and new feature representation and is performed by minimizing a notion of distance between corresponding features of f t−1 and f t , such as the distance induced by the ℓ 1 norm as

Methodology
In this section, we first discuss our main contribution, the Subspace Distillation (SD) , and its properties.Next, we will elaborate on how the SD will be used for classification and segmentation problems.
Subspace Distillation.Let F t i ∈ R d×p and F t−1 i ∈ R d×p be the extracted features from layer i of DNNs at step t and t − 1, respectively.The goal of feature distillation is to ensure that the learned knowledge at step t − 1 remain unchanged at step t while training DNNs on novel dataset D t .To avoid cluttering equations, from now on we drop the layer index, unless it is not clear from the context.In conventional feature distillation approaches, old feature maps or statistics are directly matched with corresponding new one.That is, in order to obtain for distillation per Eq. ( 4), different pooling strategies (i.e.., channel, height or width pooling) have been discussed in PODnet [13].
Our hypothesis here is that distilling geometric structure of feature distribution can enrich the distillation process and will help mitigating the catastrophic forgetting in continual learning.In doing so, we propose to model feature maps F t and F t−1 with a set of subspaces, S = {S j } τ j=1 .Soon, we will discuss how the set of subspaces will be constructed for the classification and segmentation problems, but for now, we focus on the main idea.Each subspace S ∈ S is represented by its basis as R d×m ∋ P; m ≪ d with P ⊤ P = I m .Note that, m ≤ p.Our goal here is to preserve the subspace structure of the intermediate feature maps extracted from different layers of old model to the new one.We argue that, robust knowledge distillation through maintaining similarity across subspaces will be advantageous to improve continual classification/segmentation paradigms.This is achieved by enforcing a constraint on subspace similarity between the old and new model.To do so, we propose to minimize the distance between corresponding subspace constructed from feature maps of the old and new model.A valid distance between S i and S j is a distance that is invariant to the choice of the basis of the subspace.To be more specific, assume P i ∈ R d×m and P j ∈ R d×m are the basis for S i and S j , i.e., P i P ⊤ i = P j P ⊤ j = I m .Then a distance between S i and S j should meet : where g : R d×m × R d×m → R + is the distance function and R i , R j ∈ O m with O m denoting the orthogonal group.To be precise, given P t i and P t−1 i , we opt to minimize the projection metric [48] as The projection metric is a proper distance on Grassmannian and endows intriguing properties, among them, the length of curves on Grassmannian obtained by δ p (•, •) is simply related to the length obtained by the geodesic distance via a fixed constant [48].In order to illustrate the rationale behind computing subspace distance using Eq.( 5), we present a trivial example involving the XY plane residing within three-dimensional space (R 3 ) as a two-dimensional subspace.
Consider the XY plane in R 3 as a 2D subspace in 3D space.Both represent the XY plane; hence their distance should be zero.The distance used in Eq.( 5), i.e., P t⊤ i not only satisfies the required conditions, but also closely related to the geodesics on the Grassmannian [48].In Eq. ( 5), the mapping f : R d×m → R m×m ; f (P) = PP ⊤ is a diffeomorphism between Grassmannian and the space of symmetric matrices (positive semidefinite to be adequate).The induced distance δ 2 p (P t i , is invariant to the action of the orthogonal group, satisfying the requirement of having a Grassmannian distance.Furthermore, since P ⊤ P = I m , the distance can be simplified to δ 2 p (P t i , , which is computationally very attractive in comparison to geodesics on Grassmannian that require computing SVD. Before formulating the SD loss for continual learning, note that in practice and in order to construct the subspace representing feature map, we rely on matrix decomposition techniques, and in particular on Singular Value Decomposition (SVD).In other words, given F t i and F t−1 i , we first apply SVD to attain P i and P t−1 i and then use them accordingly for distillation.This operation differs from many common operations in deep learning, in the sense that a somewhat complicated matrix operation is involved, hence one may wonder how backpropagation will work in this case.Backpropagation through SVD.First note that for δ 2 p (P t i , P t−1 i ), we have Please note that the subspace from previous model acts as a teacher for distillation and we only need the gradient with respect to P t i to update our current model, hence the above derivation.The next step to update the weights of the current model is to obtain the gradient of the loss with respect to F t i , which requires us to backpropagate ∇ P through the SVD operation.By applying chain rule (see [49] for backpropagation through matrix operations) , with Feature map F t i is decomposed as It can be shown that (see Appendix A for the derivation) where K ∈ R p×p is defined as follows Here, Σ m ∈ R m×m consists of top m rows and first m columns of Σ.Note that, A diag is the diagonal part of A (i.e., all off-diagonal elements are set to 0).

Subspace Distillation for Continual Classification
For the task of continual classification, we distill the subspace constructed across samples in a mini-batch.For a mini-batch X B , Y B of size b, let f j ∈ R d be the latent representation for input X j ∈ X B .The formulation below can be applied to any layer in a DNN and hence we drop the layer index for the sake of simplicity.Assume there are p = ⌊b/|C t |⌋ samples per class in the mini-batch.In SD, we propose to compute class-wise subspaces using latent features generated from both old and new model.More specifically, from the mini-batch, we form , where Y B ∋ y k,i = k).We note that stacking ordering is not important in forming F t−1 k , F t k as we are merely interested in subspace spanned by the samples.Next, we represent each class k from the teacher and the student model by its low-dimensional subspace P t−1 k , P t k ∈ R d×m , m ≤ p.This is achieved by applying thin SVD to F t−1 k , F t k and picking up the top left singular vectors.With the above, the SD loss is defined as

Subspace Distillation for Continual Semantic Segmentation
For class-incremental semantic segmentation problem, we propose to compute subspace across the channel dimension of intermediate feature map for each sample in a batch.To reduce the memory overhead while performing subspace distillation, instead of computing subspace from all feature maps at a layer, we split the full feature maps into several smaller group G with p channels.To compute a subspace from each group, we need to form a matrix representation from the feature map.To do so, for a particular input, we form {F t−1 g } G g=1 and {F t g } G g=1 with F t−1 g , F t g ∈ R d×p from the feature maps.Here, d = wh, where h and w denote the height and width of the feature map.We encode the geometry of F t−1 j , F t j by low-dimensional subspaces P t−1 g , P t g ∈ R d×m via SVD.The subspace distillation loss is defined as

Class-Incremental Continual Learning using Subspace Distillation
In class-incremental learning, we maintain a fixed memory to store a subset of samples using reservoir sampling strategy [50] from prior tasks.The samples in the memory are subsequently used during training the model on a novel task.We compute distillation loss between subspace constructed from latent feature maps extracted by feeding memory samples to old and new model.Afterwards, by minimizing subspace distillation loss combined with classification loss on novel and memory samples, we adapt DNNs model for class-incremental learning.The classification loss used in conjunction with the subspace distillation loss is the cross entropy defined as: where, y c , and ỹt c are the true label and predicted probability for class c respectively for the input X and ỹt = f Θ t (X).Putting everything together, the overall loss used to train our model to tackle catastrophic forgetting in continual class incremental learning is where, X represents the images belonging to novel task while X ′ represents the examples from the memory buffer.α and β are hyper-parameter used to control the contribution of the second loss term and subspace distillation, respectively.We present the overall steps of training a continual learning model using subspace distillation in Algorithm 1.

Algorithm 1 Class-Incremental Learning using Subspace Distillation
Input: Dataset D t , Memory M, and Model from step t − 1, The new model at time t with parameters Θ t 1: Initialize Θ t with Θ t−1 2: for iteration 1 to max iter do Sample a mini batch (X ′ , y ′ ) from the memory M

5:
ỸB ← h Θ t (X B ) 6: ỹ′ ← h t cls (f t ) 8: Compute Cross Entropy loss, ℓ CE between ground truth Y B and prediction ỸB using Eq. ( 14) 10: Compute Cross Entropy loss, ℓ CE between ground truth y ′ and prediction ỹ′ using Eq. ( 14) 11: Compute Subspace Distillation loss, ℓ CL SD between f t−1 and f t with Eq. ( 12) 12: Update Θ t by minimizing the overall loss defined based on two cross entropy loss, ℓ CE computed for X B , and X ′ respectively and subspace distillation loss, ℓ SD as in Eq. (15) 13: end for

Continual Semantic Segmentation using Subspace Distillation
Since image pixels belonging to prior classes are labeled as background in CSS, old model is employed for distilling knowledge.The idea of knowledge distillation is a crucial step and widely adapted in preserving prior knowledge for CSS.In CSS, we apply subspace distillation loss at intermediate layers of model to maintain consistency in geometric structure of latent features.Additionally, we use output distillation to ensure that the current model mimics the output of prior model.In other words, subspace distillation is combined with classification loss and output distillation loss to train CSS model at any step t.Bellow, we briefly discuss the classification loss and output distillation loss for CSS.

Classification Loss
In semantic segmentation, the cross entropy loss is often applied at each pixel of the output generated by the DNN.However, in CSS and to train the model at time t, the background class may include prior classes from already seen tasks.Hence, keeping background shift problem of CSS in mind, we define the cross entropy loss as follows: where y t x,c is the ground truth label at pixel x for class c and ỹt x,c is defined as follows Here, ŷt x,c is the predicted probability for class c at pixel x using model at task t.

Knowledge Distillation
Distilling knowledge from the old model to current one is crucial to mitigate the catastrophic forgetting and accurate semantic segmentation in continual learning setting.Output distillation by mimicking the predicted probability of old model to the new model has been widely used in the literature of continual learning [12,51].However, because of background shift problem, conventional knowledge distillation approach cannot be applied directly in the continual semantic segmentation.In this work, similar to MiB method, we follow the masked cross entropy by relating the old model's prediction to the new model's prediction after combining new classes probability with background.Therefore, the adapted output distillation loss is defined as follows: where y t−1 x,c is the predicted probability using f θ t−1 for class c at pixel x in image X.We define ỹt x,c as follows: Since the modified distillation loss does not directly match the predicted background class probability of old model to the new model, the distillation loss plays a vital role in tackling the background label shift problem in class-incremental learning for semantic segmentation.Finally, the combined loss of our end-to-end continual semantic segmentation method can be written as the linear combination of classification loss, ℓ CE , feature distillation using newly proposed subspace distillation loss,ℓ SD , and knowledge distillation loss, ℓ KD Here, α and β are the predefined hyperparameter used to control the contribution of ℓ CS S KD and ℓ CS S SD respectively.Algorithm 2 summarizes the steps needed to be taken to train a new model for CSS.

Algorithm 2 Subspace Distillation for CSS
5: Compute Cross Entropy loss, ℓ CE between ground truth Y B and prediction Ỹt B using Eq. ( 16) 7: Compute Output Distillation loss, ℓ KD between the prediction from current and old model, Ỹt B and Ỹt−1 B respectively using Eq. ( 18) 8: for l ← 1 to L do Compute Layer-wise Subspace Distillation loss, ℓ SD between F t−1 [l] and F t [l] using Eq. ( 13) 11: end for

12:
Update Θ t by minimizing the linear combination of ℓ CE , ℓ KD and ℓ SD with Eq. ( 20) 13: end for

Experiments
We start this section by describing the datasets, architectures, and implementation details used in our experiments for both continual image classification and semantic segmentation.For classification problem we focus on classincremental and task-incremental settings while in case of continual semantic segmentation problem, we only follow the class-incremental setting.
Continual Learning for Classification.We evaluate our proposed method on 3 different benchmark datasets: MNIST [52], CIFAR-10 [53], Tiny Imagenet [54].Following DER settings, To quantitatively demonstrate the effectiveness of our proposed subspace distillation method in different tasks settings, we split the MNIST, and CIFAR-10 datasets into sequences of 5 tasks having 2 classes per task.Tiny Imagenet is split in 10 tasks with equal number of classes in all sequential tasks (20 classes per task).In our comparative analysis we consider eight state-of-the-art regularization and distillation methods including LwF, oEWC, SI, iCARL, A-GEM, ER, DER and DER++.In our experiment, we follow class-incremental (CI) and task-incremental (TI) protocols described in [55].The identity of task is provided along with input sample in TI setting while in case of CI setting, task identity is absent.In our experiments, we partition data into distinct sets of non-overlapping classes, hence applicable in both class-incremental and task-incremental scenarios.Furthermore, we maintain a consistent ordering of all classes across all algorithms, guaranteeing that each algorithm receives identical data for every task.Compared to the class-incremental scenario, where task identity is missing, task-incremental learning benefits from having access to task identifiers, which in return helps in selecting appropriate classifiers, rendering it a comparatively easier scenario.Conversely, the class-incremental scenario poses a challenge due to the absence of task identity.
Implementation Details.Task-incremental learning can be implemented using either a multi-head or single-head classifier, depending on the specific implementation.Following the implementation of DER [51], in our approach, we employed a single-head classifier model to learn in class-incremental scenarios.However, in task-incremental settings, we relied on the output masking technique of the single-head classifier to identify task-specific classes, leveraging the availability of task identity during inference.The output masking technique is used to selectively mask specific outputs of the single-head classifier, depending on the task at hand.This approach allows the single-head classifier to effectively prioritize the relevant outputs for the current task while disregarding the irrelevant ones.Following the setting described in [51,56], a neural network with 2 fully connected layers of 100 neurons are used to extract latent feature for MNIST dataset.For CIFAR and Tiny Imagenet datasets, a modified Resnet18-like [29] structure is used for feature extraction.Finally, a single head linear classifier is used for classification across the experiments on MNIST, CIFAR and Tiny Imagenet datasets.We augment both stream and memory samples by applying random crop and horizontal flip for both CIFAR10 and Tiny-Imagenet datasets [51].
The SGD optimizer is used for training DNN model throughout the learning experiences with keeping flexibility in selection of batch size and mini-batch size for different task setting.Please refer to the appendix for the task-specific values of hyperparameters.Following DER training scheme, we train our model for one epoch at each learning step on MNIST dataset while for relatively complex dataset such as CIFAR10 and Tiny-Imagenet we use 50 and 100 epochs respectively for training.
Continual Semantic Segmentation.We benchmark our proposed method against state-of-the-art methods with different task settings on Pascal-VOC 2012 [57] dataset.We follow the experimental setup used in [11] for VOC dataset, baseline implementation and metric.Precisely, in our comparative study, we consider eight methods, elastic weight consolidation (EWC) [4], learning without forgetting (LwF) [12], Riemannian walk (RW) [25], ILT [15], MiB [11], SDR [44], GIFS [58] and PLOP [16], The Pascal-VOC 2012 dataset has 10582 training images and 1449 validation images which is used for testing.Each pixel belongs to 20 foreground classes against background.We use three different tasks settings: 19-1, 15-5 and 15-5s in the experiments.In the first two settings we incrementally add 1 and 5 novel classes on the base model trained on 19 and 15 classes respectively.However, in the 15-5 setting we add 1 class in 5 consecutive tasks while base model remains similar to the 15-5 setting.In our evaluation of continual semantic segmentation methods, we follow class overlapped setting where task consists of images that may contain classes belonging to future task with a label of background.
Implementation Details.In our implementation, we use a Deeplab-V3 [59] architecture with ResNet-101 [60] as a backbone that is pretrained on Imagenet [61].Following [62], to reduce required memory, we use inplace activated batch normalization for training our model.We use Stochastic Gradient Decent (SGD) with a momentum of 0.9 and learning decay of 1e-4 to train our model.Following [16], we crop the images to 512 × 512 followed by applying random horizontal flip while training our model at each step on Pascal VOC dataset.While computing subspace distillation, we construct subspaces at intermediate layers using a group of 32 channels and we consider top 5 subspaces in our distillation strategy.We train our model with a learning rate of .001from the second task while first task is trained with a higher learning rate of .01 and each task model is trained for 30 epochs with a batch size of 48 distributed over 4 GPU.We use α = 10 and β = 0.01 while combining output distillation and KD with pixel-wise cross-entropy loss to compute overall loss.For computational efficiency, we employ O1 optimization from Nvidia APEX library1 to train with half precision.Finally, we use the validation image set to evaluate our model.Below, we present the experimental results of our proposed subspace distillation method for both continual learning with classification and continual semantic segmentation problem.

Continual Learning
We analyze the efficacy of subspace distillation in continual learning setting for classification problem and present the result in Table 1.In comparison, we consider two knowledge distillation based methods (LwF [12], iCARL [29]), two regularization methods (SI [26], oEWC [63]) and three memory replay based methods (ER [20], DER [51], DER++ [51]).Additionally, we provide upper and lower bounds, that reflect to jointly training on tasks one by one or on all tasks with Stochastic Gradient Descent (SGD) without any specialized strategy designed for CL.
The results suggest that regularization methods performs poorly on class incremental learning setting as those methods are developed focusing on task incremental learning setting.This observation indicates that regularization towards the set of old parameters is not suitable for tackling catastrophic forgetting because of the local information modeling weight importance [51].Overall, replay based methods outperform regularization methods with big margin across the datasets.Regularization methods performs good on MNIST datasets for task incremental settings while they perform notably worse on comparatively complex datasets, e.g.CIFAR10, Tiny-Imagenet.We observe that our method outperforms state-of-the-art replay base methods in task incremental learning setting on all datasets.Noticeably, for class incremental learning setting on split CIFAR-10 dataset our method performs noticeably better by 15% and 8% percentage points compared to iCARL and ER methods, respectively.However, DER method performs slightly better than subspace distillation method as DER relies on storing additional information (i.e., logits) together with corresponding image and label from past tasks in the memory for the distillation that requires additional memory requirements.We observe that by combining dark experience replay method DER, we can further improve our result.
For instance, we can achieve about 2% and 5% performance gain on Tiny-Imagenet and CIFAR10 datasets respectively with medium size memory buffer.We evaluate our model in a more constrained setting where each sample at a task is presented to model once.In other terms, we train our model for a single epoch at each task given a tiny buffer size of 100 samples.Overall, because of the complex setting, continual learning model faces underfitting problem and struggles to mitigate catastrophic forgetting.Replay-based distillation methods also suffer from poor performance when there are insufficient exemplars available, especially for previously observed classes where no memory exemplars exist.For instance, in S-Tiny Imagenet, during the final task, with just 100 memory exemplars, only one example is stored for the 100 previously seen classes out of a total of 180 prior classes, leading to a subpar performance of replay-based distillation methods.We observe that our method considerably improves on baseline ER method across the datasets and performs significantly better than DER.For example, our method enjoys around 2% improvement on both CIFAR-10 and Tiny-Imagenet datasets than state-of-the-art DER method for class incremental setting.However, we think that this scenario deserves further investigation because of its complex nature.
In task-incremental setting, we notice better performance compared to class-incremental setting for all methods across the datasets.The reason behind such observation is the presence of task identifier at the test time in taskincremental learning that makes the problem easier.We see that SI and oEWC perform badly on relatively complex datasets such as CIFAR10, and Tiny-Imagenet though regularization based methods are designed particularly for taskincremental scenario and our method outperforms both methods with unbridgeable gap in performance.Subspace distillation method performs remarkably better than iCARL across the settings with different memory size.For example, our method outperforms iCARL by 15% percentage points with large memory buffer on Tiny-Imagenet, and by 24% percentage points with medium size memory on CIFAR10 dataset.We also note that our method shows competitive performance with other memory replay based methods.We see around 2% improvement on ER for task incremental setting on Tiny Imagenet dataset while the performance gain is around 15% on CIFAR10 dataset for both small and medium size memory.Furthermore, by combining our method with DER, we see 4%, 5% and 3% performance improvements on DER with small, medium and large size memory buffer for task-incremental setting on CIFAR10 dataset.

Continual Semantic Segmentation
Table 2 reports the IOU of different baseline strategies and subspace distillation method for three scenarios: (i) 19-1, (ii) 15-5 and (iii) 15-1 overlap scenarios on Pascal-VOC dataset.The results suggest that our proposed subspace distillation based method outperforms state-of-the-art methods in 19-1 and 15-5 task settings by a significant margin.Though our method does not match the performance of PLOP in 15-5s task setting, we observe about 13% IOU improvement on MiB at last task.Furthermore our method outperforms ILT by around 30% in IOU of last task.We note that subspace distillation shows consistency in retaining prior knowledge across the settings.To show the efficacy of subspace distillation method in tacking catastrophic forgetting, we combine SD with ILT and PLOP methods, and see improvements on IOU in different settings.We observe that by combining SD with ILT the overall performance (i.e., mIOU) of ILT method improves by about 3% and 2% for 19-1 and 15-5 task settings respectively while performance remains similar for 15-5s setting.Similarly, we notice 2% performance improvement of PLOP methods after combining SD for 19-1 and 15-5s settings which suggests that SD imposes additional constraint on distillation strategy using multi-scale POD that helps learning complimentary information and tackling catastrophic forgetting better together.Though our method shows little lower plasticity in case of learning new classes, subspace distillation shows promises in retaining prior knowledge and tackling forgetting old classes.strength in preserving already learned knowledge and at each learning step PLOP performs significantly better than MiB.We observe considerably improved performance across the learning steps when subspace distillation is added with MiB.SD facilitates MiB to preserve previously learned knowledge and therefore we note about 9%, and 12% improvement of mean IOU at task 5, and 6 respectively after adding SD with MiB.Similarly, SD helps PLOP tackling catastrophic forgetting better and we see improvements on mean IOU at the end of all learning tasks.

Ablation Studies
In this section, we evaluate our model in terms of changes in activation map, and feature representation in different learning steps in class incremental learning setting on CIFAR10 dataset.Furthermore, in case of continual semantic segmentation on Pascal VOC dataset, we investigate the contribution of each regularizer in subspace distillation strategy and analyze our proposed method with (i) varying dimensionality of subspace and (ii) different no. of channels to construct the subspace.Changes in Activation Map.We examine our model in terms of where neural network is looking at the input images for decision making and whether the activated region has been changed over time in continual learning setting.To do so, We first train a neural network on 5 tasks CIFAR10 datasets and, at the end of learning different tasks, we feed the model with an image that is presented to the model at second step followed by computing heatmap images using GradCAM [65].We present the result in Figure 3 showing the evolution of the important regions in the images.The result suggests that DER method activated the body part of cat while our method looks at face region of cat image.Furthermore subspace distillation consistently activates the same regions and the changes in the activation map is lower than the state-of-the-art of DER++ method.Similarly, both DER and DER++ show inconsistency in activating the interested region where bird is located in the image while SD method exhibits robustness in activating the body part of bird in consecutive tasks.Similarity in Representation.We analyze our models capability of retaining previously learned knowledge by maintaining similar feature representation in the intermediate layer of neural network while learning a series of tasks.Precisely, we employ Centered Kernel Alignment (CKA) metric [66] to compute similarity between intermediate feature representation generated by neural network at task t and t + 1 on test dataset of CIFAR10 and report the comparative result in Figure 5. Overall we observe that the CKA similarity steadily increases as the model faces more tasks and we think that is because the model becomes more stable as it gets adapted incrementally on novel dataset in presence of samples from prior tasks in memory buffer.We notice that our proposed subspace distillation method consistently maintains higher CKA similarity of feature representation throughout the learning experiences.For instance, SD method outperforms baseline method, ER, on CKA similarity metric by 8% in the first task while the gap reduces to 3% in the last task.Varying no. of channels for computing subspace.We evaluate the effect of dimensionality of channels being used to construct the subspace on Pascal VOC dataset for 15-5s and 19-1 tasks setting by combining SD with PLOP.With the increasing no. of channels for constructing subspace, we do not observe any considerable changes in both stability and plasticity as the IOU for new classes and mean IOU for old classes remains stable.As reported in 4, subspace distillation exhibits robustness against the varying no. of channels used to construct subspace.Subspace Dimensionality.To also examine the effect of dimensionality of subspace used in distilling structure for continual semantic segmentation, we combine subspace distillation with PLOP method and report the experimental result on Pascal VOC dataset for 19-1 overlap tasks setting in 5.The results suggest that, with the increase of subspace dimension, the performance on the already observed classes remains similar and subspace dimensionality exhibits less impact on the overall performance.Contribution of each Regularizer.In order to evaluate the contribution of each regularization term in our proposed subspace distillation method for continual semantic segmentation and class-incremental learning, we conducted experiments on the Pascal VOC dataset with varying numbers of tasks and S-CIFAR10 dataset (see Tab. 6).Results in Tab.6a suggest that both regularizers, ℓ CS S KD and ℓ CS S SD , contribute to improving overall performance across the different settings.For example, in the (19-1) 2-task setting, combining regularizer KD with the baseline using CE led to an increase in performance of around 9%.We also noted an additional 4% improvement on both previously observed classes and overall performance when regularizer SD was combined with KD in our proposed subspace distillation method.At the same time, the experimental findings displayed in Tab.6b demonstrate that the combination of subspace distillation (SD), ℓ CL SD , and memory replay, ℓ CL CE yields substantial improvements in overall accuracy and forgetting on the S-CIFAR10 dataset.Specifically, there is an approximate increase of 9% in accuracy and a reduction of 15% in forgetting.Computational Complexity.Our proposed distillation loss requires computing the subspace basis employing SVD.For an input feature vector F ∈ R d×m ; d > m, computational complexity of SVD is O d × m × min(d, m) = O(dm 2 ).Assuming, P t ∈ R d×n and P t−1 ∈ R d×n be the basis of top n subspaces used in our computation, the computation cost of P t ⊤ P t−1 is O(n × d × n) = O(dn 2 ).Therefore, the total computation cost of our proposed subspace distillation loss can be expressed as O(dm 2 + dn 2 ).During our experiments on the 5-tasks CIFAR10 dataset, we found that one iteration of our replay-based subspace distillation (SD) method requires approximately 60ms on Tesla P100-SXM2 GPU for a batch of 32 samples, while the replay-based vanilla SGD method requires about 37ms.We rely on the assumption that a limited buffer memory is available throughout the learning process to store exemplars from prior tasks that might restrict the use of our method in privacy-focused applications.

Conclusion
In this work, we propose a generalised end-to-end continual learning framework where subspace distillation is at the core of it.Here, we model low dimensional intermediate feature representations using subspaces.By imposing constraint on maintaining similar subspace between old and new model, we ensure robustness in the model towards catastrophic forgetting.Proposed subspace distillation is equally effective for classification and semantic segmentation problem in continual learning scenarios.Empirical analysis shows that our proposed framework with subspace distillation achieves state-of-the-art performance in multiple settings on Pascal-VOC dataset for continual semantic segmentation and MNIST, CIFAR10, and Tiny-Imagenet datasets for class-incremental classification problem.In future, we would like to investigate more about the efficient use of subspace distillation for long tasks setting for CSS as well as for complex continual learning setting when model is trained on data stream that is presented to model once at training time.

Figure 1 :
Figure 1: Semantic shift of background classes at different learning steps in Continual Semantic Segmentation (CSS).Classes (e.g.. table and chair) from the old task at step t and future task at step t + 1 are collapsed into background at current task t.

Figure 2 :
Figure2: Average IOU at the end of training for 19-1 task setting on Pascal VOC dataset using different CSS methods with or without our proposed Subspace Distillation (SD).We see a significant improvement of IOU when Subspace Distillation is added with output distillation based methods: ILT[15], and MiB[11] and pseudo-labeling based feature distillation method: PLOP[16] methods.

Figure 3 :
Figure 3: Distillation strategies: (Left) Knowledge Distillation works on output layer and matches the probability distribution between old and new model, (Middle) Pooled Output Distillation (POD) applies similarity constraints on the pooled feature between old and new model and (Right) Subspace Distillation on the other hand maps corresponding subspaces constructed from the intermediate feature maps from old and new model.

3 :
Sample a mini batch (XB , Y B ) from D t 4:

9 :
Split F t [l] and F t−1 [l] into sub group 10:

Fig. 4 Figure 4 :
Fig. 4 reports the continual evolution of mean IOU for 15-5s (6 tasks) overlap setting on Pascal-VOC dataset.As depicted in the figure, the performance of MiB drops significantly throughout the learning steps while PLOP shows

3 :
The transition of what neural network is looking at in different learning tasks on CIFAR10 dataset using GradCAM[64].The model trained using our subspace distillation method consistently attends in the regions of the images where the feature of the cat or bird (i.e.., around the head or body) is located while the activated regions shift significantly for state-of-the-art DER++ and DER.

Table 2 :
[16]76.0920.16 73.43 78.10 51.21 71.70 50.62 14.41 42.00 Performance of different continual semantic segmentation method in IOU on Pascal VOC dataset for three continual overlap class learning settings: (i) 19-1 (ii) 15-5 and (iii) 15-1 tasks.Best values are represented in bold.Results for EWC, LwF, RW, ILT, PLOP and MiB are extracted from PLOP[16]paper while the result for SDR and GIFS are collected from the corresponding original paper.Higher is better.SD: Subspace Distillation.

Table 4 :
57gure5: CKA Similarity of the latent feature representation learnt by models in two consecutive task while learning on class incremental CI-FAR10 dataset.Subspace Distillation consistently maintains very high representational similarity that is indicative of high capability of knowledge preservation..5755.9776.47 46.38 75.04 32 66.83 21.56 56.05 76.50 46.39 75.07 64 66.75 21.84 56.06 76.48 46.23 75.04 Mean IOU for 15-5s and 19-1 incremental task setting on Pascal-VOC overlapped dataset with different no. of channels to compute subspace using SVD.

Table 5 :
Mean IOU for 19-1 overlap incremental task setting on Pascal-VOC dataset with different subspace dimension.
b) Experimental results of CL methods trained for one epoch with 500 memory exemplars on the S-CIFAR10 dataset. (

Table 6 :
Experimental results of (a) Continual Semantic Segmentation on Pascal VOC, and (b) Class-Incremenal laerning on S-CIFAR10 indicate that incorporating distillation with memory replay significantly improves the performance of the CL method.