Abstract
Learning by ignoring, which identifies less important things and
excludes them from the learning process, is an effective learning
technique in human learning. There has been psychological studies
showing that learning to ignore certain things is a powerful tool for
helping people focus. We are interested in investigating whether this
powerful learning technique can be borrowed from humans to improve the
learning abilities of machines. We propose a novel learning approach
called learning by ignoring (LBI). Our approach automatically identifies
pretraining data examples that have large domain shift from the target
distribution by learning an ignoring variable for each example and
excludes them from the pretraining process. We propose a three-level
optimization framework to formulate LBI which involves three stages of
learning: pretraining by minimizing the losses weighed by ignoring
variables; finetuning; updating the ignoring variables by minimizing the
validation loss. We develop an efficient algorithm to solve the LBI
problem. Experiments on various datasets demonstrate the effectiveness
of our method.