7. Thoughts on Shortcut Learning by Shan Shi

1. Shortcut Learning

1.1 Have you learned shortcuts?

Due to its excellent learning ability, deep learning has become the unwavering main force driving the development of artificial intelligence. Deep learning has demonstrated excellent capabilities in various scenarios such as NLP and CV. However, deep learning also has an inherent problem: shortcut learning.

In shortcut learning, the shortcut refers to a flawed decision-making process, which is a 'shortcut' strategy adopted by machine learning models during the training process to minimize prediction errors. Under this strategy, the model prioritizes learning and predicting features that are directly related to the target, but may not be essential, leading to the possibility that the model may perform well on the training set but poorly on the test set and in practical applications.

Shortcut Learning is a special form of learning process that exists in many machine learning models. In this process, the model tries to find the fastest and most direct way to optimize its performance on the training data, which sometimes leads it to rely on some features that are not truly important or have deep meaning. To some extent, shortcut learning can be considered as a form of surface learning or shallow learning.

Suppose you want to train a machine learning algorithm model, and your task is to predict the results of future football matches by analyzing past football matches. You analyze a large amount of historical match data, including each team's players, scores, number of goals, even the weather on the day of the match, etc., to predict the match results. Ideally, you hope that the algorithm model can learn from these data how to judge the strength of the team, such as analyzing the skills, tactics, and physical fitness of the players. However, this requires a large amount of computing resources and time. But you notice a phenomenon: in the past matches, the team wearing red jerseys won most of the matches. So your algorithm quickly learns a shortcut: directly predicting that the team wearing red jerseys will win. This prediction method is fast and has a high accuracy on the training data.

However, when you use this algorithm to predict the results of future competitions, you find that its accuracy is far below your expectations. Because in real competitions, victory does not always belong to the team wearing red jerseys. This algorithm ignores the truly important factors that determine the results of the competition, such as player skills, tactics, physical fitness, etc., and relies solely on a superficial and irrelevant feature, that is, the color of the jersey.

This is an example of shortcut learning. The model learns a simple and direct prediction method, ignoring the truly important features. Although this method may work well on the training data, its performance is often poor in real-world scenarios in the future.

1.2 Is the answer correct if it is learned correctly?

After several centuries of research, theories such as mathematics and physics have formed a theoretical system that can withstand the test of reality and is self-consistent. However, compared with this, the research on deep learning is like a newborn baby, and there is still no relatively complete theoretical system. Many times, the detection ability of deep learning is quantified by detection indicators on certain datasets, and this result-oriented behavior sometimes indeed reflects the detection ability of the model, but it is deeply bound to the current dataset. Many times, we are difficult to judge whether the knowledge learned by the deep learning model is knowledge, or is a one-sided and biased view like a blind man feeling an elephant or seeing a part of a tiger through a bamboo tube.

Figure 1 Example diagram of shortcut learning

Figure 2 lists four case examples of shortcut learning generated by four deep learning models, even in some scenarios where the detection ability of deep learning models has reached or even exceeded human levels. In the first subfigure, the model identifies the green slope as sheep, in the second subfigure, the model identifies the gray image as a teapot, in the third subfigure, the model uses the hospital identification information in the upper right corner as the main learning feature for judging whether it is pneumonia. In the fourth subfigure, due to irrelevant information at specific positions in the text, the model is unable to complete the intelligent question answering task.

For these detection errors, it will bring up some deeper questions for researchers to think about: when can deep learning play a role, when the model detection fails, we need to know the cause and the improvement methods.

At the same time, we know that the research on deep learning is still in a relatively black box stage, so deep learning has the problem of non-explainability or incomplete explainability. When deep learning detection fails, the black box nature makes it difficult to find the cause of failure, and the characteristic of being guided by quantitative indicators makes it unable to provide help for correcting errors. Even in some cases, minor modifications to the input data that are difficult to detect by the human eye can lead to a significant impact on the detection results. The characteristic of these minor modifications causing the model detection results to be seriously inaccurate has also given rise to an attack behavior against deep learning: adversarial attacks, which I prefer to compare with the vulnerability exploitation of attackers against deep learning models.

2. What is Shortcut Learning Learning

Shortcut learning exists in all learning processes, whether it is natural organisms or deep learning models. For example, if you are familiar with 'Journey to the West', you might think that all people riding white horses are Tangseng, and all those carrying bamboo poles are Sha Wujing. When you read 'Journey to the West', you summarize the characteristics of Tangseng and Sha Wujing, and summarize them into classification decision rules. In a sense, this is a kind of bias of one-sided learning. For example, those riding white horses may not necessarily be princes, they may also be Tangseng. In general, the results of shortcut learning show that the learning process is valuable, but the knowledge or patterns learned are one-sided.

2.1 Inductive Reasoning from Cause to Effect

Figure 4 compares the shortcut learning of neural networks with human learning. Both learn from the data in the training set. The knowledge learned by human learning is graphic shape information: the pentagon belongs to label A, and the crescent shape belongs to label B; the knowledge learned by the neural network model is positional information: the label A is located in the lower left or upper right, and the label B is located in the upper left or lower right. From the perspective of features, humans tend to assign higher weight to shape features, while neural network models assign higher weight to positional information.

Figure 2: Comparison of the Differences in Learning Results

2.2 The Bias of Blind Men and the Elephant

The shortcut learning as an inherent characteristic of the learning process is often overlooked in the training process of machine learning, and before it is applied in engineering, the research stage of the algorithm will not be fully exposed. In response to this situation, researchers have analyzed the data distribution and feature weights corresponding to the model, as shown in Figure 5. It can be seen that different datasets can train different models, and the detection models trained by partial data have obvious limitations in the entire data distribution. Moreover, by comparing the feature situations in different datasets, it can also well explain the poor detection results of the model in new scenarios and new test datasets - giving excessive weight to incorrect features.

If the model learns insufficient information features, the performance of the model on the training set will not be good; if the model learns overfitting features, the performance of the model within the training set distribution will be good, but the performance of the model on the independent and identically distributed test set (Independent and Identically Distributed, I.I.d) will not be good; if the model learns shortcut features, the performance of the model on the out-of-distribution generalization test set (Out-of-distribution, o.o.d) will not be good; only when the model learns the expected features can it meet the expected generalization ability. However, the expected features are a relatively abstract concept and do not have a direct evaluation standard.

Figure 3: The relationship between data distribution, features, and model performance

Where does shortcut learning come from

Where does the shortcut learning come from? How does shortcut learning alleviate? The causes of shortcut learning have many aspects. The first is from shortcut features, that is, there are shortcut factors in the data; the second is from the discriminant model, that is, the imbalance of feature weights in the model. Both affect shortcut learning while also affecting whether the machine learning model can move from the research environment to the application scenario.

3.1 Data shortcuts

Figure 4: The influence of various background features

During the training process of deep learning models for target detection, if the background information of the detection target is monotonous and unique, then this background information will also become an important feature for target recognition. For example, if a deep learning model is trained for the detection of cows, but all the cows in the training set have grasslands as their background, in other words, the background information of the training dataset for cows is blue sky and green grass, rather than other backgrounds such as beaches, seas, etc. This will also lead to the fact that blue sky and green grass may have a very high feature weight in the current scenario. This problem of machine learning algorithm is also called dataset bias. Even in the case of big data, the shortcut factor still exists. Because this is not a problem of data scale, but the limitation of the data generation scenario itself leads to the lack of data distribution. For example, no matter how many photos of cows you take on the grassland, you still cannot solve the problem of the background feature of blue sky and green grass. The limitation of this training dataset cannot reflect the actual data distribution of the scene, thus producing the effect of blind men feeling an elephant.

3.2 Algorithm Shortcut

According to the analysis of the detection results in the paper <ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy>, as shown in Figure 7, the detection results of texture features are very good.

Figure 5: Target Recognition Effect of Different Feature Methods

Therefore, in the various convolution operations of CNN, the weight of texture features will also be greater. More importantly, in the end-to-end training process, this weight of texture features is obtained through the algorithm's self-training process through the convolution process, not imposed by humans on the deep learning model.

However, this is significantly different from the logic of human visual observation and judgment. The human eye will pay great attention to the external contour features of the target object, while not ignoring other important feature information. Due to the greater weight of texture features in deep learning models, it leads to misjudgments of certain images, such as the cat in Figure 8 being recognized as an elephant.

Figure 6: Cat under Texture Features

This situation is very common in different scenarios, as shown in Figure 9 below. Many objects that are the same category for humans are divided into different categories in deep learning models; at the same time, deep learning models will also put some unrelated objects into the same category.

Figure 7: Human Classification and Deep Neural Network Classification

4. Causes of Shortcut Learning

Because shortcut learning problems for deep learning have always existed, research on shortcut learning will also continue. Next, we will mainly introduce the causes of understanding and shortcut learning.

4.1 Shortcut Learning is Easier

4.1.1 Principle of Minimum Effort

Why is machine learning so prone to shortcut learning? One reason is the principle of minimum effort, for example, when speaking, everyone prefers to use smaller words, such as using 'phone' instead of 'mobile phone', and 'plane' instead of 'airplane'. The training process of machine learning is also the same, using the most direct features to find simple weight relationships to complete the learning process.

4.1.2 Impact of Key AI Modules

The impact of machine learning not only depends on data but also on the four components of machine learning algorithms: architecture, training data, loss function, and optimization method. The training process usually starts with data input into the model, which includes a fixed architecture and random initialization parameters. By comparing the predicted results with the ground truth, the optimization direction and the quality of the prediction are clearly defined through the loss function. By integrating these four components, the algorithm model's ease of shortcut learning is clarified from the algorithmic level.

4.2 The Importance of Result Evaluation

4.2.1 Scores Are Not Equivalent to Ability

The most dangerous time for shortcut learning is when it is not discovered. The most widely used evaluation standard for current machine learning is still to test the i.i.d. test set and output quantitative indicators. However, these quantitative indicators only reflect the detection results of the model under the current limited data, and cannot fully measure the true detection ability of the model. For example, an open-book exam with a teacher-defined exam scope will inevitably have high scores. Therefore, 99.9!% within the limited data range cannot necessarily indicate that the current model's detection ability is necessarily excellent.

4.2.2 Models Are Not Equivalent to Brains

Lloyd Morgan's Canon Rule: If a behavior can be explained by a lower psychological mechanism, then we can by no means attribute it to a higher psychological process. The Morgan rule is also effective in machine learning.

Some opinions believe that neural networks simulate the structure of animal neural networks, but we cannot directly assert that the training and testing process of neural networks is equivalent to the learning and application of animal brains, and there are obvious differences between the two. In short, similar behavior to humans does not necessarily mean the same thinking and decision-making methods as humans. In machine learning, situations that can be explained by shortcut learning should not be considered as the model itself may have more advanced decision-making capabilities.

5. How to Discover Short Cut Learning

5.1 More Perfect Test Data

When evaluating the detection ability of a model, we should mainly consider the model's ability to detect out-of-distribution data. We also use the example of cow object recognition to illustrate this. When testing the effectiveness of the model, we can test the model's ability to detect cows in the forest, on the grassland, in the river, and in the cowshed to determine that the knowledge (pattern) learned by the model is about cows themselves, not about片面 information such as forests, grasslands, etc.

A good out-of-distribution generalization test dataset should have the following conditions: first, it should have clear distribution transformations, which may be difficult for humans to distinguish, such as cows in different scenarios. Second, a good out-of-distribution generalization test set should have samples that are difficult for the model to detect, such as adding some white noise to the images in the OOD test set during image processing, or increasing the sample data of cows whose limbs are partially obscured by green leaves or fog. Third, the test dataset can include data that is not included in the training, that is, some extremely challenging data that is difficult for the model to detect. Common methods include adversarial attacks, which can discover the weak links of the model against o.o.d test data through adversarial attacks and use them as diagnostic tools for model optimization; ARCT, which deletes known agile learning sample data from the test set to build a more complex test dataset; and对抗conflicting features, such as the texture features and shape features of images, which is easy to compare with expert systems. At the same time, it also includes other data augmentation methods.

5.2 Model Deconstruction Analysis

By deconstructing and analyzing the model, we can try to understand which features the model's decision depends on. For example, we can use tools like SHAP (SHapley Additive exPlanations) or LIME (Local Interpretable Model-agnostic Explanations) to understand how the model's predictions are affected by various features. If the model over-relies on some low-value features, it may be shortcut learning.

At the same time, we can also use visualization techniques such as Class Activation Maps (CAM) to intuitively see which parts of the input data the model focuses on when making decisions, and analyze the information of the data of interest. Based on domain knowledge, determine whether it matches the expected information value.

5.3 Baseline Model Comparison

Testing with a baseline model, the presence of shortcut learning problems in the current target model can be judged based on the detection results of the baseline model. It is analyzed whether the baseline model will produce unexpected detection results under incomplete feature dimensions. If the detection results of the baseline model are greatly beyond expectations in this case, the risk of shortcut learning in the current application scenario is relatively high. In this situation, it cannot be directly denied that deep neural networks have higher detection capabilities, but it needs to be clearly stated that the detection performance of the model on the dataset and the actual detection capability of the model cannot be confused.

6. Mitigating the Impact of Shortcut Learning

Throughout the entire machine learning process, the problem of lack of out-of-distribution generalization is普遍存在. In the following, some methods for optimizing shortcut learning will be provided. It is important to remember that shortcut learning is an intrinsic characteristic of the learning process and cannot be completely eliminated, but can only be optimized by different methods to reduce its impact.

6.1 Robustness

Adversarial samples are specially designed inputs with the purpose of deceiving machine learning models to make incorrect predictions or decisions. Adversarial samples are very similar to normal inputs and may be indistinguishable to humans, but they can significantly reduce the detection ability of machine learning models. Adversarial samples are usually generated by applying small perturbations to the input data through specially designed optimization algorithms, which are calculated based on the model's loss function.Adversarial attacks are an effective method to analyze the weaknesses of model capabilities.Adversarial samples can be regarded as counterexamples, which can reveal the problems that machine learning models have not yet been able to solve. By using adversarial samples, the generalization ability of the model can be optimized, and adversarial samples can be understood as the weaknesses of the model. Testing the model with adversarial samples can optimize the model's generalization ability. At the same time, regularization techniques can also be used to try to prevent the model from learning shortcuts. These regularization techniques include weight decay, early stopping, dropout, and so on.

6.2 Sample Balancing

Fairness research aims to make machine learning decisions more fair, including ensuring that machine learning does not have decision biases for minority groups or groups with fewer samples. Setting aside political correctness, we can solve the problems of balance and fairness through methods such as sample balancing and weighting. This includes oversampling for categories with few samples and undersampling for categories with many samples.

6.3 Meta-Learning

Meta-Learning, also known as 'learning how to learn', is mainly through the knowledge and experience gained from the past to guide the learning of new tasks, thereby designing machine learning algorithms that can quickly learn new tasks from a series of different tasks. The goal of meta-learning is to learn from many different tasks, discover the common patterns between them, and avoid adapting to changes in new conditions, so that it can quickly adapt and learn when facing new tasks.

7. Thoughts on Shortcut Learning by Shan Shi

7.1 Shortcut Learning is Learning Universal Characteristics

It is clear that the formation of erroneous knowledge after learning is not a failure of the learning process, nor is it a failure of the pattern generalization behavior itself. It is because the pattern generalization did not reach the expected direction of the learning goal, or it may be caused by the defects in the data information relied on by the learning process.

7.2 Proper Guidance of Prior Knowledge for AI Learning

Prior knowledge is information about the task, data, or field that is known before training AI models. This information can affect our model selection, feature selection, model training, and evaluation processes. Prior knowledge can properly guide AI from the following aspects to reduce the risk of shortcut learning:

Feature Evaluation: By utilizing prior knowledge, we can evaluate many features and help the model filter out the features that are substantially helpful. For example, if we know that house prices are mainly determined by factors such as the size of the house, location, and construction year when predicting house prices, we can focus on these features rather than other possibly irrelevant features, such as the latitude of the house and the gender of the homeowner.

Model Constraints: Prior knowledge can help us set reasonable model constraints to prevent the model from learning patterns that we know are wrong or impossible. For example, if we know that prices cannot be negative in the task of predicting product prices, we can add this constraint to the model.

Model Evaluation: Prior knowledge can also help us design more effective model evaluation strategies. We can create some special test cases based on prior knowledge to check if the model has learned the correct patterns rather than superficial and misleading patterns.

Data CorrectionIf our prior knowledge tells us that certain data features may cause the model to produce shortcut learning, we can preprocess the data before training, correct or delete related features.

It should be noted that over-reliance on prior knowledge may also limit the ability of the model, because sometimes the model may discover important patterns or features beyond prior knowledge. Therefore, it is necessary to find a proper balance when using prior knowledge.

7.3 Continuous Evaluation and Dynamic Optimization

Continuous Evaluation (Continuous Evaluation): In the process of machine learning, we cannot simply consider the model trained after completion as the optimal model, as the data distribution may change over time (this phenomenon is called "data drift"), which may lead to a decrease in model performance. Therefore, we need to regularly or continuously evaluate the model to ensure that the model's performance still meets the requirements. Once the model performance decreases, we need to adjust or update the model.

Dynamic Optimization (Dynamic Optimization): Dynamic optimization refers to the process of dynamically adjusting or optimizing the model based on the continuous evaluation results of the model. This may involve adjusting the parameters of the model, updating the training data of the model, or even changing the structure or algorithm of the model. The purpose of dynamic optimization is to make the model able to adapt to changes in data and maintain optimal performance.

In practice, continuous evaluation and dynamic optimization usually need to be used in conjunction. For example, we can set up a monitoring system to regularly evaluate the model, and once the model performance decreases, trigger the optimization process of the model.

During the optimization process, we may need to collect or clean data again, adjust the parameters or structure of the model, or even retrain it. At the same time, in order to achieve the ability of continuous evaluation and dynamic optimization, it is necessary to build an intelligent closed-loop system for the machine learning model, forming a dynamic closed-loop chain of model application, performance evaluation, model optimization, and model update, ensuring that the model can also maintain its detection ability as time progresses. This provides new ideas for discovering and mitigating shortcut learning from the perspective of engineering.

Figure 8 AI Model Closed-loop Optimization

7.4 Horizontal Correlation Between Functional Modules

Under the scenario of network security, from the perspective of attack events, the same network attack behavior will leave traces under different security capability modules, such as brute force attacks will leave a large number of network sessions in the network, leave a large number of login behavior events on terminal devices, and even leave traces in deception defense devices such as honeypots. All these traces point to the same attack behavior. Through the horizontal correlation analysis of security capabilities, the automated analysis results provide a more scenario-based evaluation direction for AI models in security capability modules, and these analysis results are also of great practical value for helping models discover their potential shortcomings and optimizing shortcut learning, as well as discovering the weak points of the model.

Figure 9 Horizontal correlation analysis of safety functions

8. Summary and Prospects

: Shortcut learning is also a form of learning, but it does not learn the complexity of tasks in real-world scenarios from the data, but chooses the simplest and most direct way to learn. This one-sided and incomplete learning induction is outstanding on limited local data, but its performance in real application scenarios is very poor.

As an inherent characteristic of learning, shortcut learning is difficult to eliminate. The research on alleviating shortcut learning can be carried out from the following aspects:

1. In-depth theoretical research: Although we have confirmed the existence of shortcut learning and can observe it in experiments, our understanding of why the model chooses this one-sided learning method and how to systematically avoid this learning method is still limited. Therefore, more in-depth theoretical research will be an important research direction in the future.
2. Improved data evaluation methods: The formation of shortcut learning is closely related to data, therefore, it is necessary to conduct a complete and in-depth analysis of the data before training, to explore the correlation between shortcut learning and certain characteristics of the dataset, and to alleviate the problem of shortcut learning through improving the data evaluation methods.
3. New learning strategies: Many current learning methods, including supervised learning, unsupervised learning, and reinforcement learning, may have shortcut learning problems. Developing new learning strategies, such as self-supervised learning, and other new learning methods.
4. More scientific evaluation indicators: The results of shortcut learning are often excellent on the training data, therefore, we need more scientific evaluation indicators to measure the performance of the model in the face of unknown samples, so as to more accurately identify and avoid shortcut learning.

9. References

[1] Geirhos R, Jacobsen J H, Michaelis C, et al. Shortcut learning in deep neural networks[J]. Nature Machine Intelligence, 2020, 2(11): 665-673.

[2] Scimeca L, Oh S J, Chun S, et al. Which shortcut cues will dnns choose a study from the parameter-space perspective[J]. arXiv preprint arXiv:2110.03095, 2021.

[3] Baldock R, Maennel H, Neyshabur B. Deep learning through the lens of example difficulty[J]. Advances in Neural Information Processing Systems, 2021, 34: 10876-10889.

[4] Geirhos R, Rubisch P, Michaelis C, et al. ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness[J]. arXiv preprint arXiv:1811.12231, 2018.

[5] Beery S, Van Horn G, Perona P. Recognition in terra incognita[C]//Proceedings of the European conference on computer vision (ECCV). 2018: 456-473.