Dealing with Outliers

How to Find Fraudulent Transactions in a Real-World Dataset

Oleg Kremnyov, QA Engineer; Mikhail Averbukh, Software Engineer; and Ivan Kuzmin, Software Engineering Manager; Intel Corporation

One challenging―but also very important―task in data analytics is dealing with outliers. We generally define outliers as samples or events that are inconsistent with the rest of data population¹. An outlier often contains useful information about abnormal characteristics of the systems and entities that impact the data generation process².

Common applications for outlier detection algorithms include:

• Intrusion detection systems
• Credit card fraud
• Interesting sensor events
• Medical diagnosis

In this article, we’ll focus on one of the most common applications of outlier detection—credit card fraud. With some simple outlier detection approaches, it’s possible to find 75 to 85 percent of fraudulent transactions―with false alarms less than 1 percent of the time―on a real-world dataset.

Defining the Approaches

There are two basic approaches to detecting outliers:

1. Supervised, where we use fraudulent and non-fraudulent examples to predict the class of a new observation
2. Unsupervised, where we don’t have labeled examples and detect fraudulent examples as outliers or abnormalities

Breaking it down further, there are three basic supervised approaches:

1. Neural networks
2. SVM
3. Logistic regression

Unsupervised approaches include:

1. Statistical-based techniques like BACON* outlier detection
2. Multivariate outlier detection
3. Clustering-based techniques^{3, 4} like k-means, expectation-maximization (EM), and DBSCAN*⁵, an algorithm that detects noise in data that can be treated as outliers⁶.

We’ll compare these techniques and evaluate each methodology based on certain design criteria. Accuracy metrics will include both detection and false positive rates⁷.

It’s important to remember that fraud detection is about more than capturing fraudulent events. It also requires capturing these activities as quickly as possible—so an algorithm’s performance is an important metric⁸. In our test, we’ll use some popular machine learning libraries, including Intel® Data Analytics Acceleration Library (Intel® DAAL), that implement algorithms for outlier detection to compare results for both accuracy and performance.

Figure 1. Dataset attributes

Dataset Description

We used a dataset⁹ from Kaggle*, a platform for predictive modeling and analytics competitions in which companies and researchers post their data and statisticians and data miners from all over the world compete to produce the best models¹⁰. We chose the most popular real-world dataset on credit card fraud detection from Kaggle¹¹. It contains 31 numerical input variables (Figure 1).

The time attribute represents seconds elapsed between each transaction and the first transaction in the dataset. Amount represents the transaction amount, as the name implies. The remaining 28 features are the principal components obtained with PCA. The original features are not provided due to confidentiality issues.

Class is the response variable. It equals 1 in case of fraud and 0 otherwise. The dataset is highly unbalanced because fraud transactions are far less common than normal ones. Frauds represent only 0.172 percent of all transactions (492 frauds out of 284,807 transactions).

Let’s take a look at conditional distributions of some of principal components (Figures 2 and 3). Principal components are not correlated, so dropping those with similar conditional distribution for fraudulent and normal examples could help us build more accurate models. We’ll use a point-biserial correlation coefficient¹² to measure the relationship between each principal component and target variable (Table 1). We’ll use this information later for feature engineering.

Credit Card Fraud Detection Approaches

Now we’ll apply different outlier detection techniques to the data, evaluate their accuracy and performance, and compare them. For credit card fraud detection, we’ll use both supervised and unsupervised methods.

Figure 2. Attribute with similar distribution between two types of transactions

Figure 3. Attribute with different distribution between two types of transactions

Table 1. Point-biserial correlation coefficients for principal components

Unsupervised Approaches

Statistical approaches were the earliest algorithms used for outlier detection¹³. We’ll start from outlier-detection-specific algorithms―multivariate outlier detection and BACON* outlier detection. A great advantage of these methods is that they are so computationally efficient that it’s easy to apply them to large datasets¹⁴. We achieved the best results with six principal components with a high point-biserial correlation coefficient (V11, V12, V14, V16, V17, V18) as features. We used implementations from Intel DAAL and the R* package robustX*. Multivariate outlier detection gave better results for accuracy than BACON outlier detection (Table 2).

Table 2. Statistical-based outlier detection approaches

Clustering and outlier detection have a complementary relationship. In clustering, the goal is to partition the points into dense subsets. In outlier detection, the goal is to identify points that don’t seem to fit naturally into these dense subsets². Some clustering algorithms, like DBSCAN*, provide sub-products, which can be considered outliers.

There are many clustering-based approaches to outlier detection^2,3,4,15. We’ll apply some of them in our tests. Clustering is inherent in credit card fraud because perpetrators usually produce a group of fraudulent transactions¹⁶.

There are two types of outliers, local and global. Different clustering-based algorithms can detect both types. The attribute values of a global outlier are outlying with respect to the values taken by the majority of the data points. The attribute values of a local outlier are extreme when compared to those of its neighbors¹⁷.

To detect global outliers, we can perform clustering on a dataset and define some clusters as outliers¹⁸ (Table 3). We’ll perform clustering with k-means and EM, treating small clusters as outliers¹⁹. Values of k and n_components were obtained by grid search from range [2, 3, …, 50]. We used the same initial centroids for k-means for Intel DAAL and Scikit-learn*, obtaining them with k-means++ initialization using the implementation in Intel DAAL. We achieved the best results using 17 principal components with high point-biserial correlation coefficients as features. With these methods, we achieved a relatively low false positive rate.

Table 3. Clustering-based methods for global outlier detection

We’ll also apply clustering techniques that can detect local outliers²⁰. A naïve approach is to cluster the transactions using the k-means algorithm²¹ and treat points far from cluster centers as outliers (Table 4).

We obtained a good rate of detections. However, the number of false alarms is high. The reason is that the k-means algorithm is extremely sensitive to outliers, which have a large impact on cluster configuration. As a result, data points that should be declared as outliers are masked by the clustering. Thus, we need to use a robust version of k-means that can gracefully handle the presence of outliers―k,l-means¹⁵ (Table 5). Like k-means, this algorithm requires initial centroids. We’ll compute them with different initialization algorithms available in Intel DAAL―random, k-means++²², and k-means||²³―and compare the results.

Table 4. K-means-based method for local outliers detection

 

Table 5. K,l-means algorithm results

We can see that this approach provides better accuracy than naïve k-means. Results are different for different initializations. The random method proved to be the best for this problem.

We’ll also use a density-based clustering algorithm that can detect noise in data (Table 6). A point is detected as noise if it differs significantly from its neighbors. A natural assumption is to treat noise data as local outliers. We used the same six features as for statistical methods. Values of parameters were obtained by grid search. We used the R package dbscan*. We reached a good level of detected frauds with a moderate level of false alarms. However, the performance is significantly worse compared to previous approaches.

Table 6. Dbscan for outlier detection

Supervised Approaches

Supervised learning methods require training data that contain examples of each class—in our case, normal and fraudulent transactions². We’ll divide our dataset into training and evaluation sets, with 80 percent of the data used for training and 20 percent used for model evaluation.

We start with the neural network (Table 7) proposed in the Kaggle kernel²⁴. A simple multi-layer perceptron with three hidden layers was used by the kernel author. Multi-layered perceptrons (MLPs) are widely used neural networks for classification as well as outlier detection²⁵.

Table 7. Neural network

Another supervised learning algorithm used in outlier detection is logistic regression¹⁶ (Tables 8 and 9). We’ll use 17 attributes with the highest point-biserial correlation coefficients (Table 1) and the Amount feature, which was pre-processed with Z-score normalization. From the histograms²⁴, we can see that fraud and normal observation are linearly separable due to the use of logistic regression. Our dataset is unbalanced, so use of random undersampling could help to build a model that detects frauds more accurately. This omits many non-fraudulent examples from training, so we’ll use a voting classifier²⁶, with each estimator trained on a randomly under-sampled training dataset. (The regularization parameter, C, was set to one after experimentation with a wide range of values.) With this technique, we improved the detection rate by 3 percent with almost the same level of false alarms.

Table 8. Logistic regression

Table 9. Ensemble of logistic regression models

Let’s see how support vector machines (SVM), another supervised learning algorithm which can also be used for outlier detection¹⁶, works on the same data. We run SVM with 100,000 iterations, a linear kernel, and C=1. (These parameters were obtained with a grid search.) The same features were used as for logistic regression. Use of SVM ensembles improves the detection rate significantly while giving a low false alarm rate (Table 10).

Evaluation of the Different Anomaly Detection Approaches

There are a variety of unsupervised approaches for fraud detection. We applied a few of them in this article. Outlier detection-specific algorithms, k,l-means and DBSCAN, achieved the best detection rates, reaching 85 percent of frauds detected, while use of k-means to detect clusters of fraudulent transactions gives the lowest number of false alarms. The sooner fraudulent transactions are detected, the more losses can be avoided by stopping transactions made with fraudulent credit cards¹⁶―making the performance of the algorithm another key metric. The final choice of fraud detection approach will depend on many factors, including the cost of the fraudulent behavior detected and the cost associated with stopping it⁷. Unsupervised methods can be useful in situations where there is no prior knowledge of classes of observations.

Table 10. SVM

When we have prior knowledge of classes of observations, we can use supervised learning algorithms. We used techniques for unbalanced classification, like undersampling, to help increase the number of frauds detected. Ensembles of SVM classifiers give the highest detection rates, while random forests have the lowest false alarm rates. Although supervised methods like SVM gave us significantly better accuracy compared to unsupervised ones, use of them can pose problems:

• Accurately labeled training data might be prohibitively expensive to obtain²⁵ and a data scientist has to be confident about the true classes of the training data used to build the models.
• The time to train a model can be high depending on the number of samples and the number of features per sample.
• Unlike unsupervised approaches, supervised learning cannot detect new types of fraud without retraining^16,27.

As we’ve shown, choosing the right machine learning algorithm for a given application is a non-trivial problem that requires a lot of thought.

Configuration Information

Hardware:

• Processor: Intel® Xeon® Platinum 8168 processor @ 2.70GHz
• Core(s) per socket: 24
• Socket(s): 2
• Memory: 63 GB

Software

• Intel® Data Analytics Acceleration Library 2018 Gold
• R version 3.4.1
• Scikit-learn version 0.19.1
• robustX package 1.2-2
• dbscan package 1.1-1