2.Unauthorized Access of DataNode Node Restful API

I. Introduction

Hadoop is a distributed system framework released by the Apache Foundation, which performs distributed processing through the MapReduce algorithm. Hadoop is responsible for the lowest-level operations in the big data environment, MapReduce is responsible for data processing, and HDFS is a special file system used to store data in Hadoop. Above these two is Yarn, which is the resource manager of the Hadoop cluster. As shown in the following figure:

Figure 1 Hadoop Operation Framework Diagram

Big data has渗透 into every industry and business function field today, playing an important role in the company's business. At present, the company's information system has been built and connected to the Hadoop big data-related business.

II. Description of the Phenomenon

The big data platform generally has HDFS, MapReduce, and YARN as basic components, and uses components such as Mongodb and ZooKeeper on the upper layer. By default, the components hdfs and MR of the Hadoop big data platform will listen to the following ports:

Table 1 Listening List

In the actual business process, third-party application components such as ZooKeeper and Mongodb will also be running.

During the deployment process of the big data platform, the following vulnerabilities generally exist as shown below:

1. Unauthorized access to the NameNode node's WEBUI

By default, the NameNode's WEBUI is in an unauthenticated state. By accessing the 50070 port of the NameNode WebUI management interface, you can download any file in the file directory browsing area.

Verification method as follows:

Access the 50070 port of the NameNode node using a browserhttp://x.x.x.x:50070Then visit the directory browsing page to directly download the file.

Figure 2 Namenode Unauthorized Access Verification-1

Figure 3 Namenode Unauthorized Access Verification-2

2.Unauthorized Access of DataNode Node Restful API

By default, the API interface of DataNode is in an unauthenticated state, and the API listening port 50075 of DataNode is open to the outside. Attackers can operate the data stored in HDFS through the Restful API provided by HDFS.

Verification method as follows: use hdfsbrowser.py to detect NameNode.

Figure 4 Datanode Unauthorized Access Diagram

3.Anytime Access to Cluster Environment Configuration File

Since the overall security protection of Hadoop is relatively weak, the cluster configuration files have not been effectively protected. Tools can be used to directly obtain the environment configuration files of the hadoop cluster without authorization.

Verification method as follows:

Figure 5 Datanode Unauthorized Access Diagram

4.Leakage of Third-party Management Console Configuration File and Algorithm Code

Cloudera Manager, a third-party Web management platform for Hadoop clusters, has an unauthorized download vulnerability. By traversing the ID number, user configuration project configuration files and MapReduce algorithm files can be directly downloaded.

Verification method as follows:

By traversing the ID number of services in the URL with hackbar, you can obtain the compressed package of the configuration file and the source code of the algorithm.

Figure 6 Arbitrary Download of User Configuration File

5.Leakage of Cluster Configuration File

Since the cluster is not enabled by default, it can directly request /conf in the native WebUI (HDFS WebUI/JobHistory/ResourceManager) to directly obtain the configuration file of the cluster.

Verification method:

Access the 8088 port of the cluster directly through the browser, the access path is /conf, and the configuration file can be obtained.

Figure 7 Configuration File Leaked

6.Unauthorized Access of ZooKeeper Component

The hadoop big data platform uses ZooKeeper as the service discovery registry center, but it has an unauthorized access vulnerability that can cause any user to access and read configuration data without authorization when the network is reachable.

Verification method:

Using the command echo envi|nc x.x.x.x 2181, if there is a vulnerability, it will return the current configuration information of the ZooKeeper cluster.

Figure 8 Component Unauthorized Access

7.Unauthorized Access of Mongodb

Under the default configuration of MongoDB database, there is no permission verification, and the database can be accessed remotely without a password through the default 27017 port.

Verification method:

Using the Mongodb-databases script of nmap, it can be identified whether the Mongodb database is in an unauthorized access state.

Figure 9 Unauthorized access to Mongodb components

Figure 10 Unauthorized access to Mongodb components

3. Processing process/main methods

The above problems occur because hadoop did not fully consider the security authentication issues at the beginning of the design, although it supports authentication, it is not enabled by default.

Security strengthening plan:

1) Enable Kerberos authentication throughout the entire hadoop cluster, and the method varies depending on the cluster, so please refer to the official documentation: http://hadoop.apache.org/docs/r2.7.3/hadoop-auth/Configuration.html;

The following is an example:

Modify hadoop's configuration file.

Figure 9 hadoop's configuration file

2) Close unnecessary ports for external service provision, or use iptables for access restriction;

3) Timely upgrade the third-party management components of the hadoop cluster, such as Cloudera Manager, ranger, etc.

4) For unauthorized access to ZooKeeper, configure network access restrictions and add authentication configuration. You can refer to the following configuration:

a) Add an authentication user
Command: addauth digest 【username】:【password】
Example: addauth digest user1:password1
b) Set permissions
Command: setAcl 【path】 auth:【username】:【password】:【permissions】

Example: setAcl / auth:user1:password1:crwda

Note: CREATE, READ, WRITE, DELETE, ADMIN, that is, increase, delete, modify, query, and management permissions, abbreviated as crwda
c) View Acl settings
Command: getAcl 【path】

Example: getAcl /

d) After authorization is completed, when connecting to the zookeeper service with another session, the following prompt will appear:

Figure 10 Authentication Authorization

5) For unauthorized access to Mongodb, configure network access restrictions and add username and password authentication. The following is a reference plan.

1. 1. Add a user to system.user and start authentication
   2. Connect to the Mongodb database through CLI and add a user with admin privileges (take 'admin:adminz' as an example)

Four, Experience Summary

Closely track the application of new business and new technologies in information systems, and keep up with the disclosure of information security vulnerabilities in a timely manner. When establishing or connecting related hadoop business, reinforce the hadoop cluster through the aforementioned strengthening methods, which can effectively prevent the leakage of hadoop data and improve the security of the information system.