First, background
1. Background and needs of threat intelligence library construction
1) Changes in the trends and needs of new era attack and defense.
With the development of the Internet, especially mobile Internet, the network environment is becoming more and more complex, and different attack behaviors are more industrialized and gangsterized, with more diverse and complex intrusion methods. Traditional security strategies centered on vulnerability defense are difficult to detect, intercept, and analyze new, persistent, and advanced threats in a timely and effective manner. The security attack and defense needs are gradually evolving from the traditional, vulnerability-centered to an active, intelligence-centered construction model.
2) Requirements of UnionPay's own business.
Malicious behaviors such as false registration, batch card binding, and malicious刷单刷券 can affect the daily operation and marketing promotion of corporate products, while the traditional financial industry itself lacks security data related to the Internet and needs high-quality intelligence data to support related risk prevention and control work. Therefore, it is necessary to combine security with business, introduce threat intelligence related to risk control and other business, and help improve the company's risk prevention and control capabilities.
2. Existing threat intelligence library
The data sources of the threat intelligence library are divided into three aspects, including internal intelligence, professional institutions, and industry alliances. Internal intelligence includes the analysis of traditional security equipment interception, background SIEM and other security analysis systems, as well as the discovery of business risk control systems; professional institutions will provide multi-source intelligence of their respective professions, complementing each other according to the professional characteristics of each institution, and making judgments on data in the event of intelligence conflicts; the threat intelligence sharing of industry alliances is currently under research and exploration. After the data of these three aspects are aggregated, a local library is formed, processed, and the final data results are fed back to the top-level application layer, and pushed to background applications, defense equipment, or manual calls. After considering the needs of the threat intelligence library, we conducted research and testing on domestic related manufacturers, and finally chose the local threat intelligence management platform under Beijing Weibu Online Technology Co., Ltd. as the carrier platform for the UnionPay threat intelligence library.
The establishment of a threat intelligence library can assist us in timely detecting various tactics, methods, and behavioral patterns of hackers or malicious attackers, mastering the latest attack trends against payment scenarios, and efficiently preventing and handling various cyber security incidents; on the other hand, high-quality intelligence data can provide strong support for risk prevention and control, achieving precise identification of external network abnormal access behaviors. It is of great value to the defense team's security risk control team in timely grasping the security situation and making correct responses.
2. Research Objectives
In the application process of threat intelligence, we have found that there are still some problems to be solved.
1. Legal and Orderly Intelligence Sharing
After the intelligence database is established, the demand for intelligence sharing is immediately proposed. Intelligence sharing with internal companies, industry institutions, and others can not only quickly achieve the improvement of threat awareness and risk sharing, but also the legal and orderly sharing of intelligence is also conducive to the healthy and sustainable operation of the entire ecosystem, reducing operating costs.
2. Private Intelligence Production
UnionPay is a typical multi-office, multi-organization collaborative defense structure, with a large number of security equipment and sensitivity to attacks, resulting in a massive amount of alert information. How to obtain real attack behavior from a massive amount of alert information is a big challenge. At the same time, external threat intelligence data cannot fully support the detection, blocking, and traceability analysis of real attacks. When attackers perform jumping attacks on peripheral assets, it may also lead to difficulties in joint defense. In these scenarios, we have strong demands for threat intelligence data and threat intelligence production.
In order to achieve these two goals, we have conducted research on intelligence sharing technology and full traffic threat hunting.
3. Research on Intelligence Sharing Technology
1. Basic Logic of Intelligence Upload and Distribution
Deploy an intelligence management platform in the existing intelligence database, deploy branch intelligence management platforms in various branches, push intelligence to each branch, and automatically report private intelligence from each branch to the control center. The control center can control whether to distribute the private intelligence reported by each branch for secondary distribution.
The overall solution is mainly composed of two parts: the core intelligence management platform and the downstream intelligence management platform:
The main functions of the core intelligence management platform include receiving cloud intelligence for intelligence query; distributing intelligence to downstream intelligence management platforms; receiving intelligence reported by downstream intelligence management platforms; and entering self-owned intelligence for intelligence query and distribution.
The main functions of the downstream intelligence management platform include the entry of self-owned intelligence for intelligence query and reporting; reporting local entered intelligence to the core intelligence management platform; and receiving intelligence distributed by the core intelligence management platform.
For institutions with their own intelligence platforms, the intelligence management platform can adopt the industry standard of peers for intelligence exchange.
2. Complete Intelligence Circulation Process
Based on the structure of the UnionPay threat intelligence database, the first step is to automatically generate and screen intelligence from internal logs, industry intelligence, and alliance data, and pull community intelligence and regular intelligence updates from the cloud. After enabling the collaborative analysis function, the threat intelligence database will automatically pull additional information through an encrypted channel from the cloud and merge it with the local simplified private intelligence, forming a complete threat intelligence form. Local private intelligence can be directly viewed, deleted, exported, and applied to third-party systems.
During the collaborative research process, the threat intelligence team members will jointly complete the attack organization methods, and finally form an independent attacker portrait database. The intelligence management platform supports horizontal comparison of multiple intelligence source data of a single intelligence, comprehensive assessment of the overall picture of intelligence from multiple perspectives, and then shares the threat capabilities through Restful API, realizing push, reporting, and controllable secondary distribution between the central control and branches.
The threat intelligence library will automatically supplement auxiliary information through unified storage, retrieval, and comparison of machine-readable intelligence, advanced reports and monthly reports, third-party machine-readable intelligence, private intelligence manually imported by users, and global multiple open-source intelligence. It uses unified lifecycle management intelligence from generation, use, silence, and extinction throughout the entire process.
3. Technical focus and difficulties
During the development and deployment testing of the project, many important nodes and technical difficulties were also found. After fully discussing and evaluating in line with actual needs, they were broken through one by one, including:
1) How to build and implement a 24*7 fully automatic multi-institutional intelligence sharing and consumption mechanism
The target consists of the process of issuing intelligence by the core intelligence management platform, writing, reporting, and consuming intelligence by the downstream intelligence management platform, among which the key point is to clarify who writes intelligence to the downstream intelligence management platform, who consumes the intelligence of the downstream intelligence management platform, and how it is consumed.
"Who writes intelligence to the downstream intelligence management platform": In addition to the core intelligence management platform, downstream institutions actually do not have direct and accurate intelligence that can be written. What can be written is the alarms generated by security devices deployed in the network environment of downstream institutions. When many downstream institutions report alarms, the core intelligence management platform can generate industry threat intelligence in real-time and dynamically according to intelligent strategies, for example: when the network of downstream institutions is not continuous, an external IP attacks multiple downstream institutions continuously, the attack IP is quite suspicious, and according to the strategy, it is produced as industry intelligence and pushed to other downstream institutions that have not yet perceived the risk.
"Who consumes the intelligence of the downstream intelligence management platform and how": Some sensitive businesses such as payment ports and login ports can use pushed intelligence for active defense. Even if users from attack IPs provide the correct username and password, the business still requires上行短信验证 and下行短信验证, and for certain types of internet industry lottery activities, it can also reduce the chance of winning in the network segment. It ensures business security without offending users or interrupting business.
2) How to solve the intelligence conflict and false alarm in branches
The intelligence conflict issue was not fully considered at the beginning of the project, but during the test deployment, we found that some IPs were marked as whitelist and malicious addresses by different branches, causing downstream institutions to ask questions. After investigation, the main reason was found to be: Institution A marked its office network exit address as a whitelist, but there was a worm virus within its office network, which scanned other institutions through the office network exit, and some of them were marked as malicious.
The problem of false positives in intelligence mainly due to multiple downstream institutions purchasing the same manufacturer's security products. After the false positives of similar security products are uploaded to the core intelligence management platform through the downstream intelligence management platform, it causes misjudgment problems. Such problems are planned to be further avoided in the future by adding the identification of security device types.
3) Problem of normalizing security device log formats
Due to the diversity of security equipment purchased by various downstream institutions, it inevitably leads to the problem of log normalization, which currently does not have a very perfect solution. Using technical means for unified format processing at the entrance of the downstream intelligence management platform is currently a feasible solution.
Four, full traffic threat hunting
1. Definition and process of threat hunting
Threat hunting is the self-inspection of enterprise institutions based on threat intelligence. Threat hunting requires the early acquisition of some basic vague characteristics and clues of the attacker, that is, threat intelligence, and then based on the intelligence, through side-channel traffic detection, system log detection, or host behavior detection to explore ongoing attack behaviors or compromised internal network hosts. Its effectiveness varies with the accuracy, timeliness, and diversity of clues or intelligence.
Threat hunting process (flowchart)
2. Key points and difficulties in threat hunting technology
The key points and difficulties of threat hunting technology mainly come from the following three aspects:
1) How to obtain accurate, timely, and diverse threat intelligence with a large amount of auxiliary context;
To meet the needs of threat intelligence accuracy, timeliness, and diversity, the threat intelligence library adopts the collection and management of multi-source threat intelligence. The local intelligence platform should at least be able to load four types of intelligence, have two intelligence capabilities, including multi-source machine-readable intelligence, advanced human-readable reports, vulnerability intelligence, and custom intelligence, etc. Intelligence capabilities should include intelligence agent capabilities, local production intelligence capabilities, etc. In addition, intelligence sharing and cascading are one of the key capabilities to ensure the successful implementation of threat hunting.
2) How to automate the screening of more clues expanded and further convert effective clues into intelligence;
Intelligence assistance in context is a key point that cannot be ignored. The richness of the machine-readable intelligence fields determines whether the threat intelligence can actually be implemented. The existing intelligence accurately describes the attack behavior or gang characteristics, including but not limited to discovery time, port protocol, severity level, URL, related samples, domain owner, family tags, and targeted industries, among more than 30 fields. In comparison, simply providing a blacklist with no auxiliary context information, open-source intelligence is almost impossible to be used in financial production environments. Automation screening and effective clues converted into intelligence mainly test our private threat intelligence localization production capacity in the threat intelligence library.
3) How to implement threat intelligence in side-channel traffic detection, system log detection, or host behavior detection products.
This requires us to open the interconnection of external SIEM/SOC data platforms, firewalls or WAF devices, host management products, routing and switching equipment, and other security products. And analyze the interconnection of devices in specific scenarios, for high-risk intelligence hits, call downstream devices to implement blocking, and promote intelligence from detection and response scenarios to comprehensive security protection.
3. Full traffic threat hunting technology
Through the system planning of data source access, underlying hardware and software architecture, big data standardized processing, model algorithm, and top-level security business application, relying on core technologies such as threat intelligence big data knowledge graph technology, advanced intrusion detection and analysis technology, hacker profiling and tracking and tracing technology, intelligence data sharing technology, and automated orchestration and disposal technology of response strategy, a complete closed-loop solution covering threat detection and analysis, threat event clue extraction, attacker profiling and tracing analysis, threat blocking response and collaborative linkage is constructed. It forms detection, analysis, response, tracing, and predictive capabilities driven by threat intelligence data.
1) Data source access
Comprehensive collection of both inbound and outbound traffic at the enterprise-related network boundaries, and horizontal east-west traffic between various internal network areas. All full data generated within the network are collected in real-time by the traffic collector through the traffic mirroring method. The main collected data includes traffic information, traffic information, and terminal log data. Among them, traffic information includes 8 protocols such as DNS, HTTP, TCP, SMTP, POP3, RSYNC, RDP, and TFTP; DNS logs include information such as the parsed domain name, query type, source address and port, and destination address and port in the bidirectional logs of Request and Response. Based on this, a full traffic threat continuous detection access data is formed.
Based on threat intelligence for log correlation analysis
2) Hardware and software and data parsing basic components
Relying on big data hardware and software basic components such as microservices, distributed clusters, massive data storage, and message queues, a
For the data access form of accessing the original traffic, the system needs to parse the binary data in the original traffic into structured DNS packets. After investigation and testing of common traffic analysis tools (including Bro, Suricata, etc.), it was found that they are not suitable for DNS parsing scenarios. The reasons are as follows: 1. The architecture design of these tools is aimed at full traffic parsing, and in order to be compatible with other protocol features, a large part of the system resources are used for data stream session maintenance, traffic caching, etc., which are unnecessary system overhead in the DNS parsing process. 2. In addition, the length of DNS data packets is small, and the QPS is high under the same traffic, and the performance test results on Bro and Suricata are not ideal. Therefore, this system adopts a self-developed packet capture module in terms of technical route, and performs performance optimization for the protocol characteristics of DNS.
3) Threat detection and analysis model
This architecture has ten types of threat detection models, including big data collision models based on threat intelligence, DNS stealth channel detection models, dynamic sandbox detection models, DGA random domain generation detection models, internal network lateral movement detection models, deep learning algorithm self-learning detection models, etc. The detection coverage stage includes sniffing, vulnerability exploitation, weapon delivery, remote control, lateral movement, external attacks, operations (ransomware, mining, data theft), and the identified types of attacks and threats include at least: port scanning, application scanning, subdomain brute force, remote overflow, WEB attacks, SQL injection, configuration vulnerabilities, command injection, CC destructive attacks, XSS, SSRF, file leakage, directory traversal, brute force, web malware, malware execution, webshell, zombie and worm detection, high-risk vulnerability exploitation, ransomware, mining malware, advanced APT organizations, stealth channel communication, etc.
Relying on visualization and correlation analysis technology, conducting correlation analysis on threat intelligence, network original logs, terminal logs, and alert logs, completely restoring the attack path from the attacker's perspective, completely depicting the network behavior of the controlled host from the perspective of the controlled host, and fully presenting the entire picture of the threat.
V. Output and significance of the research
1. Ensuring the continuity of security construction
Adding more capabilities to the existing intelligence management platform and threat detection platform, continuously upgrading based on existing capabilities, ensuring the continuous operation of existing capabilities, seamless integration with new technologies, and ensuring continuous and effective security construction.
2. Increasing the visibility of threats across the entire network
On the basis of the original threat detection platform's ability to detect compromised hosts at the boundary, enhancing the coverage of traffic, covering internal network traffic, and applying traffic file reconstruction technology, engine and dynamic sandbox technology, machine learning technology, etc. to improve detection capabilities, analyzing and hunting the threat attack process, and improving the visibility of threats and the entire attack process within the internal network.
3. Enhancing response capability through industry intelligence sharing
On the basis of the original intelligence management platform's ability to integrate intelligence and provide intelligence detection interfaces, enhancing the industry intelligence sharing capability, providing batch access to mine intelligence interfaces, and building the ability to link with existing security devices, and automatically responding based on threat intelligence.
4. Significance of the construction and development of the whole network cyber threat intelligence sharing system
Building on the capabilities of the early threat intelligence center, enhancing traffic monitoring and threat hunting analysis capabilities, accurately locating the entire attack process; at the same time, improving the ability to mine intelligence, and exploring and studying the industry intelligence sharing mechanism, laying a theoretical and technical foundation for future practical applications.
*Author: Baian Security Team, please indicate the source as FreeBuf.COM when reproduced
评论已关闭