In this world of ever evolving malware, its behavior and the methods used to fight against it become essential knowledge for anyone involved in the IT industry. This article aims to shine a light on the aspects stated above, in order to help computer users stay safe and to stimulate the development of better security solutions.

Shall we ? . . .

I. OS processes: why are they so important?

• when a file is executed, it becomes a process among the other already existing ones
• processes have loaded modules (DLLs) and threads
• each process can write into the virtual memory of another process (code injection)
• many malicious entities use clean processes (e.g. cmd.exe, regedit.exe, explorer.exe) to execute their payloads
• they are directly used by the malware detection solutions

II. Which are the malware detection methods?

III. Which are the challenges and goals of solutions based on dynamic detection?

• a single action performed by a process is usually insufficient to distinguish between malware and legit applications
• advanced malware can avoid detection by separating malicious actions into multiple processes through process creation or code injection => necessity to monitor malicious groups of processes at once instead of individual ones.
• system remediation after detection: find which process created / handled the malicious files, check if those files were executed, if the process compromised other processes, or if any registry changes were made => in the end, everything should be reverted
• definitive goal: classify processes as clean (Negative) or infected (Positive). The result can be:
  • True Positive (TP) – the entity was correctly classified as malicious
  • True Negative (TN) – the entity was correctly classified as clean
  • False Positive (FP) – the entity was erroneously identified as malicious (false alarm)
  • False Negative (FN) – the entity was erroneously identified as clean

IV. But what do we need for dynamic detection?

• heuristics – each heuristic is an algorithm (function) that analyzes the actions intercepted through monitoring; in case of any peculiar action identified, the heuristic triggers an alert which is passed to an evaluation engine
• interception mechanisms of the OS (for file system, processes, API etc., which entail complex components – drivers, DLLs) that provide the actions to the heuristics – we’ll focus on Windows here, as this is the most targeted OS

V. Which are the interception (filtering) mechanisms of the Windows OS?

• File System Filtering – uses file system drivers, kernel-mode components that can monitor / modify / prevent file system operations. They are of 2 types:
  • Legacy – problematic, they must handle every type of I/O operation that a file system driver performs
  • Minifilter drivers – more reliable, performant and with a simplified development process. They can choose which file system operations to monitor and can register pre-operation and post-operation callback routines that are called by the OS each time the monitored operations are performed.
• Process Filtering – a minifilter driver can also be used to monitor the creation and termination of processes. It can call PsSetCreateProcessNotifyRoutineEx API to register a callback routine. Whenever a new process is created, the registered routine is called before the initial thread of the process begins running. The routine can also terminate the process.
• Registry Filtering – a minifilter driver can also be notified whenever an operation that targets the registry is performed by registering a callback routine using the CmRegisterCallbackEx API. It is called before and after the registry operation is completed and receives specific info about that operation as parameters.
  By writing in registry keys like:
  HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\Run , or
  HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Run ,
  the malware ensures that it will run at log-on, gaining persistence.
• WIN API Hooking – intercept API calls by redirecting their execution to an associated routine, called hook, to analyze / extract info about the call (parameters or return value) => the process calls a hooked API. Methods:
  • Import Address Table (IAT) Hooking – the destination addresses of the functions that are statically imported by the process are altered so that they point to the hook functions
  • In-Line Hooking – alter the instructions of the functions to be intercepted, in memory, at run-time, by changing the first several bytes (to transfer the execution to the associated hook routine). When the routine completes processing, control is transferred back to the original API (change the first several bytes back or invoke a function that contains those bytes – called Trampoline).
  • Detours – the Detours library is provided by Microsoft to intercept functions similar to in-line hooking, by applying code dynamically at run-time. The detour function can be used to replace the hooked function or extend its functionality.
• Event Tracing for Windows (ETW) – allows the interception of system events generated by the Windows OS in real time. As ETW is a kernel-level feature, it is not susceptible to some type of evasion that affects WIN API Hooking, such as hook removal, or direct syscall usage. It can also avoid application crashes caused by API hooking.
  Disadvantage: code injection and other memory manipulation actions cannot be reliably identified without the Microsoft-Windows-Threat-Intelligence provider, that was included in Windows 10.
• Anti-Malware Scan Interface (AMSI) – feature included in Windows 10 that contributes to real-time malware detection. Apps can use it to invoke the security solution available on the system to analyze various objects (files, memory areas, streams, URLs, IPs, etc.). It’s especially useful in detecting malicious scripts that can be highly obfuscated / difficult to detect. The Windows 10 script hosts (PowerShell / Windows Script Host / JavaScript engine) integrate AMSI and invoke scans of the code, which must be and is de-obfuscated right before execution.

VI. Which are the common actions performed by malware?

There are 3 actions, very common in malware, that are especially encountered in installers & uninstallers ( => risk of False Positives): writing executable files in the System / Windows / Temporary folders, executing the created files and registering an executable to run at start-up => there’s a need to distinguish between ordinary executable files and installers/uninstallers. Legit installers are usually generated by common software installation packages, and can be identified by scanning the file for certain signatures.

VII. How to smartly evaluate the intercepted actions?

A generic approach is using a scoring engine, which contains heuristics able to detect samples that use various new combinations of malicious actions. A set of predefined features are extracted (from the executable file or based on the actions of the analyzed process). Each feature has an associated score, that is used to compute a general score for the sample. If that score exceeds a predefined threshold, the sample is either categorized as a certain type of malware or as clean.

When thinking of complex malware scoring mechanisms, one may believe that the solution can be implemented using artificial intelligence (AI). An AI algorithm performs well enough in face recognition, for example, because human faces do not change their definitory characteristics over time. But malware evolves at a rapid pace, to use the latest features provided by operating systems and programming languages, as well as to exploit the latest unpatched ( zero-day ) vulnerabilities. Therefore, the scoring engine must be easy to understand and maintain, precise, predictable to changes, and needs to be updated very quickly. This is NOT easily achieved with an AI algorithm, where the training is time consuming and the results cannot be anticipated, not to mention the constantly required re-training. Also, adding a new heuristic may damage the entire scoring mechanism. Furthermore, an appropriate training set for dynamic malware detection is almost impossible to find.

In practice, adapting the scoring engine to a new threat should require writing a couple of heuristics and calibrating the scores and weights only for them. More exactly, when another malware technique appears in the wild, a security researcher needs to test the security solution against samples or proof of concepts (POCs) that exhibit the new behavior. He / She may also manually capture the behavior of the processes by simply using a tool like Process Monitor. If the current detection model does not identify the new malicious technique, it’s a clear sign that the current model must be extended.

In a broader perspective, such a solution should be integrated in a modern security application, together with other components such as URL blocking, firewall, classic AV signatures, etc.

VIII. Now, which are the smart tricks used to stay under the radar?

• solutions that use dependency graphs constructed from API / system calls, may be evaded by replacing a call sequence with its semantic equivalent and/or inserting redundant calls
• instead of executing all the malicious actions in a single process, distribute the payload to multiple, distinct processes, to be executed over a long period of time. Because behavior-based detection cannot identify a process based on a single action (due to the risk of False Positives), multiple individual processes, each performing a smaller set of actions, may go unnoticed.
  Injecting the payload in multiple processes also makes cleanup difficult: if only one affected process is terminated, the malware is capable of re-instantiating itself from another injected process.
  The distinct malicious processes (that make up an attack) may communicate using traditional inter-process communication, supported by the OS, or through purposely implemented mechanisms.

• Windows OS problems that make exploitation easier:
  • it does not keep a strict relation between child processes and parent processes => managing related processes for detection is more difficult, requiring OS specific knowledge
  • it allows code to be injected in a trivial way and does not provide a synchronous notification when injections occur => detecting all code injection methods is also considerably hard

IX. How do we combat multi-process malware, though?

• represent the actions performed by each process as feature vectors, then correlate them with the actions performed by the child processes. Disadvantages: difficult to implement, does not consider code injection when correlating processes.
• divide the processes into categories: group creators (they create other processes, not necessarily related to them), group inheritors and unmonitored processes. By assigning a category / role to each process, the groups of processes are much easier to identify and manage. The category of a process can be identified based on features like: the file path, the digital signature or a hash computed for the executable file.
• as installers can be used as a spreading mechanism, when an installation starts, the solution should create a process lineage tree, in which the root is the initial installer application. When the root or a descendant creates a new process, it should be added to the lineage tree as a child of the process that created it. This way, we can observe better the deepness and the exact activity of that installation, as it happens.

X. How about the advanced cyberattacks & cyberweapons detection?…

Well, as more and more people are being affected by this NSA-level madness ( shouldn’t come as a surprise, since some of their most sophisticated tools & exploits were leaked online by The Shadow Brokers – but that’s a story for another time ), we have to talk about this too.

These attacks are usually orchestrated by Advanced Persistent Threat (APT) actors – highly skilled, motivated and well-funded hackers, with ample resources at their disposal (usually nation state sponsored, as now it is much cheaper and effective to attack and spy your enemies in the cyberspace).

• APT attacks are carefully planned and often designed for a specific victim after a significant amount of time is spent researching the target => challenging to detect
• some security researchers try to combat this by combining the capabilities of the behavioral security solution with the MITRE ATT&CK knowledge base of adversary tactics and techniques, which are classified in 14 categories:

1. Reconnaissance - gather critical information to plan future operations
2. Resource Development - create/purchase/compromise/steal resources that can be used to support operations
3. Initial Access - get into the target network
4. Execution - run malicious code
5. Persistence - maintain your access
6. Privilege Escalation - get higher-level permissions
7. Defense Evasion - avoid being detected
8. Credential Access - steal account names and passwords
9. Discovery - figure out & map the environment you're in
10. Lateral Movement - move through the environment
11. Collection - gather data of interest to your goal
12. Command and Control - communicate with & control compromised systems
13. Exfiltration - steal sensitive data
14. Impact - manipulate and/or destroy the systems and data

• these tactics also represent reasons for malware to perform certain actions. Moreover, each category is broken-down into multiple techniques and sub-techniques that indicate how a tactical goal is achieved.

Conclusion

To wrap up, there are two serious problems that always seem to haunt the defense professionals: the time gap between the moment a new malware is released, until a behavioral model is available for that malware type (as systems are exposed to attacks), and the sophisticated cyberattacks (APT‘s specialty), which are particularly difficult to detect and they are often discovered when it’s already too late. Thus, there’s still an acute need for proactive behavioral detection solutions with fast response capabilities.

Publications used for this article:
Sushil Kumar et al. An emerging threat fileless malware: a survey and research challenges, 2020
Steve Mansfield-Devine. The malware arms race. Computer Fraud & Security, 2018
Jaime Devesa, Igor Santos, Xabier Cantero, Yoseba K. Penya, and Pablo Garcia Bringas. Automatic behaviour-based analysis and classification system for malware detection, 2010
Romanch Agarwal, Prabhat Kumar Singh, Nitin Jyoti, Harinath Ramachetty Vishwanath, and Palasamudram Ramagopal Prashanth. System and method for non-signature based detection of malicious processes, 2016
Ishai Rosenberg and Ehud Gudes. Bypassing system calls based intrusion detection systems, 2017
Weiqin Ma, Pu Duan, Sanmin Liu, Guofei Gu, and Jyh-Charn Liu. Shadow attacks: automatically evading system-call-behavior based malware detection, 2012
Jithin Pavithran, Milan Patnaik, and Chester Rebeiro. D-time: distributed threadless independent malware execution for runtime obfuscation, 2019
Gheorghe Hajmasan, Alexandra Mondoc, and Octavian Cret. Dynamic behavior evaluation for malware detection. In 2017 5th International Symposium on Digital Forensic and Security (ISDFS), Tirgu Mures, 2017
Gheorghe Hajmasan, Alexandra Mondoc, Radu Portase, and Octavian Cret. Evasive Malware Detection Using Groups of Processes, 2017
Gheorghe Hajmasan, Radu Portase. Systems and methods for tracking malicious behavior across multiple software entities, 2020
Sandor Lukacs, Raul Tosa, Paul Boca, Gheorghe Hajmasan, Andrei Lutas. Complex scoring for malware detection, 2016
Sandor Lukacs, Raul Tosa, Paul Boca, Gheorghe Hajmasan, Andrei Lutas. Process evaluation for malware detection in virtual machines, 2015
Bill Blunden. The Rootkit Arsenal: Escape and Evasion in the Dark Corners of the System, 2009