Security software and analysts always pose a threat to malware. Malware uses several techniques to protect itself. We can consider packers and rootkits as two of those techniques. Here are a few types of software that can pose a threat to malware:

• Windows troubleshooting tools: Task Manager and Registry Editor are tools that can be used to troubleshoot Windows. Task Manager can show the list of running processes on the system, hence a malware process may be identified. Registry Editor can be used to remove run entries (explained in malware persistence in section 3.3) used by malware. These tools are a threat to malware itself. So malware needs to disarm them.
• Malware analysis tools: Researchers use a number of tools to analyze malware. Here are few of the tools:
  • Debuggers: In simple terms, a debugger is a tool that can be used to test and find bugs in software. Ollydbg and IDA pro are some of the famous debuggers used for more than a decade. Malware researchers can debug malware with debuggers. In this case, it is not meant to find a bug in the malware, but to find out how the malware works.
  • System monitoring tools: There are other tools which analysts use to monitor files, registry, process, and network. Filemon, Regmon, and ProcMon are the famous ones. Wireshark is one the most used network sniffing tools.

Malware tries to detect these tools in a number of ways. One well-known trick to detect the debugger is by using the IsDebuggerPresent() API provided by Microsoft. Malware uses this API to find out it is running under a debugger. Malware tries to detect the presence of files and processes related to these tools. Malware can look for the presence of ollydbg.exe, tcpdump.exe, wireshark.exe, and so on. Malware researchers mostly use the virtual machine to execute malware in a restricted environment. VMware, VirtualBox, and Qemu are the most famous ones. Malware also tries to detect the presence of virtual machines. In a virtual machine, a host operating system is installed, which consists of all types of tools needs for analysis. The virtual machine has the capability to take a snapshot instance of the guest operating system. A snapshot of a clean instance of the guest operating system is kept and, post analysis, it is reverted back to the clean snapshot. Malware tries to figure out whether it is being executed from inside a virtual machine.

Here are a few methods that malware can use if executed in Windows guest OS on VMware:

• VMware process in the guest: The guest operating system has few processes of VMware running in it. A guest OS has the following processes running in it: Vmwaretrat.exe, Vmtoolsd.exe, Vmwareuser.exe, and Vmacthlp.exe. Malware can detect these processes with the help of a Windows API used in enumerating processes.
• VMware-related files: Malware can check for the presence of the files vmtray.dll, mmouse.sys, and vmGuestLib.dll in Windows driver folders.
• Registry keys: Malware can check for the registry key HKLMHARDWAREDEVICEMAPScsiScsi Port 0Scsi Bus 0Target Id 0Logical Unit Id 0Identifier for VMware.

If the malware is able to detect tools or virtual machines, it can simply exit and analysts can't figure out what happened unless somebody deep dives into it.

Now, with the increase in malware, automated malware analysis is carried out. The automated malware analysis is termed a sandbox. A sandbox consists of a virtual machine in which a guest operating system is installed with malware analysis tools. A sandbox keeps a clean snapshot of the virtual machine with tools installed that are used to log the various activities of malware, such as file modifications, network connections, and registry changes. 

Automation is used to place malware inside the virtual machine and then execute it. After executing the malware, the automation code extracts the logs from the guest operating system and restores the virtual machine snapshot back to a clean state. The logs can consist of API traces, file modifications, registry modifications, and network activities done by the malware. After extracting the logs, the virtual machine snapshot is restored to a clean one. Cuckoo is a one well-known open source sandbox. Other sandboxes include Joe Sandbox.

Malware uses similar techniques to detect a sandbox as a sandbox comprises a virtual machine and security tools. But there are some techniques specific to sandbox detection too. One of the most popular ones is using the sleep() API to wait for a long time before actually executing the malicious part. Most sandboxes are designated a particular time frame to execute a malware. After the time lapses, the virtual machine is restored to a clean instance. So if malware sleeps for a longer duration, the sandbox cannot find out the actual functionality of the malware.