Theory scares IT professionals for some reason. This is not truly warranted, as theory is the underlying bedrock of all of your troubleshooting. It may be the axioms you have learned through your X years of hard-knocks trial and error. In the land of qualitative research, this is literally called the Grounded Theory Research Method. The base theory for reverse engineering is that the outputs infer the interior behavior of the application. When you are faced with a piece of malware, you are going to start making working hypotheses from a mixture of the following:
- Prior knowledge from recalled interactions with malware perceived as similar
- Generalizing perceived outcomes of interactions with the malware under test
Tip
Hacker Tip
It is probably not useful to label an application in an a priori manner. It may mask data to apply the "if it walks like a duck and quacks like a duck, it is probably a duck" axiom to the application. Especially with malware, it is likely that the design includes some deceptive features that are expected to set you off on the wrong track. Consider the Trojans and rootkits that remove other Trojans and rootkits as their first task. They are cleaning up your environment, but, are they really your friend?
Malware applications are designed to provide outputs from inputs, but be aware that the outputs and inputs do not truly give you a good idea of how the outputs are achieved. The outputs can be produced in several different ways, and you may find it matters how the developer chose to create the application.
This theory was published by Lee and Johnson-Laird in 2013 in the Journal of Cognitive Psychology, and is useful for information security practitioners, because it is shown on a Boolean system. A Boolean system is a logic gate. Either a condition is true or it is false. A very common definition of the problem might be as follows:
"Any system to be reverse-engineered contains a finite number of components that work together in giving rise to the system's behaviour. Some of these components are variable, that is, they can be in more than one distinct state that affects the performance of the system, e.g., the setting on a digital camera that allows for the playback or erasing of photographs. Other components of the system do not vary, e.g., a wire leading from a switch to a bulb. The system has a number of distinct inputs from the user and a number of consequent outputs, and they are mediated by a finite number of interconnected components. In some systems, a component may have a potentially infinite number of particular states, e.g., different voltages. But, for purposes of reverse engineering, we assume that all variable components can be treated as having a finite number of distinct states, i.e., the system as a whole is equivalent to a finite-state automaton. In other words, analogue systems can be digitised, as in digital cameras, CDs, and other formerly analogue devices. We also assume that the device is intended to be deterministic, though a nondeterministic finite-state device can always be emulated by one that is deterministic (Lee & Johnson-Laird, 2013)."
The Lee and Johnson-Laird model uses only Boolean internal models for the possible internal conditions that reveal the behaviors noted. Since it is not possible to test an infinite number of inputs, it is more useful to test only a subset of the possible inputs, and outputs. We can start with a simple example, for instance:
- If the malware lands on an Apple platform, and is designed to exploit a Windows vulnerability, it is likely not to run at all (switch 1)
- If it lands on a Windows machine, but is aimed at a vulnerability of the XP version, it may test for that OS version and do nothing if it finds itself on Windows Server 2012 (switch 2)
- If it happens to be Windows XP, but is patched for the sought vulnerability, it might also do nothing (switch 3)
- If it lands on a Windows XP machine that contains the sought-after unpatched vulnerability, it drops its payload
