4.4.3.1 Emergency Exploit Detector

Most system operators fear zero-day vulnerabilities, as there is little that can be done about them until the vendor or maintainer of the software releases a fix. Furthermore, even after a vulnerability is made public, a patch takes time to be developed and must be put through a QA cycle. The challenge is even greater in environments with high availability concerns and stringent change control requirements; even if a patch is available, many times it is not possible to restart the system or service until a regular maintenance window. This leaves operators with a difficult decision: risk damage from restarting a system with a new patch, or risk damage from running an unpatched system.

Consider the CVE-2008-0600 vulnerability, which resulted in a local root exploit through the vmsplice() system call [24,25]. This example represents a highly dangerous buffer overflow, since a successful exploit allows one to arbitrarily execute code in ring 0 using a program that is publicly available on the Internet. Since this exploit involves the base kernel code (i.e., not a loadable module), patching it would require installation of a new kernel followed by a system reboot. As discussed earlier, in many operational cases, a system reboot or OS patch can be conducted only during a predetermined maintenance window. Furthermore, many organizations would be hesitant to run a fresh kernel image on production systems without having gone through a proper testing cycle first.

The vmsplice() system call is used to perform a zero-copy map of user memory into a pipe. At a high level, the CVE-2008-0600 vmsplice() constructs specially crafted compound page structures in user space. A compound page is a structure that allows one to treat a set of pages as a single data structure. Every compound page structure has a pointer to a destructor function that handles the cleanup of those underlying pages. The exploit works by using an integer overflow to corrupt the kernel stack such that it references the compound page structures crafted in user space. Before calling vmsplice(), the exploit closes the pipe, so that when the system call runs, it deallocates the pages, resulting in a call to the compound pages' destructor function. The destructor is set to privilege escalation shellcode that allows an attacker to hijack the system.

The emergency detector works by checking the arguments of a system call for a potential integer overflow. This differs in functionality from the upstream patch, which checks whether the memory region (specified by the struct iovec argument) is accessible to the user program. One could write a probe handler that performs a similar function by checking whether the entire region referred to by the struct iovec pointer + iov_len is in the appropriate range (e.g., by walking the page tables that belong to that process). However, a temporary measure to protect against an attack should be as lightweight and simple as possible to avoid unpredictable side effects. One major benefit of using an hprobe handler is that developing such a detector does not require a deep understanding of the vulnerability; the developer of the emergency detector only needs to understand that there is an integer overflow in an argument. This is far simpler than developing and maintaining a patch for a core kernel function (a system call), especially when reasoning about the risk of running a home-patched kernel (a process that would void most enterprise support agreements).

Our solution uses a monitoring system that resides outside of the VM and relies on a hardware-enforced int3 event. A would-be attacker cannot circumvent this event without having first compromised the hypervisor or modified the guest's kernel code. He or she could do so with a code injection attack that causes a different sys_vmsplice() system call handler to be invoked. However, it is unlikely that an attacker who already has the privileges necessary for code injection into the kernel would have anything to gain by exploiting a local privilege escalation vulnerability. While the proposed emergency detector cannot defeat an attacker who has previously obtained root access, its ease of rapid deployment sufficiently mitigates this risk.

Since no reboot is required and the detector can be used in a “read-only” monitoring mode (only reporting the attack, and otherwise not taking an action), the risk of using this detector on a running production system is minimal. To test the CVE-2008-0600 detector, we used a CENTOS5 VM and the publicly available exploit. As an unprivileged user, we ran an exploit script on the unpatched OS and were able to obtain root access. With the monitor in place, all attempts to obtain root access using the exploit code were detected.

4.4.3.2 Application Heartbeat Detector

One of the most basic reliability techniques used to monitor computing system liveness is a heartbeat detector. In that class of detector, a periodic signal is sent to an external monitor to indicate that the system is functioning properly. A heartbeat serves as an illustrative example of how an hprobe-based reliability detector can be implemented. Using hprobes, we can construct a monitor that directly measures the application's execution. That is, since probes are triggered by application execution itself, they can be viewed as a mechanism for direct validation that the application is functioning correctly. Many applications execute a repetitive code block that is periodically reentered (e.g., a Monte Carlo simulation that runs with a main loop, or an HTTP server that constantly listens for new connections). If one profiles the application, it is possible to identify a period (defined in units of time or using a counter, like the number of instructions) at which this code block is reentered. During correct operation of the application, one can expect that the code block will be executed at the profiled interval.

The hprobe-based application heartbeat detector is built on the principle described in the previous paragraph and illustrated in Figure 4.4. The test detector (i.e., one of the detectors on the left side of Figure 4.2) is a kernel module that is installed in the host OS. An hprobe is inserted at the start of the code block that is expected to be periodically reentered. When the hprobe is inserted, a delayed workqueue5 is scheduled for the timeout corresponding to the reentry period for the code block. When the timeout expires, the workqueue function is executed and declares failure. (If the user desires a more aggressive watchdog-style detector, it is possible to have the hprobe handler perform an action such as restart of the application or VM.) During correct operation (i.e., when the hprobe is hit), the workqueue is canceled and a new workqueue is scheduled for the same interval, starting a new timeout period. This continues until the application finishes or the user no longer desires to monitor it and removes the hprobe. If having an hprobe hit on every iteration of the main loop is too costly, one can ensure that the probe is active for an acceptable time interval; it can be added/removed until a desirable performance is achieved. (The detection latency would still be low, as a tight loop would have a small timeout value.)

We use the open-source Path Integral Quantum Monte Carlo (pi-qmc) simulator [26] as a test application. This application represents a long-running scientific program that can take many hours or days to complete. As is typical with scientific computing applications, pi-qmc has a large main loop. Since Monte Carlo simulation involves repeated sampling and therefore repeated execution of the same functions, we need to run the main loop only a handful of times to determine the time per iteration. After determining the expected duration of each iteration, we set the heartbeat timeout to twice the expected value, set the detector to a statement at the end of the main loop, injected hangs (e.g., SIGSTOP), and crashed the application (e.g., SIGKILL). All crashes (including any VM crashes that occurred after the timer was executed in the hypervisor) were detected.

4.4.4.1 Microbenchmarks

We performed microbenchmarks that estimated the latency of a single hprobe, which is the time from execution of int3 by the VM until the VM is resumed (steps 1–3 in Figure 4.3). We ran these microbenchmarks without a probe handler function to determine the lower bound of hprobe-based detector overhead. Since the round-trip latency of an individual VM Exit on Sandy Bridge CPUs has been estimated to take roughly 290 ns [27] and our hypercall measurement scheme induces additional VM Exits, it would be difficult to accurately measure the individual probe latency. Instead, we obtained a mean round-trip latency by repeatedly executing a probed function a large number of times (one million) and dividing by the total time taken for those executions. That approach helped remove jitter due to timer inaccuracies as well as the actual latency of the hypercall measurement system itself. For the test probe function, we added a no-op kernel module to the guest OS that creates a dummy no-op device with an ioctl that calls a noop_func() kernel function that performs no useful work (return 0). First, we inserted an hprobe at the noop_func()'s location. Our microbenchmarking application started by issuing a hypercall to start the timer and then an ioctl against the no-op device. When the no-op module in the guest OS received the ioctl, it called noop_func() one million times. Afterward, another hypercall was issued from the benchmarking application to read the timer value.

For the microbenchmarking experiment, we used a 32-bit Ubuntu 14.04 guest and measured 1000 samples. The mean latency (across samples) was found to be 2.6 μs. In addition to the Sandy Bridge CPU, we have also included data for an older generation 2.66 GHz Xeon E5430 “Harpertown” processor (running the same kernel, KVM version, and VM image), which had a mean latency of 4.1 μs. The distribution of latencies for these experiments is shown in Figure 4.5. The remainder of the benchmarks presented used the Sandy Bridge E5-2660. The hprobe prototype requires multiple VM Exits per probe hit. However, in many practical cases, the flexibility of dynamic monitoring and the reduced maintenance costs resulting from a simple implementation outweigh that cost. The flexibility can increase performance in many practical cases by allowing one to add and remove probes throughout the VM's lifetime. Furthermore, CPU manufacturers are constantly working to reduce the impact of VM Exits; for example, Intel's VT-x saw an 80% reduction in VM Exit latency over its first 6 years [27].

4.4.4.2 Detector Performance

In addition to microbenchmarking individual probes, we measured the overhead of the example hprobe-based detectors presented in the previous section. All measurements in this section were obtained using the hypercall-based timer.

1. Emergency Exploit Detector: Our integer overflow detector that protects against the CVE-2008-0600 vmsplice() vulnerability is extremely lightweight. Unless vmsplice() is used, the overhead of the detector is zero since the probe will not be executed. The vmsplice() system call is rare (at least in the open-source repositories that we searched), so zero overhead is overwhelmingly the common case. One application that does use vmsplice() is Checkpoint/Restart in User space (CRIU). CRIU uses vmsplice() to capture the state of open file descriptors referring to pipes. We used the Folding@Home molecular dynamics simulator [28] and the pi-qmc Monte Carlo simulator from earlier as test programs. We ran these applications in a 64-bit Ubuntu 14.04 VM. At each sample, we allowed the application to warm up (load input data and start the main simulation) and then checkpointed it. The timing hypercalls were inserted into CRIU to measure how long it takes to dump the application. This was repeated 100 times for each case with and without the detector, and the results are tabulated in Table 4.1. In the table, we can see that there is a slight difference between the mean checkpoint times (roughly 3.3% for F@H and 1.7% for pi-qmc) and that the variance in the experiment with the detector active is higher for the Folding@Home case. Sys_vmsplice() was called 28 times when Folding@Home was being checkpointed, and 11 times for pi-qmc. We can attribute this difference to the negative cache effects of the context switch when probes are being activated.
2. Application Heartbeat Detector: We used the pi-qmc simulator to measure the performance overhead of the application watchdog detector. The pi-qmc simulator allows configuration of its internal sampling, and we utilized this feature to vary the length of the main loop. In order to determine how the detector impacts performance, we measured the total runtime of each iteration of the main loop when the probe was inserted and ran the program for 15 min. The results of our experiments are shown in Figure 4.6.

From Figure 4.6, we can see that the detector did not affect performance in a statistically significant way. The reason is that pi-qmc, like many scientific computing applications, does a large amount of work in each iteration of its main loop. However, by setting the threshold of the detector to a conservative value (like twice the mean runtime), one can achieve fault detection in a far more acceptable timeframe than with other methods, like manual inspection. Furthermore, this detector goes beyond checking whether the process is still running; it can detect any fault that causes a main loop iteration to halt (e.g., a disk I/O hang, a network outage when using MPI, or a software bug that does not lead to a crash).

4.5.1.1 VMI Monitor

In order to test the effectiveness of HI, we implemented a VMI monitor. To do so, we used LibVMI [18]. LibVMI is a software library that helps with the development of VMI monitors. It focuses on abstracting the process of accessing a guest VM's volatile memory. The volatile memory of a VM contains information about the guest VM's OS, such as kernel data structures, which can be examined to determine runtime details such as running processes or active kernel modules.

Because the polling rate of the VMI monitor is directly related to the performance overhead of the monitor, the polling rate must be chosen carefully so as not to introduce an unacceptable decrease in VM performance. We configured the VMI monitor to poll the guest VM every second. This rate introduced only a 5% overhead in VM performance, based on benchmark results from UnixBench.

4.5.1.2 VM Suspend Side-Channel

Because HI revolves around making measurements from a side-channel and inferring hypervisor activity, we first had to identify the actual side-channel to be exploited. We noted that whenever the hypervisor wants to perform a monitoring check on a guest VM, the VM has to be paused so that a consistent view of the hardware state can be obtained. If an observer can detect these VM suspends, then that observer might be able to learn about the monitoring checks performed by the hypervisor. We call this approach the VM suspend side-channel (see Figure 4.7). We came up with two potential methods for measuring the VM suspends: network-based timing measurements and host-based, in-VM timing measurements. Since the network is a noisy medium because of dropped packets and routing inconsistencies, we focused on performing in-VM measurements to detect VM suspends.

4.5.1.3 Limitations of Hypervisor Introspection

HI has two limitations: the low accuracy of the monitoring intervals it measures, and the difficulty of determining the threshold value for identifying VM suspends. Our testing of HI showed that it is capable of determining monitoring intervals as short as 0.1 s, as we determined by testing HI against increasingly frequent monitoring checks; the maximum frequency resolved by HI was one check every 0.1 s (10 Hz).

Because the threshold value for HI is found through empirical testing, development of HI is not straightforward for any given system. Our testing of HI indicated that threshold values ranging from 5 to 32 ms yielded the same accuracy when we were performing HI and detecting VM suspends. As future work, the threshold value may be correlated with various system specifications, such as kernel version, CPU model/frequency, and system load. After threshold values on various systems are found via empirical testing, a formal relationship between the various system specifications and the threshold value may be derived so that a threshold can be obtained without testing.

4.5.2.1 Insider Attack Model and Assumptions

We present an insider threat attack model in which the insider already has administrator (i.e., root) access to VMs running in a company's public IaaS cloud. The insider knows that he will be leaving the company soon but would like to maintain a presence on the VMs to which he has access. The insider does not have access to the underlying hypervisor hosting the VMs but knows that the company is utilizing some form of passive VMI monitoring. We also assume that the company's VMI monitor is similar to the monitor we implemented, which regularly checks for unauthorized processes. As insiders have full control over their VMs, the company relies on VMI for monitoring, so changes to the VM, such as kernel modifications or creations of new files, are not detected. In this attack model, the insider can utilize HI to detect invocations of the monitor and to hide malicious activities from the VMI monitor.

4.5.2.2 Large File Transfer

Attackers commonly want to exfiltrate data out of the network after compromising a system. The attacker may leverage various tools and protocols to accomplish this, such as secure copy (SCP), file transfer protocol (FTP), the attacker's own custom utility, or any number of other file transfer methods. A passive VMI system may detect attempts at data exfiltration by maintaining a restrictive whitelist of processes that are allowed to run with a network socket open. A restricted VM would have a whitelist with zero or few entries.

For transfer of sufficiently large files via a nonwhitelisted process, the transfer time would be longer than the monitoring interval, and the VMI monitor would detect the file transfer. This scenario is illustrated in Figure 4.8. We tested the effectiveness of our VMI monitor in this scenario by transferring a 250 MB file over TCP from the monitored VM to a coresident VM, and our VMI monitor detected the Python process running with an open socket and signaled an alarm.

It is possible to transfer a large file over a network without triggering any VMI alarms by splitting the file into chunks and transferring the chunks individually between monitoring checks. This process is illustrated in Figure 4.9. Thus, HI can be used to time the transfer of each chunk to evade the VMI system.

We implemented this attack by writing a Python script that transfers a chunk of a file each time it is executed. The offset of the current chunk is written to an “offset log” file after each execution, so that the next execution knows which chunk to transfer. We extended our HI kernel module with the user mode helper kernel function that is invoked by the Python script after a suspected VM suspends. With our modified HI kernel module and Python script, we were able to transfer the same 250 MB file without being detected by our VMI monitor.

On our test system, naïve transfer of the entire 250 MB file at once over a TCP connection between two coresident VMs took on average (over 100 tests) 4.905 s, with a standard deviation of 0.414 s. Using our technique with 8 MB chunks, the entire transfer took 32 calls to the Python script, and thus 32 monitoring intervals. Because each monitoring interval is 1 s long, the entire transfer took around 32 s, which is about eight times the transfer time of the naïve method.

We note that the transfer speed of the large file when this method is used depends on the polling frequency of the VMI monitor. Lower polling frequencies lead to longer transfer times because each chunk transfer is triggered by a monitoring check, so chunks would be transferred less frequently. However, after learning the polling frequency through HI, the attacker can increase the chunk size to counteract the delay. Increasing the chunk size is risky, as it may end up triggering the VMI system, so the attacker would likely verify a larger chunk size on his own system before performing the attack. Figure 4.10 shows the trade-offs involved in increasing the chunk size for the file transfer. Increased chunk size leads to a higher detection rate, but the whole file is transferred in fewer monitoring checks.

4.5.3.1 Introducing Noise to VM Clocks

Because HI relies on fine-grained timing measurements to determine occurrences of VM suspends, it follows that reducing the accuracy or granularity of VM clocks could prevent HI from working. Previous research has looked at reducing the granularity or accuracy of time sources to prevent cross-VM side-channel attacks. Although that work aimed to address cross-VM side-channel attacks, much of it is somewhat applicable to the hindering of HI.

Vattikonda et al. explored the possibility of fuzzing timers to reduce the granularity of measurements needed for side-channel attacks [24]. They modified the Xen hypervisor to perturb the x86 RDTSC instruction by rounding it off by 4096 cycles. Because the RDTSC instruction is commonly used to obtain high-resolution times tamps, side-channel attacks may be prevented. An additional benefit of fuzzing the RDTSC instruction is that timing system calls, such as gettimeofday and clock gettime, are fuzzed as well. Although HI relies on one of these system calls for time stamping, the perturbations caused only a 2 μs change in the true RDTSC value. HI needs measurements on the order of milliseconds, so the fuzzing does not perturb the RDTSC value enough to hinder HI.

Li et al. developed a system called StopWatch, in which the VM clock was replaced with a virtual clock that constantly skews [25]. Because the virtual clock depends only on the number of instructions executed, it hid VM suspends from in-VM timing measurements. However, applications with real-time requirements would not be able to use a virtual clock. In addition, StopWatch has a worst-case performance of 2.8× for workloads that require heavy network usage, which might not be acceptable for high-performance workloads.

4.5.3.2 Scheduler-Based Defenses

Recently, scheduling policies have been explored as another means to prevent cross-VM, cache-based, side-channel attacks. Varadarajan et al. proposed using the scheduler to force each process to run for a minimum runtime before another process is allowed to run [30]. If all processes are forced to run for a certain amount of time without being preempted, an attacker would obtain less information from observing a process's cache usage. Adjusting the scheduling policy could prevent part of our HI technique from working if the minimum runtime were greater than the VM suspend threshold. If that were the case, then process scheduling could not be used as one of the events observed for the in-VM timing measurements. However, many other events that occur during normal OS operation could still be observed, such as network operations or memory allocation and deallocation. An attacker could also artificially spawn processes that purposely utilize specific OS operations to increase the frequency of the events and improve the granularity of the measurements needed for HI. Thus, changing the scheduling policy might hinder HI, but it would not altogether block it. Further, enforcing a minimum runtime could degrade performance because CPU-intensive workloads would have to compete with less intensive workloads.

4.5.3.3 Randomized Monitoring Interval

Because HI looks for regular VM suspends to determine when monitoring checks occur, it may seem that simply randomizing the monitoring interval would prevent HI. This is not the case because randomized intervals have a lower bound on the duration between monitoring checks. A patient attacker could use HI to establish the lower bound on the monitoring interval and craft his attacks around that lower bound. Thus, the randomized monitoring interval forces the attacker to be inefficient, but it cannot prevent the attacker from evading the VMI system.

For example, consider a passive VMI system that polls the guest VM on a randomized interval that lasts anywhere from 1 to 10 s, inclusive. Assuming that the monitoring interval is a discrete random variable (i.e., the monitoring interval is an integer), one would expect to observe 10 monitoring checks before the smallest possible interval is seen.

Even if the monitoring interval is a uniform random variable (i.e., the monitoring interval can be a noninteger value), the attacker can transform the problem into the discrete-value case by taking the floor of the observed monitoring interval. In the case of an interval that lasts anywhere from 1 to 10 s, flooring the value leaves 10 possible values. Thus, the expected number of observed monitoring checks before the minimum is obtained is still 10.

The range of the random monitoring intervals is directly related to how difficult it is for an attacker to establish the minimum possible monitoring interval. Because security improves with a lower monitoring interval (and higher monitor frequency), the range of monitoring interval values would remain small in practice to improve security, but that would also make it easier for an attacker to determine the lower bound of the randomized monitoring interval through HI.

We reconfigured our VMI monitor to use a randomized monitoring interval between 0 and 2 s that would change after each monitoring check. We chose this interval because it kept the performance overhead around 5% (based on the same UnixBench benchmarks from earlier), and we expected the lower monitoring intervals (under 1 s) to detect the attack described in the earlier section. We performed the attack 30 times and found that the reconfigured monitor was able to detect the large file transfer 70% of the time.

We were surprised that some large file transfer attacks succeeded against the randomized monitoring defense. Some of the large file transfers went undetected because the randomized intervals that triggered the chunk transfers in those cases were long enough to thwart detection. However, the majority of the large file transfers were detected with randomized monitoring. Based on these tests, we argue that a randomized monitoring interval is sufficient for preventing a large file transfer attack but not a backdoor shell attack. Further, randomized monitoring does not prevent HI from learning that there is a passive VMI system in place, and HI can be used to learn the distribution sampled by the randomized monitor.

4.6.1.1 Data and Alerts

The log produced by the target machines, which is collected with the syslog protocol, is used to identify the users that access the infrastructure.

In the following, we describe the alerts produced by the monitoring tools when incidents that involve stolen credentials occur. As detailed below, we have assigned to the alerts IDs that are used throughout the remainder of this section.

First, we will discuss alerts that can be triggered when a login violates a user's profile. The detection of potential violations is done by checking login and profile data against rules set in the Simple Event Correlator (SEC):

• Unknown address (A1): A login comes from a previously unknown IP address, that is, the user has never before logged in from that IP, according to his or her profile.
• Multiple login (A2): The same external IP address is used by multiple users to log into the system.
• Command anomaly (A3): A suspicious command is executed by the user.
• Unknown authentication (A10): According to the profile data, the user has never before logged into the system using that authentication mechanism.
• Anomalous host (A11): The login is reported by a node within the infrastructure that has never before been used by the user.
• Last login > 90 days (A12): The last login performed by the user occurred more than 90 days before the current one.

In most cases, the occurrence of a profile alert does not by itself provide definitive proof that a user has been compromised. For example, A1 is raised each time a user (legitimate or not) logs in for the first time from a remote site that is not stored in the profile. In order to increase the chances of correctly detecting compromised users, the analysis of profile alerts is combined with the data provided by the security tools, for example, IDS and NetFlows (Figure 4.11), which are available in the NCSA network infrastructure. The security alerts used in the detection process are as follows:

• HotClusterConn (A4): A node of the computing infrastructure performs a download, although it is expected never to do so. An additional alert (A13) is introduced to indicate that the downloaded file exhibits a sensitive extension, for example, .c, .sh, .bin, or .tgz;
• HTTP (A5) and FTP (A9) sensitive URI: These alerts are triggered upon detection of well-known exploits, rootkits, and malware.
• Watchlist (A7): The user logs in from a blacklisted IP address (the list of suspicious addresses is held by and distributed among security professionals).
• Suspicious download (A14): A node of the computing infrastructure downloads a file with a sensitive extension.

Finally, two further alerts, A6 and A8, were designed by combining profile and security data. Alert A6 is generated whenever the remote IP address used to perform a login is involved in subsequent anomalous activities, such as a suspicious file download. Similarly, the alert A8 is generated if a user responsible for a multiple login is potentially related to other alerts in the security logs.

It should be noted that an event could trigger more than one alert. For example, the download of a file with a sensitive extension, if performed by a node that is not supposed to download any files, can trigger alerts A4, A13, and A14. While the occurrence of a profile alert leads to an initial level of suspiciousness about a user, a set of subsequent notifications, such as command anomalies or suspicious downloads, might actually be the symptoms of ongoing system misuse. Correlation of multiple data sources is valuable to improving detection capabilities and ruling out potential false alarms.

4.6.1.2 Automating the Analysis of Alerts

The timely investigation of alerts is crucial in identifying compromised users and initiating proper recovery actions. However, the analysis can become time-consuming because of the need to correlate alerts coming from multiple sources. In order to automate the alert analysis, we developed a software tool to (1) parse the content of heterogeneous security logs and (2) produce a representation of the security data more suitable for facilitating the Bayesian network approach.

Given the data logs (both syslogs and logs produced by the security tools), the tool returns a user/alerts table, which provides (1) the list of users that logged into the system during the time when the logs were being collected and (2) a set of 14-bit vectors (one for each user), with each bit of the vectors representing one of the alerts introduced in the previous section. Given a user in the system, a bit in the vector assumes value 1 if at least one alert of that type (observed in the security log) has potentially been triggered by that user. In order to illustrate the concept, Table 4.2 shows a hypothetical user/alerts table. For example, a binary vector of [10010000001000] is associated with user_1, which indicates that during the observation period, user_1 was potentially responsible for triggering three alerts: unknown address (A1), HotClusterConn (A4), and anomalous host (A11).

4.6.3.1 The Model: Bayesian Network

We have proposed a data-driven Bayesian network approach [33,34] to facilitate identification of compromised users in the target infrastructure. A Bayesian network is a direct acyclic graph in which each node of the network represents a variable of interest in the reference domain. The network makes it possible to estimate the probability of one or more hypothesis variable(s) given the evidence provided by a set of information variables. In the context of this work, the hypothesis variable is “the user is compromised,” while the information variables are the alerts related to the user.

We modeled the problem using a naïve Bayesian network, that is, a network in which a single hypothesis node is connected to each information variable. It was assumed that no connections exist among the information variables. The structure of the network is shown in Figure 4.14. A naïve network estimates the probability of the hypothesis variable by assuming the independence of the information variables. In other words, given a system user, the presence of an alert in a bit vector does not affect the probability of observing the other alerts. A set of vectors (the ones composing the training set described later in the chapter) was used to validate the assumption of independence of information variables. For each combination of the alerts (A_i, A_j) (for i, j ∈ {1, 2, 3,…, 14}), we counted how many vectors in the training set contained A_i, A_j, or both A_i and A_j. Then, the chi-squared test was applied to verify the null hypothesis H₀, that is, alerts A_i and A_j are independent of each other. Of the 91 possible combinations of alert pairs, in 28 cases (around 30% of the combinations) the chi-squared test (with a 95% significance level) indicated that H₀ had to be rejected. Regardless, in the majority of cases, the assumption of alerts' independence held.

It has to be noted that assuming the independence of a pair of dependent alerts leads to overestimation of the statistical evidence provided by the joint occurrence of the alerts. Thus, because the dependencies among the information variables are neglected, it might happen that the a posteriori probability of the hypothesis variable “the user is compromised” is greater than the real value. Consequently, the assumption of alerts' independence makes the analysis more conservative; however, more importantly, it does not compromise the detection capability of the adopted Bayesian network.

4.6.3.2 Training of the Bayesian Network

We used the logs collected for a subset of the available incidents, that is, 5 out of 16, as a training set for the network. Let T denote the adopted training set. T consists of 717 users (and corresponding bit vectors) and includes 6 compromised users. T contains the minimum number of training incidents, ensuring that almost all the alerts are covered by the actually compromised users, as shown in Table 4.3 (where comp_i denotes a compromised user). The strategy taken in this study has two main advantages: (1) It makes it possible to analyze the performance of the Bayesian network with conservative assumptions, that is, only a few training incidents are considered. (2) It does not bias the results, because all the adopted alerts are represented by the training set. Although other criteria might have been adopted, this selection of the training set was reasonable for our preliminary analysis of the performance of the proposed approach.

The training stage makes it possible to tune the network parameters, that is, (1) the a priori probability of the hypothesis variable, and (2) the conditional probability table (CPT) for each information variable A_i. The a priori probability of the hypothesis node C (Figure 4.14) is estimated as P(C) = 6/717 = 0.008. Calculation of CPTs requires additional effort. Four parameters must be estimated for each alert, as shown in Table 4.4. For example, P(Ai|C) denotes the probability that an alert of type A_i is related to a particular user, given that the user is compromised. Similarly, P(Ai|¬C) represents the probability that an alert of type A_i is related to a user, given that the user is not compromised.

The probability values of the CPTs are estimated as follows. The overall number of users in the training set T is divided into two disjoint subsets: good and compromised. Let G and C denote the two subsets, respectively. Note that |T| = |G| + |C|. For each alert Ai, let Li be the set of users (good or compromised) in the training set T that exhibits that type of alert, for example, all the users in T responsible for a command anomaly alert. P(Ai|C) is the ratio |Li ∩ C|/|C|, that is, the cardinality of the intersection of Li and C divided by the cardinality of C. Similarly, P(Ai|¬C) = |Li ∩ G|/|G|, that is, the cardinality of the intersection of Li and G divided by the cardinality of G. P(¬A_i|C) and P(¬A_i|¬C) are the complement to 1 of P(Ai|C) and P(Ai|¬C), respectively.

Table 4.5 summarizes the obtained results. It can be noted that P(Ai|C) assumes a relatively high value, which depends on the number of compromised users included in the training set T. Furthermore, P(Ai|¬C) is a measure of the quality of the alerts, in terms of the number of false positives they are likely to raise. For example, there is a high chance that an uncompromised user will be responsible for an unknown address (A1) or HotClusterConn (A4) alert. Similarly, the chances of observing a watchlist (A7) alert if the user is not compromised are extremely low.

By means of the proposed Bayesian network, given a user and the related vector of alerts, it is possible to perform the query “What is the probability P(C) that the user will be compromised, given that the user is responsible for 0 or more alerts?” In the following, we analyze how P(C) varies across the incidents and investigate the possibility of using the network as the decision tool.

4.6.4.1 Sample Incident

Example: Incident #5 (see Table 4.6) occurred on July 24, 2009. On the day the incident occurred, 476 users logged into the system. We estimated P(C) for each bit vector (i.e., user) in the user/alerts table computed for this incident. Figure 4.15 gives a histogram of the numbers of users with respect to the observed P(C) values. For the majority of users (410 out of 476), the computed P(C) is 0.02%. A closer look into the data reveals that none of these users were responsible for alerts observed in the security logs, and hence we can conclude that those users were not compromised.

In all the other cases, that is, 66 out of 476, at least one alert (in the security log) can be associated with each user. These 66 users are considered potentially compromised. For example, during the day the incident occurred, 33 users triggered a multiple login alert. However, 24 of those alerts were false positives caused by a training class that was using the GPU cluster at NCSA, for which all the users were logged in from the same IP address. Therefore, the presence of the multiple login alert alone does not provide strong enough evidence that a user is compromised, and the Bayesian network returns a small value of P(C) (i.e., 0.21%). The compromised user in incident #5 exhibits a value of P(C) around 28.54% (dotted arrow in Figure 4.15). An unknown address alert, a multiple logins alert, and a command anomaly alert are associated with the user.

We also observed cases in which a large value of P(C) was computed for an uncompromised user. For example, a user that jointly triggered unknown address, multiple login, unknown authentication, and anomalous host alerts resulted in a probability value of 87.73%. Nevertheless, it has to be noted that the number of false indications produced by the Bayesian network is small. In incident #5, only for three uncompromised users was the P(C) value greater than the one computed for the actually compromised user (to the right of the dotted arrow in Figure 4.15). For this incident, the proposed approach would have brought the number of manual investigations of potentially compromised users from 66 down to 4.

4.6.4.2 Discussion

The described analysis was conducted for each incident in the validation set. The analysis results reported in Table 4.6 reveal that P(C), for different incidents, varies across a large range of values.

One can see that in all but two cases (incidents #6 and #7), the P(C) values for the compromised users were relatively large. For incident #6, a single alert, HotClusterConn (with a sensitive extension), was associated with the compromised user. However, this alert, if observed alone, is quite unreliable. As shown in Figure 4.12 (A13), around 43 users per day are potentially responsible for this type of alert. Our Bayesian network returns a very small value for P(C), 0.7%.

As discussed, most of the suspicious users, that is, the ones related to at least one alert, are not actually compromised: P(C) is generally small. This finding suggests that the Bayesian network can be used to remove the noise (false positives) induced by the alerts. In other words, it is feasible to define a classification threshold, that is, (C), that will allow suppression of a significant fraction of false positives while still identifying all compromised users.

4.6.5.1 Analysis of the Incidents

According to the results provided in Table 4.6, the minimum classification threshold that allows detection of all the compromised users is 0.7% (the value of P(C) observed for the compromised user in incident #6). We used this value to quantify the effectiveness of the Bayesian network approach.

Table 4.7 summarizes the obtained results. Column 1 lists the IDs of incidents in the validation set. (The IDs are the same as in Table 4.6.) For each incident, column 2 reports the number of users that logged into the system during the day the incident occurred, and column 3 provides the number of suspicious users, that is, the ones related to at least one alert.

Column 4 gives the number of actionable users, that is, the subset of the suspicious users whose probability of having been compromised is ≥0.7%. For each incident, the ratio between the numbers of actionable and suspicious users quantifies the effectiveness of the proposed approach in reducing the number of false indications/positives. That ratio is reported in column 5. For example, in the worst case (represented by incident #3, in which 134 out of 312 suspicious users are actionable) the tool removed around 58% ((1–134/312)*100%) of false indications. As shown in the last row of Table 4.7, the average ratio (estimated across all the incidents) is around 0.20 when the classification threshold is set to 0.7%. In other words, for the analyzed data set, the Bayesian network approach automatically removed around 80% of false positives (i.e., claims that an uncompromised user was compromised).

The effectiveness of the proposed strategy is bounded by the need to select a relatively low threshold. In fact, 0.7% is a conservative value, which makes it possible to avoid false negatives (wherein actually compromised users are missed). We analyzed how the number of actionable users varies when the value of the classification threshold increases. Results are reported in columns 6 and 7, and in columns 8 and 9, of Table 4.7, for the threshold values 1.2 and 4.3%, respectively. According to Table 4.6, 1.2 and 4.3% are the next two smallest P(C) values observed in the analysis. The compromised user of incident #6 went undetected when the threshold was 1.2%. However, the network removed around 88% of false positives. Similarly, when the classification threshold was set to 4.3%, the compromised users of incidents #6 and #7 were undetected. In that case, the network removed around 93% of false positives.

Analysis results also suggest that the Bayesian network approach can help system administrators by directing them toward the users that are likely to be compromised. After the security logs have been collected, the tool can be used to obtain a list of users exhibiting a particularly large value of P(C), for example, all the users whose probability of being compromised is ≥4.3%. This procedure reduces the work burden of the administrators. As indicated in the last row of Table 4.7, on average, around 19 users out of 398, that is, only 4% of all users that log into NCSA during a normal day of operation, surpass the 4.3% classification threshold. If an actually compromised user is not detected (i.e., the classification threshold is set to a large value), the analyst can decrease the threshold in order to augment the set of possible suspicious users to be investigated.

4.6.5.2 Analysis of the Borderline Cases

We also assess the effectiveness of the network for borderline cases, such as incidents reported by third parties, or for normal days, that is, days on which no incidents occurred. In conducting the analysis, we used 0.7% as the classification threshold. The obtained results are summarized in Table 4.8. The meanings of columns 2, 3, 4, and 5 are the same as in Table 4.7. Furthermore, for each case (when applicable), columns 6 and 7 report the maximum observed P(C) and the P(C) for the compromised user, respectively. The main findings of the analysis are discussed in the following.

4.7.2.1 Environment

We represent the context of an access as an environment and model the environment as a vector of environment attributes, each of which is represented by an environment variable (called an attribute name) associated with an attribute value in a domain. An environment is defined by n attributes. Let v_i ∈ D_i, i = 1,…, n, be the ith environment variable, where D_i is the domain of that environment variable. Then, a vector (v₁,…, v_n), in which all variables are instantiated, is called an environment state (denoted by s). The set of all possible environment states is denoted by E. The choice of environment attributes (and hence the environment state) is domain-dependent. Environment attributes, particularly the dynamic attributes, are gathered by an access control engine at runtime.

Example (environment state): Assume that the environment is defined by three attributes: mode, access location, and access time. Then, mode = “normal,” access location = “station 1,” and access time = “8:00 a.m. Monday” together is an environment state.

An environment pattern, denoted by e, is treated as a unique pattern in domain EP, but is semantically defined by a first-order logical expression of assertions involving environment attributes. An environment pattern defines a set of environment states in which every environment state satisfies the environment pattern, that is,

Hereafter, we will sometimes directly use e to denote the set of environment states defined by the environment pattern.

4.7.2.2 User-Role Assignments

A particular user-role assignment associates a user, a role, and an environment pattern:

(4.1)

where EP is the set of all environment patterns that have been defined for the system of interest.

The semantics of a user-role assignment, (u, r, e) ∈ URA^e, is defined as

(4.2)

which states that if the real environment state s matches the given environment pattern e, then user u is assigned to role r. We assume that the RBAC engine can understand the semantics of each environment pattern as defined.

Basic RBAC models define user-role assignments simply as mappings from users to roles,

(4.3)

We have extended this notion with a dependency on the environment, as a means of integrating certain extended RBAC features of context and constraints. The environment pattern associated with a user-role assignment is the environment-dependent condition that is sufficient for the assignment. This feature can be regarded as constrained user-role assignment. If there are no constraints on user-role assignments, the associated environment patterns are simply empty, so the model becomes the common one. The relation between URA^e and URA is given as

(4.4)

Following this formalism, an example user-role assignment with an environment extension may be expressed as follows:

1. User id: com:ab:zn1:amy; Role id: manager.zone1; Environment pattern: Device = “Station_1.2” & Time = “Weekday”
2. User id: com:ab:zn1:ben; Role id: operator.zone1; Environment pattern: Mode = “emergency”

4.7.2.3 Role-Permission Assignments

A role-permission assignment associates a role, a permission, and an environment pattern. Thus, the set of all such assignments is a subset,

(4.5)

The semantics of a role-permission assignment, (r, p, e) ∈ RPA^e, is defined as

(4.6)

which states that if the real environment state s matches the pattern e, then permission p is assigned to role r.

Similar to the approach for user-role assignments, we also extended the common role-permission assignment with environment patterns. The relation between RPA^e and RPA is given as

(4.7)

As with user-role assignments, the role-permission assignment may also be organized in tabular fashion.

4.7.3.1 Role-Permission Assignment Policy

The role-permission assignment policy is a set of rules. Each rule has the following structure:

rule id {
target {
role pattern; permission pattern {
operator pattern;
object pattern; }; environment pattern;
} condition;
decision. }

where all of the patterns and the condition are FOL (first-order logic) expressions; an environment pattern defines a set of environment states; a role pattern defines a set of roles by specifying their common attributes; a permission pattern consisting of an operator pattern and one object pattern defines a set of permissions by specifying their common features with respect to the attributes of the operator and the object in a permission; an operator pattern defines a set of operators (or operation types); each object pattern defines a set of objects; the target defines the range of (role, permission, environment pattern) triples to which this rule applies; and the condition is a logical expression defining a relation among the attributes of the roles, the permissions (operators and objects), and the environment. Rules of this form state that when the condition is true, a role covered by the role pattern can be assigned with a permission covered by the permission pattern in the specified environment pattern.

4.7.3.2 User-Role Assignment Policy

User-role assignment is highly dependent on business rules and constraints. In our view, the task of assigning users to roles can be approached like the role-permission assignment problem, in terms of policies that enforce those rules and constraints. Such policies would be formulated in terms of user and role attributes, and would be crafted to enforce things like separation of duty. However, unlike role-permission assignment, user assignment may have to balance competing or conflicting policy rules against each other. Correspondingly, a complete policy-oriented formulation will need to specify how to combine rules and arrive at a final assignment decision. In what we present below, an attribute-based user-role assignment policy is used only to identify potential assignments. A rule-combining algorithm is used for making the final assignments.

The user-role assignment rule is similar to the role-permission assignment rule and consists of

rule id {
target {
user pattern;
role pattern;
environment pattern;
} condition;
decision. }

where a user/role/environment pattern defines a set of users/roles/environment states by specifying the common attributes; all of the patterns and the condition are expressions in first-order logic; and (unlike the case in the role-permission assignment policy) the decision of a rule marks a (user, role, environment pattern) triple as a potential assignment.

4.7.4.1 RBAC Model-Building Process

In the following, we briefly explain how to apply the proposed framework in building an RBAC model instance for an ICS. More detailed discussion of RBAC for ICS can be found in Ref. [32].

1. Identify security objects and object hierarchy: In a plant, security engineers identify devices and points to be protected by RBAC as a set of security objects and organize those objects in hierarchical structures (usually an acyclic graph). In ICS, it is common to find that the control devices and other assets involved in monitoring and controlling the physical process and in overall management of a plant are organized in a hierarchical and modular manner as per ISA-88 and ISA-95 standards. This hierarchical structure that groups objects is called an object hierarchy in this chapter.
2. Identify operation types: The types of the operations applied to each security object are identified, for example, a specific type of control parameter may be of type “read” and “reset.” At a minimum, an operation type could be defined by just an operator; in general, an operation type can be defined by a permission pattern.
3. Identify role templates: ICS tend to have very well-defined job functions, with well-defined access needs, to monitor and control the physical processes. For example, “Operator” and “Engineer” are very well-understood job functions in the ICS community, even though there might be some variations in their functions across different types of ICS. In the world of electric power, “Operator” is actually a licensed position. Such well-defined job functions with well-defined access needs are good candidates for use in defining “roles.” However, in practice, users who perform such well-defined job functions are limited in scope to either particular subprocesses or particular geographical areas, in accordance with the organization of their plants. In other words, the job functions identified in ICS, such as “Operator” and “Engineer,” perform certain types of operations; the objects to which those operations can apply are dependent on the attributes of the users and those objects. From this point of view, we defined in our previous work [25,32] a role template and proto-permission that leverage some common characteristics of ICS to simplify the creation and management of roles. Here we show how our framework can be used to represent these concepts and realize a manageable RBAC solution for ICS. Unlike a permission that comprises a pair of an operator and an object, a proto-permission consists of an operator and the object type of the operand. Thus, a proto-permission represents just an operation type, rather than a specific operation on a specific object. Proto-permission is a specific form of permission pattern. Unlike a “role” in RBAC, a role template is associated with a set of proto-permissions. A role template tells what types of operations can be performed by a role created from the role template. Let RT denote the set of all role templates, and let PP denote the set of proto-permissions. Every role template has an attribute (or, equivalently, function) of a proto-permission set, denoted by pps; every proto-permission has an attribute (or function) of an operator, and an attribute (or function) of an object type, denoted by objType.

   We use the role template to formally represent each well-defined job function in an ICS, and use proto-permissions to represent each allowed operation type associated with a role template. Assume that the plant has a number of basic types of job functions, such as “Operator,” “Engineer,” and “Manager.” They are identified as role templates. Each of them is associated with a set of operation types, for example, “Engineer” has operation type “reset parameter” on objects of type “XYZ”; “Manager” has operation type “view schedule” on “System”; and so forth. All identified operation types should be covered by role templates.

4. Identify roles and their privilege ranges: Based on business workflow analysis, a number of roles can be identified for each role template. Each role has an assigned working (access) environment pattern, such as access through station B in zone 1 in daytime. Furthermore, as discussed earlier, in practice each role has access only to a certain range of objects; this range of accessible objects is called the privilege range of the role. Formally,
   (4.8)

   A privilege range is used to define the boundary of objects for which a role is responsible, and is used as constraints in role-permission assignment. A role has access to an object only if the object is within the role's privilege range. A privilege range can be defined over an object hierarchy. An object hierarchy (denoted by OH) is simply a subset of the power set of all objects considered, that is, OH ⊆ 2^O. A node in the object hierarchy is called an object group, which is a subset of the objects, that is, og ∈ 2^O. In an object hierarchy, if one object group is a child of another in OH, it means that the former is a subset of the latter.

5. Analyze security policies and identify attributes: The security engineers analyze all security policies and requirements applied to the plant and list all of the attributes of users, roles, objects, and the access environment. For example, privilege range is a critical attribute of a role. A particular attribute that needs to be considered is security level. It is common to assign a security object a “security level,” requiring a subject of access to have a corresponding security level.
6. Develop attribute-based policies: Security engineers construct the underground level of an RBAC model by developing attribute-based policies for role-permission assignment and user-role assignment.

   Let us consider a simple yet general case: (1) A role can perform only the types of operations specified by the proto-permissions of the role's template. (2) There is a privilege range constraint, such that a role can access an object only if that object is within the privilege range of that role. (3) There is a security-level constraint, in that to access an object, the role's security level needs to be greater than or equivalent to the one of the object. This can be represented with the attribute-based policy for role-permission assignment:

   rule{
   target:{} condition:{
   memberOf(o, r.pr);
   r.securityLevel >= o.securityLevel; memberOf(pp, r.template.pps)
   and op = pp.op and o.type = pp.objType;} decision: add
   (r, p(op,o), phi) in RPAe.
   }

7. Create RBAC assignment tables: Use the specified attribute-based policies to create role-permission assignment and user-role assignment in the aboveground level.

   This task can be illustrated by considering how the above example rule would be used to make a role-permission assignment. For each pair of role and permission, if the pair or the role's assigned working environment did not match the target of the rule, then this pair would be skipped; otherwise, the condition part of the rule would continue to be evaluated. If the condition is found to be true, then the permission would be assigned to the role in that assigned environment, in the role-permission assignment table of the aboveground level. Consider a role “Engineer Chem Zone1 Daytime” and a permission “reset parameter T” on object “point 1.2.7” (i.e., it represents the seventh point in zone 1 sector 2), in the environment pattern stated in the example rule. This (role, permission) pair is within the target. Assume that object “point 1.2.7” is within the privilege range of the role and the access station. If the role's security level dominates the object's, and the professional domains match, then the permission is assigned to the role in the specified environment pattern. Consider another role, “Engineer Chem Zone2 Daytime,” which has privilege in zone 2 that does not cover “point 1.2.7”; thus, permission would not be granted to it for this role.

8. Repeat the process: If the above logical verification and review failed, it would be necessary to go back to an earlier step to revise the RBAC model, and then verify and review it again.

4.7.4.2 Discussion of Case Study

Migration to RBAC from a legacy system could be a great challenge for ICS. The proposed framework could support both building of an RBAC model for ICS and the migration. We highlight some major features as follows.