2 Background

2.1 GPU and Its Memory Hierarchy

A GPU comprises multiple stream multiprocessors (SMs), called compute units in AMD architectures, each consisting of a rich memory hierarchy including register files, shared memory, cache arrays, and a number of simple cores called stream processor (SP). Generally, SMs are known as SIMT cores. GPUs use SMs as vector processors to achieve TLP by executing single instruction over multiple threads. A GPU application comprises one or more kernels. Each kernel launches a hierarchy of threads on GPU. Hundreds of threads create a grid composed of cooperative thread arrays (CTAs) or thread-blocks (TBs). The CTAs are allocated to SMs as single units of work. TBs decompose to groups of 32 threads for execution on an SM. These groups of threads are called warp in NVIDIA and wavefront in AMD terminologies, respectively. In other words, warps are the smallest fetched units within this hierarchy.

Each thread can access different data types that are annotated in GPGPU programming with separated address ranges in the main memory. These data types comprise global, local, texture, and constant data. A thread commonly accesses these data types through a rich memory hierarchy, as shown in Fig. 2B. The local data is usually placed in a dense register file array that is private to each SM. Global data refer to the data shared among all threads within a grid.

When a thread requests data from the off-chip main memory, the accesses pass through a two-level cache hierarchy. Each data type is placed in its specific cache memory in the first level of memory hierarchy. Local and global data are placed in D-caches while texture and constant data are placed in texture and constant caches, respectively. Instructions are placed in instruction caches. Additionally, unlike D-cache, which is a read-write cache memory, instruction, texture and constant caches are read-only caches. Each SM is also associated with a software-managed local memory for shared data accesses by threads within a TB. Note that shared data are not in this memory hierarchy and are directly placed in the shared memory. Shared memory is not the concern in this study; however, interested readers can refer to [22–25] for more details.

The existence of different data types and caches in GPU brings up different policies for read and write accesses. The L1 caches are private to SMs with no coherency among them because there is no data communication between SMs. Because L1 caches are not coherent, all write operations of global data are handled as write-evicts (on hit) or write-no-allocates (on miss). Local data are written into the associated L1 cache with a write-back mechanism (Fig. 2A). The L2 cache is a banked cache array shared by all SMs and backs all types of data. Each L2 bank communicates with L1 caches of different cores through an interconnection network. L2 cache uses write-back policy with respect to the main memory. At the highest level of hierarchy, each L2 bank has a point-to-point connection with a dedicated off-chip memory controller. A more complete description of GPU memory system is available in Ref. [26].

The L1 data cache and shared memory in NVIDIA Fermi GPUs are uniform memory in each SM with the ability of reconfiguration and have a fixed total (L1 + shared memory) capacity of 64 KB per SM. L1 data cache and shared memory can be configured as (16 KB + 48 KB) and (48 KB + 16 KB). This gives flexibility to programmers to set cache and shared memory sizes based on the requirements of nonshared and shared data, respectively. In the new Kepler GK100 (32 KB + 32 KB), configuration is implemented, too. Moreover, texture and constant caches are unified as read-only data cache in the Kepler architecture. In the new NVIDIA Maxwell GM204 architecture, a fixed 64 KB dedicated shared memory is made possible by combining the functionality of the L1 and texture caches into a single unit [27].

2.2 Nonvolatile Memories

In this section, we briefly overview ReRAM, STT-RAM, and racetrack memory technologies because, among nonvolatile memories, they have features closer to SRAM. More detailed description can be found in Refs. [13, 14, 28]. Then we will focus on STT-RAM based on its capability of reducing retention in order to overcome one of its main challenges and and make its read latency comparable to SRAM’s.

The ReRAM memory element consists of one inert electrode at the top (e.g., platinum), an electrochemically active electrode at the bottom (e.g., copper), and the oxide layer between inert and active electrodes. When a positive bias is applied to the active electrode, the metal ions are forced to migrate through the oxide and eventually reach the electrode on the other side (inert electrode). The ion path is highly conductive, and the cells, equivalent resistance value is low (1). The low-resistance state can be changed again to a high-resistance state by positively biasing the inert electrode (0). The biggest advantage of the ReRAM technology is the high compatibility with CMOS processes, which means that it can be fabricated at the same die that processor and SRAM caches are fabricated. Furthermore, the voltage required for ion path-forming process proportionally relates to the oxide layer thickness. These advantages gives ReRAM a strong cost-competitive advantage over SRAM. However, its low cell endurance (up to ∼10¹¹ writes [29]) and high access latency limits its use at near-processor cache levels.

STT-RAM technology is a type of scalable MRAM. The basic structure of an STT-RAM cell is composed of a standard NMOS access transistor and a magnetic tunnel junction (MTJ) as the information carrier (Fig. 3). MTJ consists of two ferromagnetic layers and one oxide barrier layer in between (Fig. 3). One of the ferromagnetic layers (i.e., the reference layer) has a fixed magnetic direction while the magnetic direction of the other layer (i.e., the free layer) can be changed by directly passing a spin-polarized current. If these two layers are magnetized in antiparallel (parallel) directions, the MTJ resistance is high (low), indicating bit “1” (“0”). STT-RAM has a limited write endurance per cell (up to ∼ 10¹² writes [30]), and when the number of writes to a cell reach its maximum limit, the cell may lose its ability to change the magnetization status of the free layer, thus creating data errors.

Racetrack memory, also known as domain wall memory (DWM), is a spin-based memory technology, which represents information using the spin orientation of the magnetic domains in a ferromagnetic wire, as shown in Fig. 4A. Each of these domains can independently take any magnetic orientation (up-spin or down-spin). Therefore multiple bits of data can be packed into a single DWM device, resulting in very high density. Three basic operations can be performed on a DWM device: read, shift, and write. The MTJ structure, shown in Fig. 4A, is used to read data from the DWM device similar to STT-RAM. In a DWM device, all the magnetic domains in the ferromagnetic wire share a single MTJ. The bit to be read/written needs to be aligned with the MTJ before it can be accessed. This is accomplished using a property that is unique to DWM, called domain wall motion, which refers to the shifting of magnetic orientations of domains in the ferromagnetic wire. When a current pulse of a suitable magnitude is applied through the ferromagnetic wire, the magnetic orientation of each domain is shifted in a direction opposite the direction of current. Additional domains are provided in the device to prevent loss of data at the extrema while shifting.

Formerly, the write operation was also performed with the MTJ similar to STT-RAM. This write operation is highly energy-consuming. However, a recent development in DWM has eliminated this inefficiency. It has been experimentally shown that domain wall motion (originally intended to realize shifts) can also be used to perform fast, energy-efficient writes in DWMs. This operation, often referred to as shift-based writes, is demonstrated in Fig. 4B. The structure of the write operation consists of a ferromagnetic wire with three domains: two fixed domains and a free domain. The magnetization of the two fixed domains are hard-wired to up-spin and down-spin during fabrication. However, the magnetization of the free domain, which is sandwiched between the fixed domains, can be varied by shifting the magnetization of one of the fixed domains by applying a current pulse in the appropriate direction.

The main challenge of racetrack memory is the access latency to data stored in a DWM tape that is variable depending on the number of shifts required. Varying the number of bits stored in each DWM tape brings a unique trade-off between density (which increases with more bits-per-tape) and average access latency. On the other hand, there is another mechanism to decrease the write operation problems in MTJ cells. Reducing the MTJ cell retention time results in decreasing write operation energy, in addition to reducing cell reliability. In other words, there is a trade-off between data retention and write energy in MTJ cells. Reduction of the write time period results in more current; therefore, for faster write operations, a higher amount of current and more power is needed. Due to the constraint on maximum current in chips, the write pulse period must be increased in order for the write current to be decreased. By reducing the retention time, the level of write current decreases for different write pulses, so shorter write time becomes possible. Fig. 5 shows different write currents as a function of the width of write pulses for 10 years, 40 ms, and 100 μs retention times. More details on STT-RAM and the model used in this chapter can be found in Ref. [21].

GPU memory hierarchy

Many researchers have considered the memory hierarchy of GPUs to reduce access latency of the memory [31], improve memory access coalescing [23], and increase the parallelism among memory controllers [32]. DRAM scheduling [33] and prefetching [34] techniques have also been introduced for GPUs. In addition, a number of recent works concentrate on the GPU cache from various perspectives.

Jia et al. [35] specified the memory access patterns in GPUs. Based on such information, the suggested algorithm estimates how much of the bandwidth is required when cache memory is on/off and then decides which instruction data is placed in the cache. In Ref. [36], Tarjan et al. tried to quickly cross the interconnection network to access L2 by handling L1 misses using the copies kept at other L1 caches in other SMs. To do so, they used a sharing tracker to identify whether a missed data block in L1 cache is present in other L1 caches. Rogers et al. [2] presented a method called Cache-Conscious Wavefront that employs a hardware unit to examine intrawarp locality loss when a warp is selected for fetching. Hence, their mechanism prevents the issuing of new warps that might result in thrashing previously scheduled warps, data in the cache. Similarly, in Refs. [37–39], the same problem is tackled by modifying the cache indexing function, warp throttling, and detecting critical warps in order to maintain useful data in the cache. Gebhart et al. [40] designed a uniform memory that can be configured to be register file, cache, or shared memory regarding the requirements of the running application. Moreover some other works, such as [41, 42], tried to reduce the power consumption of GPU by observing and considering GPU memory hierarchy from the main memory to the register file.

STT-RAM

Dong et al. [15] investigated the possibility of replacing SRAM with MRAM by introducing a model for power, area, and performance of MRAMs, which is used in our power and performance calculations, too. As the main STT-RAM challenges are the write operation’s latency and energy, most of the research focused on these aspects of STT-RAM. Rasquinha et al. [19] proposed an energy-efficient cache design in order to decrease STT-RAM dynamic energy by managing operations on cache. Zhou et al. [18] introduced the early write termination (EWT) technique to prevent writing unchanged data bits. Some other researches such as Refs. [16, 17] used a combination of MRAM and SRAM to overcome disadvantages of both technologies and use their advantages. On the other hand, Smullen et al. [20] considered, the relaxing nonvolatility of speed and energy-efficient STT-RAMs. Sun et al. [21] used STT-RAM cells with different retention times; they used a single low-retention STT-RAM L1 cache. For higher level caches, a two-retention STT-RAM structure was used, where data blocks with frequent writes are migrated to the cache blocks with low-retention time. In Refs. [43, 44], different refreshing mechanisms are proposed to efficiently employ low-retention STT-RAM.

In Ref. [45], a two-part main memory consisting of a small DRAM part and a large PCM part is presented. In contrast to our proposal, DRAM and PCM parts are managed in a hierarchical manner, that is, DRAM works as a cache for the large PCM memory. Additionally, the differences between the features of the cache and main memory make this approach inapplicable for designing a cache. For instance, in Ref. [45], the writes are forwarded to the PCM part instead of DRAM, because it is assumed that the locality of the data evicted from the cache was already exploited. However, in the cache, due to the existence of data locality, there could be multiple writes on a block, so it is better to forward those to the write-friendly part.

GPU memory and STT-RAM

Recent work [46, 47] used the nonvolatile DWM in both register files and the cache memory of GPUs. Also, Refs. [48–50] used STT-RAM in GPU register file with different approaches such as hybrid memory, write awareness, and multilevel cell design, respectively. Al-Maashri et al. [51] focused on using STT-RAM technology for 3D stacking of texture memory. They did not really address the use of GPUs for general-purpose high-performance computing but considered graphics applications. In research by Goswami et al. [52], a single and uniform low-retention time STT-RAM was used for the register file and constant and texture caches, and a hybrid memory (STT-RAM and SRAM) was employed for the shared memory of GPGPUs. To remedy the high-energy consumption of write operations, the EWT mechanism [18] with a higher granularity was used. Also, an SRAM write-back buffer was used to coalesce write operations of different threads to the STT-RAM register file.

Because L1 cache latency is critical to the overall application performance, in this chapter, we concentrate on studying trade-offs between STT-RAM retention time and write latency/energy and propose an architecture to surpass the cache shortcomings using this promising technology in GPU’s last-level cache (LLC). The L2 cache is typically a unified large cache (e.g., 1.5 MB in Nvidia’s GK110 and 2 MB in Nvidia’s GM107), which is greater than the total L1 caches in terms of size. Additionally, L2 cache is responsible for improving the effective memory access bandwidth of the SMs by reducing the number of off-chip memory accesses. Typically, the latency of L2 cache accesses is hidden by switching between multiple warps that are active in the SMs. Therefore implementing L2 cache using STT-RAM technology seems more reasonable and effective in GPUs.

4 Two-Part L2 Cache Architecture

4.1 Motivation

GPGPU applications perform different write patterns on cache blocks. In Ref. [53], the coefficient of variation (COV) parameter was used to indicate inter and intracache set write variations in chip multiprocessor (CMP) applications. Fig. 6 shows inter and intracache set writes COV for GPGPU workloads. Applications such as bfs, bfs-p, lavaMD, leukocyte, RAY, STO, streamcluster, and stencil have more writes on fewer cache blocks, whereas other applications have more balanced writes over L2 blocks. This justifies using an STT-RAM region having a reduced retention time that favors write operations. Ideally, we expect this region to hold data blocks with high rewrite frequency in order to minimize write energy and improve the cache performance.

Reducing the retention time of STT-RAM cells increases the error rate because of early data bit collapse (with respect to high-retention cells). Table 1 shows different magnetization stability heights (△) and their corresponding retention times (RT), write operation energies (WE), and latencies (WL). The 10-year retention time STT-RAM cell has the most write energy and latency with no error rate because of its high data stability. Any reduction in the retention time of STT-RAM cells from this baseline reduces write operation energy and latency due to the reduction in the current needed for writing, but more refreshes are required to prevent data loss. This leads us to using a nonuniform STT-RAM cache structure with different retention times.

Utilizing low-retention STT-RAM is meaningful when it can hold WWS of an application to minimize the write energy. Also, refreshing overhead of cache blocks in low-retention part of the cache must be reasonable.

As mentioned before, the WWS of most GPGPU applications is small and within short periods of time. Fig. 7 schematically shows spatial and temporal distribution of write accesses to the LLC. At each TWWS (t₁ and t₂ in Fig. 7), a number of data blocks are accessed for write, and then there are no significant write accesses to the LLC for a while (t₁′ and t₂′ in Fig. 7). In a TWWS, each data block may be accessed one or more times. We define write-to-write time (W2W) as the time difference of two successive accesses to a block and use it indirectly to estimate the average TWWS duration for each workload. For this purpose, in the first phase, we count data blocks with W2W greater than a specific time range (time range 1 in Fig. 7). Then we increase the time range (time ranges 2, 3, …, n in Fig. 7) and again count data blocks with W2W greater than each of the time ranges. This is repeated until a time range is found such that the number of data blocks with W2W greater than that remains almost unchanged (Time ranges 3, 4, …, n in Fig. 7). This range marks the end point for each workload and is an approximation for the time difference between the beginning of two consecutive TWWSs (t₁ + t₁′ in Fig. 7).

Afterward, to determine the length of TWWS, we observe the distribution of RI, the time difference between first and last accesses to a block in TWWS, for example, t_A in Fig. 7. Wehave chosen RI as a representative for TWWS duration, since a decline in accesses to most of the blocks constituting TWWS is observed in a short period of time. Therefore the time range that has the most RI distribution in it is probably the TWWS duration. For each workload, the TWWS duration is calculated based on this method. Table 2 shows the estimated TWWS durations in cycles.

Finally, Fig. 8 shows the distribution of data blocks whose RIs are in the estimated TWWS duration. More than 80% of rewrite accesses to the data blocks are under each workload’s estimated TWWS duration, which proves our estimation accuracy. Moreover, this justifies the assumption that we have short-time periods in GPGPU applications with burst write accesses to a specific number of blocks in L2 cache. Additionally, Table 2 shows the maximum number of distinct cache blocks accessed in the TWWS duration for each workload. With respect to the average of maximum TWWS sizes, a small write-optimized part of the cache is needed for containing the TWWS of workloads. Moreover, a large high-retention part of the cache is used to hold read-only and less frequently written data blocks.

4.2 The Proposed Two-Part Cache Architecture

In this section, we describe our proposed STT-RAM L2 architecture for replacing SRAM L2 in GPUs. Because it was justified in the previous section, we need to have two parallel low-retention and high-retention parts (LR and HR) to keep more frequently and less frequently written blocks, respectively. Therefore, we propose a two-part L2 cache with different data retention times to manage write operation energy and data refreshing problems by migrating more frequently written data blocks from HR to LR, and vice versa. Fig. 9 depicts the block diagram of the proposed STT-RAM L2 cache architecture. Two parallel structures with different data retention times, swap buffers between themselves, and counters and some hardware logic for managing search in and migration between two cache parts are employed. Our L2 cache architecture has three main characteristics that make it distinct from typical GPU L2 caches, which are explained next.

Search mechanism

The second difference is the search mechanism among two cache parts. Because we have two parallel cache arrays, different approaches can be considered in order to handle cache accesses. Two possible approaches include parallel and sequential searches in LR and HR. Searching in LR and HR in parallel results in better performance, but is more power-consuming. In contrast, sequential search may cause more latency, but saves power.

Fig. 10 shows the timing for both search mechanisms on different scenarios. Sequential search mechanism adds an extra cycle when the first cache selected for searching does not hold the requested data block (Fig. 10A and C). However, the overhead of searching both LR and HR tag arrays can be avoided if the requested block is found in the first part (Fig. 10B and D). In order to address the shortcomings of each of the parallel and sequential search mechanisms, the data and access type aware cache search mechanism (DAACS), shown in Fig. 9, is employed.

The DAACS determines which part of the L2 cache must be searched first, regarding the access types (read or write) and data types (instruction, texture, constant, global or local data). If the requested data is not found in the first cache part, the second cache part will be searched. Obviously, as frequently written data are kept in LR, it is searched first for a write request. For read accesses, on the contrary, the HR part is searched first. Moreover, read-only data (i.e., texture and constant data and instructions) are never migrated to the LR part. Thus only the HR will be searched upon receiving access to these data types, regardless of whether the data exists in HR or not. Fig. 11 shows a flow diagram of the DAACS.

Refreshing mechanism

Because the retention time of the cache parts are reduced to favor write operations on STT-RAM cells, data refreshing is a new issue that needs to be addressed. Error prevention [54–56] or data recovery [57] are the common solutions, but they are not applicable here because of numerous bit collapses when a cache line with low-retention time expires. A simple mechanism based on Ref. [43] can be used to handle retention time expiration. In Ref. [43], a counter is used per cache line to keep track of the lifetime of the cache block. Similar to Ref. [43], in this work, we employ such a counter, retention counter (RC), with different policies. With respect to the value of RC, we can opt to refresh or invalidate the cache block, or perform a write-back to the data block. Our architecture uses all three approaches appropriately. If an RC saturates, this means that its corresponding cache block retention time has expired. Therefore the cache block will be invalidated. Note that before RC saturates (i.e., one cycle before counter saturation), the proper decision must be taken.

Fig. 12A shows that the RI distribution in LR. The 100 μs is a reasonable retention time due to its low write latency/energy and low overhead of refreshing. Most of the workloads have RIs lower than 100 μs. In order to refresh a cache block, it is read and pushed into a buffer and then rewritten in the cache to prevent data loss. The LR-to-HR buffer is used for refreshing cache blocks in LR. Because the LR-to-HR buffer is also used for data block migration between LR and HR, by using a tag in this buffer, the blocks migrating to HR and those needing to be refreshed are differentiated. When the LR-to-HR buffer is full, the cache block that needs refreshing is forced to be written back to the main memory before its retention time expires. Fig. 12B shows RIs for cache blocks in HR. We can draw two deductions from Fig. 12B. First, a 10-year retention time for HR is not needed, and 40-ms retention time covers the lifetime of more than 97% of cache blocks in HR. We can invalidate the few remaining cache blocks (i.e., less than 3%) with higher lifetimes and force them to be written back if they are dirty.

5 Dynamic Write Threshold Detection Mechanism

5.1 Motivation

It is important to minimize the number of block transfers between HR and LR to reduce the overall write overhead with respect to the baseline architecture. Low WTs cause a high number of swaps between HR and LR and, consequently, increase the dynamic power. As dynamic power of STT-RAM memory is intrinsically more than that of SRAM, it is a critical factor for utilizing STT-RAM in our design. On the other hand, if WT is high, a lower number of blocks are migrated from HR to LR. As a result, more writes are performed on HR, and more dynamic power is consumed because of higher dynamic power of write operations in HR, whereas LR is left underutilized. This means that the effective size of cache decreases, which may cause performance degradation. Because of the different types of workloads on GPGPU regarding memory intensity and variation in the number of write operations, having a fixed threshold is not efficient. Fig. 13 shows the write cost in our two-part cache for different WTs. During various phases of runtime of an application, minimum write cost may be caused by different WTs. On the other hand, appropriate WTs may differ from one application to another. We design a dynamic architecture to regulate WT in the runtime based on the write costs of LR and HR, along with the number of block swaps between them.

5.2 Threshold Analysis

For designing an adaptive monitoring logic, we need to determine the parameters affecting cache write cost, and the affect of WT on these parameters. Eq. (1) indicates that the total cache write cost (TWCost) is composed of write costs of HR (WCost_HR) and LR (WCost_LR), each calculated by the multiplication of its write cost per block (WCB) and write rate (Eqs. 2, 3).

$\begin{array}{ll}\hfill TWCost& =WCos{t}_{HR}+WCos{t}_{LR}\hfill \end{array}$

$\begin{array}{ll}\hfill WCos{t}_{HR}& =W_{HR}*WC{B}_{HR}\hfill \end{array}$

$\begin{array}{ll}\hfill WCos{t}_{LR}& =W_{LR}*WC{B}_{LR}\hfill \end{array}$

W_HR is the summation of write hits (WHit_HR), misses, and swaps (Eq. 4). The same parameters, excluding misses, form W_LR (Eq. 5). Misses are excluded from W_LR, because the missed data blocks are written in HR first.

$\begin{array}{ll}\hfill W_{HR}& =Miss+WHi{t}_{HR}+Swap\hfill \end{array}$

$\begin{array}{ll}\hfill W_{LR}& =WHi{t}_{LR}+Swap\hfill \end{array}$

On the other hand, WCB_HR can be calculated as the product of WCB_LR and a constant factor (Eq. 6).

$\begin{array}{l}\hfill WC{B}_{HR}=n*WC{B}_{LR}\end{array}$

Therefore TWCost can be calculated as:

$\begin{array}{l}\hfill TWCost=BWCos{t}_{LR}*(n*WHi{t}_{HR}+WHi{t}_{LR}+n*Miss+(n+1)*Swap)\end{array}$

Thus Miss, Swap, WHit_HR, and WHit_LR are the factors that must be considered in regulating WT. Our observations show that the effect of WT changes on miss rate is less than 0.13%. Thus they can be ignored as an affecting parameter for the sake of simplicity. Table 3 explains the effects of the other three parameters on WT.

5.3 Proposed Mechanism

Each of the parameters mentioned in Table 3 is a function of WT, which is dependent on the behavior of the applications themselves. The feedback from changes in these parameters during the runtime of an application can be used to regulate WT. However, such an approach is sensitive to the details of the implementation. In other words, the question, is how much should the variation in the aforementioned parameters change the WT? Additionally, the appropriate relation between these parameters and WT may differ in various applications. These complexities motivate us to employ a mechanism that has the least possible dependency to the behavior of running applications. To this end, we propose a new structure, DTR, in order to effectively take these parameters into account for regulating WT, without having to deal with the complexities mentioned before. Fig. 14 depicts DTR. It simply sets WT to the minimum value of WCs in LR.

Using the minimum value of WCs in LR mimics the desired behavior mentioned in Table 3. First, WHit_LR is roughly related to the minimum value of WCs in LR. In other words, growth in WHit_LR results in increase in the min(WCs) most of the time, hence the WT goes up. Second, min(WCs) creates meritorious data blocks into LR. This results in two desired consequences: first, unnecessary Swaps are avoided; second, min(WCs) will probably increase per swap.

WHit_HR is another factor that must affect WT. However, WHit_HR cannot be considered directly by our proposal. To overcome this issue, we look at the number of writes per block in TWWS of different applications. By setting the maximum possible value of the WC to the average of writes per block in TWWS, the WCs will probably be reset at the end of the lifetime of the blocks in TWWS. This naturally prevents the WCs from keeping large values and hence averts directing the majority of write operations to HR.

We use two tree-structured minimum-finder (TMF) units, each comprising 15 comparators to find min(WCs). One of the TMFs, the local TMF, is responsible for determining the minimum WC value per set. On each write access, the WCs’ value of the corresponding set are directed to the local TMF by MUXes, and the minimum WCs of that set are updated via a DEMUX. The other TMF, the global TMF, updates the global minimum of the WCs in LR, using the local minima of sets.

The proposed mechanism can easily adapt itself to the behavior of applications, without having to deal with the complexities of determining the proper coefficients for WHit_LR, WHit_HR, and Swap values in making decisions about the value of WT. The only dependency of our design to the behavior of the application is the maximum possible value of WCs that we set to the average of writes per block in the TWWS of applications (see Table 2) in our experiments.

LR cache size and structure

Regarding the average size of TWWS in Table 2, we opt to set the size of LR to 256 blocks. Ideally, we want LR to keep the TWWS of the workload and HR to keep the read-only blocks. It is obvious that structures with higher associativities provide higher LR utilizations, but due to their higher complexity, they are not appropriate for large caches. On the other hand, direct-mapped structure has the least complexity, but the least utilization, too. Because the size of LR in our design is small, we adopt a high associativity. Fig. 15 shows different latency and energy values of LR with different associativities. By enhancing associativity, latency and dynamic power increase insignificantly, but LR utilization increases dramatically. Thus we set the associativity of LR to 16, because it yields proper utilization and reasonable energy/latency. In addition, with such an associativity level, our two comparator trees in DTR incur the least overhead of area and latency (i.e., two 15-comparator trees for our 16 × 16 LR structure).

LR and HR cache retention times

With respect to Fig. 12A and B, LR and HR retention times are selected to be 100 μs [52] and 40 ms [43], respectively. For each cache line, there are 4-bit counters working at 160 kHz in LR to track data retention, and 2-bit counters are used in HR working at 100 Hz. The use of larger RCs in LR is due to the lower retention time of blocks in this part of the cache, which requires a more accurate refreshing mechanism. Therefore a higher bit width enables tracking smaller slots of time, which leads to more accuracy [43].

As stated previously, the maximum possible value of WCs (i.e., 64) is selected based on the average write accesses per block in TWWS in Table 2.

The main overhead of the proposed architecture includes RCs, WCs, LR-to-HR and HR-to-LR buffers, and DAACS and DTR logics. The total capacity of RCs, WCs, and buffers is 6 KB, which is less than 1% of the cache capacity. Table 4 demonstrates the details of latencies and energies of LR, HR, and buffers extracted from CACTI 6.5 and its modified version [58, 59]. DAACS and DTR logics, on the other hand, incur an area overhead of less than 3% in our design.

7.1 Lifetime Evaluation

Although endurance of the STT-RAM cache is not a concern in this work, we have investigated the effects of our proposed architecture on the lifetime of the cache. To do so, we used the equation presented in Ref. [53]. Fig. 18 demonstrates intra and interset variations for both HR and LR in the proposed architecture with respect to the baseline nonsplit STT-RAM cache. Using our dynamic threshold mechanism, intra and interset variations had improvements of 4.79% and 21.8%, respectively. This is due to the fact that we moved frequently written blocks to LR, and consequently the migration of data blocks from LR to HR results in a more even distribution of writes on cache blocks in HR, which is critical for lifetime. Intra and interset variation improvements for LR are 79% and 68%, respectively. The reason for these large improvements is that a significant portion of the writes are conducted in LR, with a relatively small size. This large number of writes in a small cache results in a more uniform write distribution. Because there are different retention times, write variation and count are not the only factors affecting the lifetime. It was shown that lifetime and write energy are directly related to each other [64–66]. This fact can be used to calculate the lifetime of the LR and HR part. Table 6 demonstrates the lifetime improvement (LI) values for LR and HR. The third column shows the LI for the whole cache structure, which is the minimum of the LI values of LR and HR parts. Our results show that the lifetime is improved by up to 336% (111%, on average) with respect to the baseline.

7.2 Parallel vs Sequential Search Mechanism

Fig. 19 shows the dynamic power consumption reduction and performance overhead of the DAACS, in comparison to the parallel search mechanism. The figure shows an average reduction of 38.9% in the power consumption of the cache controller. Note that the performance overhead of using the power-efficient DAACS with respect to the parallel search mechanism is negligible (about 1%). This is mainly due to the efficient LR and HR block mapping and migration mechanisms we adopted.