Starfire DR-Detach Details
The software mechanism by which Dynamic Reconfiguration reduces resources available to the operating system is called detaching (see FIGURE 8-8).
Figure 8-8. DR-Detaching a Single System Board from a DSD
The DR-Detach process requires you to do strategic planning because it reduces the overall CPU and memory resources available to the DSD, and it may involve removing I/O interfaces that are critical to the operation of the machine. Before removing a system board, it is critical that all on-board I/O devices be closed since I/O drivers will fail to detach when an application has open device instances.
Note that removing CPU and memory resources from an already busy system (a system running short of memory and using excessive amounts of CPU) will significantly delay the DR-Detach process because the removed memory will have to be transferred to the swap disk.
DR-Detach Prerequisites and Actions
To guarantee a successful DR-detach process the following rerequisites must be fulfilled:
Pre-Boot Requirements
The following prerequisites require a system reboot to take effect:
All system memory must not be interleaved between system boards.
In the Solaris 2.5.1 and Solaris 2.6 5/98 operating environments, the dr_max_mem OBP environment variable must be set to a value equal to or greater than 512 Mbytes. The dr_max_mem OBP environment variable is not required after Solaris 7 5/99 because the required VM structures are dynamically allocated.
The soc driver requires the following entry in the /etc/system file to enable the suspend-resume/detach-attach features:
set soc:soc_enable_detach_suspend=1
In the Solaris 2.6 5/98 operating environment, the socal driver requires the following entry in the /etc/system file to enable the suspend-resume features:
set socal:socal_enable_suspend=1
The pln driver requires the following entry in the /etc/system file to enable the suspend-resume/detach-attach features:
set pln:pln_enable_detach_suspend=1
Activation of the kernel cage requires the following entry in the /etc/system file:
set kernel_cage_enable=1
System Requirements
The following prerequisites need to be fulfilled immediately before the DR-Detach operation takes place:
All on-board non-DR-compliant drivers need to be unloaded using the modunload(1M) utility.
Real-time processes are not allowed whene the kernel cage is being relocated because their behavior becomes unpredictable with the operating system quiescence. All real-time processes must be reassigned an alternate scheduling class using the priocntl(1) command or killed using the kill(1) command. The hostview(1M) interface provides the force option to override this requirement.
Processor sets or processes bound to CPUs hosted by the system board to be removed are not allowed. Processor sets must be torn down using the psrset(1M) command and processes bound to on-board processors must be unbounded using the pbind(1M)command.
All swap devices contained by the system board must be deleted using the swap(1M) command and removed from the /etc/vfstab file.
All on-board non-network I/O interfaces must be brought down.
All file systems hosted by the system board to be detached must be unmounted.
If AP is not used, disks hosted by the system board to be detached must be removed from Veritas VxVM control using the vxdiskadm(1M) command with the replace option.
Tape drives hosted by the system board that is to be detached must be empty or have a loaded tape sitting at BOT.
DR-Detach Actions
The following steps are automatically executed on both the SSP and DSD sides during the DR-Detach operation:
1. |
All free pages associated with the system board to be detached are locked to avoid further use by applications. All dirty pages are flushed to swap (swap could be either memory or disk). |
2. |
ISM
must be relocated to memory available on the remaining system. If there isn't available memory, dirty pages from the remaining system are flushed to disk to make room for the ISM footprint. |
3. |
If the kernel cage is located on the system board to be detached, the dr driver invokes the DDI_SUSPEND driver function on all I/O devices in the system and the operating system is quiesced (suspended) to allow the safe relocation of the entire address range associated with the system board to be detached. Note When the kernel cage is relocated, a target system board is selected (out of the remaining system) to absorb the entire memory address range assigned to the board being detached. The relocation process involves releasing the memory content from the target system board as defined in steps 1-2 above. Once the memory content is evacuated out of the target system board, the entire memory range for the board to be detached is transferred to the target system board through the copy-rename routine (running out of a single CPU cache). When the copy-rename routine completes, the memory controller on the target system board is updated to reflect the physical address range of the copied memory and the whole system is then resumed. |
4. |
If the board to be detached does not contain the kernel cage, its associated address range is removed from the system. If the kernel cage is located on the system board to be detached, the address range associated with the target system board is removed from the system. |
5. |
The memory controller on the detached board is disabled and the removed memory is taken out of the kernel memory pool. |
6. |
All processors resident on the system board to be detached are offlined and removed from the kernel configuration. The Boot processor is reassigned if needed. |
7. | |
8. | |
9. |