IBM Elastic Storage System architecture
This chapter describes the architecture, hardware, and software of the IBM Elastic Storage System (IBM ESS), including the following components:
•IBM ESS Management Server (IBM EMS)
•I/O/data servers
•Storage enclosures
This chapter also describes the different building block models and software components and features, such as:
•Operating system
•Ansible playbooks and container-based software upgrades
•IBM Spectrum Scale
•IBM Spectrum Scale RAID
 
Note: IBM ESS is an integrated packaged solution. The user should not install different kernel levels or drivers into the IBM ESS system. The user also should not run any non IBM ESS (client) application or non IBM ESS workloads on the I/O nodes or IBM EMS.
This chapter includes the following topics:
 
2.1 Architecture overview
In this section, we describe the overall hardware and software architecture of the IBM ESS solution.
An IBM ESS is defined as a combine set of hardware and software that are tightly coupled together and tested as a single unit. The major components of an IBM ESS solution release include the following items:
•Server hardware
•Storage hardware
•IBM Spectrum Scale software
•Embedded Red Hat Enterprise Linux (RHEL) operating system
The minor components, which are integration-tested with the major components, include (but not limited to) the following items:
•Server firmware
•Server host bus adapters (HBAs) and related firmware
•NVIDIA network adapter, drivers, firmware, and network switches
•Storage enclosure and drive firmware
•IBM racks, power distribution units (PDUs), and cabling management
In addition, many of the hardware and software components of the IBM ESS solution communicate to each other over an IP network. IBM ESS requires all networking connections to be in place and provisioned to install the system. This IP network can use NVIDIA network switches, which are tested with IBM ESS by IBM, or the network switches can be provided by the client. These IP network components consist of the following switches:
•Low speed 1 Gb Ethernet (GbE) network switches for management and service networks
•High-speed network switches for data read and write over a high-speed data network
2.2 Hardware components
In this section, we describe the following hardware components of the IBM ESS:
•Solution models
•Server hardware (IBM ESS 5000 uses IBM POWER9 processor-based servers.
IBM ESS 3500, IBM ESS 3200, and IBM ESS 3000 use x86 servers.)
•Storage enclosures
•Network interface cards (NICs) and network switches
•Rack
IBM ESS is a rack-mounted IBM Spectrum Scale storage solution. The initial IBM ESS storage building block in an IBM Spectrum Scale cluster includes the following components:
•One IBM EMS (one is required for every IBM Spectrum Scale cluster).
•Two I/O data servers.
•The IBM ESS 3500 supports up to four storage enclosures. The IBM ESS 5000 supports up to seven or nine storage enclosures (depending on the IBM ESS 5000 model).
Third-generation IBM ESS models include various models that provide NVMe, serial-attached SCSI (SAS), or Serial Advanced Technology Attachment (SATA) hard disk drives (HDDs) in selected sizes.
The IBM ESS 3500, which is the latest third-generation IBM ESS model, was announced and delivered in 2Q2022. The IBM ESS 3500 design is based on the IBM ESS 3200, but has a faster x86 based processor and various design improvements that improve serviceability. The IBM ESS 3500 was expanded in 3Q2022 to include support for up to four storage enclosures.
The IBM ESS 5000, which is also part of the third-generation IBM ESS models, was announced and delivered in 3Q2020. IBM ESS 5000 is a fully integrated IBM Spectrum Scale HDD storage building block with 1 - 9 storage enclosures (depending on the model) and a pair of POWER9 processor-based IBM Spectrum Scale Data servers.
The IBM ESS 3200, which also is a third-generation IBM ESS model, was announced and delivered in 2Q2021. IBM ESS 3200 is a fully integrated 2U storage building block that has 12 or 24 NVMe drives in the front of the 2U24 enclosure. It features a faster pair of PCI Gen4-based x86 server canisters in the back of the 2U enclosure.
The IBM ESS 3000, which was the first of the third-generation IBM ESS models, was announced and delivered in 4Q2019. IBM ESS 3000 is a fully integrated 2U storage building block that has 12 or 24 NVMe drives in the front of the 2U24 enclosure, and a pair of x86 server canisters in the back of the 2U enclosure.
First- and second-generation IBM ESS storage building blocks contain a pair of
IBM POWER8® processor-based IBM Spectrum Scale Data servers.
2.2.1 IBM Elastic Storage System generations
This section describes the different generations and various IBM ESS models.
All IBM ESS generations can coexist in an IBM Spectrum Scale cluster.
Third-generation IBM ESS models
A third generation of IBM ESS was announced in November 2019 with the IBM ESS 3000 NVMe flash storage system. This announcement was followed by the announcement of the IBM ESS 5000 HDD offering in July 2020, the IBM ESS 3200 offering in May 2021, and the IBM ESS 3500 offering in May 2022.
The third-generation IBM ESS models include the following IBM ESS unique IBM machine types:
•IBM ESS 3500: IBM machine type 5141-FN2 for the 2U24 form factor IBM ESS with embedded NVMe flash storage, and optionally 1 - 4 of the 4U105 form factor IBM ESS Storage Enclosures model 5147-102
•IBM ESS 5000:
 – IBM machine type 5105-22E for the IBM ESS IBM POWER9 processor-based data servers, management server, and protocol nodes
 – IBM machine type 5147-092 or 5147-106 for the IBM ESS IBM Storage Enclosures
•IBM ESS 3200: IBM machine type 5141-FN1 for the 2U24 form factor IBM ESS with embedded NVMe flash storage. IBM announced the hardware withdrawal for IBM ESS 3200 on 23 August 2022, which is effective 31 March 31 2023. On or after the effective date of withdrawal, you can no longer order this product directly from IBM. The IBM ESS 3500 is the replacement.
•IBM ESS 3000: IBM machine type 5148-AF8 for the 2U24 form factor IBM ESS with embedded NVMe flash storage. New orders for IBM ESS 3000 were withdrawn from marketing effective 15 July 2022. The IBM ESS 3200 is the replacement.
These machine types uniquely identify these hardware machines as part of an IBM ESS solution.
Second-generation IBM ESS models
The second generation of IBM Elastic Storage Server (IBM ESS) was announced in April 2017.
Second-generation IBM ESS models do not include a solution IBM machine type. The
IBM ESS server and storage hardware components feature the following IBM ESS unique machine types:
•IBM machine type 5148 for IBM ESS POWER8 processor-based servers
•IBM machine type 5147 for IBM ESS IBM Storage Enclosures
These machine types uniquely identify these hardware components as part of an IBM ESS solution. The models of second-generation IBM ESS are available in the following categories:
•Solid-state drive (SSD) flash storage IBM ESS models
•HDD IBM ESS models
•Hybrid IBM ESS models that consist of enclosures of SSD flash drives and HDDs
New orders for second-generation IBM ESS models can no longer be miscellaneous equipment specification (MES) upgraded to add more storage enclosures. They were announced to be withdrawn from marketing effective 31 December 2021.
 
Note: The second-generation IBM ESS models are still in full service and supported by IBM. At the time of writing, no end of service (EoS) date has been announced yet.
For more information about second-generation IBM ESS models, see Appendix A, “IBM Elastic Storage System models” on page 65.
First-generation IBM ESS
The first generation of IBM ESS was announced in October 2014. It includes a solution IBM machine type 5146. GL and GS models are at End of Support.
2.2.2 IBM Elastic Storage System 5000
IBM ESS 5000 is the IBM HDD-based IBM Spectrum Scale storage platform. This storage platform provides high capacity and high-performance IBM Spectrum Scale storage by using HDD storage drives.
IBM ESS 5000 can include 6 TB, 10 TB, 14 TB, 16 TB, and 18 TB HDD drives depending on the model. At the time of writing, the IBM ESS 5000 SL model supports a superset of the drive sizes in the IBM ESS 5000 SC.
For more information about the full collection of manuals and documentation for
IBM ESS 5000, see IBM Elastic Storage System documentation.
IBM ESS 5000 is based on IBM Spectrum Scale and IBM Spectrum Scale RAID, as are all IBM ESS models. IBM ESS 5000 includes the following features:
•IBM POWER9 processor-based data servers, which provide the latest advances in
IBM POWER® high memory and memory bandwidth, PCI Gen4-based internal bus transfer speeds for higher internal transfer rates, and support of faster NICs.
•Containerized Red Hat Ansible playbooks that provide significantly improved ease of use and orchestration of complex IBM ESS administration tasks, such as cluster configuration, file system creation, and code update.
•Higher density and better HDD performance per rack than previous HDD-based IBM ESS models.
•IBM ESS 5000 supports nondisruptive capacity upgrade. You can add 5147-092 to an SLx model or a 5147-106 to a SCx model. Therefore, an installed SC1 to SC8 model, or SL1 to SL6 model, can be upgraded to any model configuration up to SC9 or SL7, without causing downtime or an interruption in service. (the additional HDD drives must be of the same size as the previous drives).
Figure 2-1 shows the IBM ESS 5000 SL models.
Figure 2-1 IBM ESS 5000 SL models
Figure 2-2 shows the IBM ESS 5000 SC models.
Figure 2-2 IBM ESS 5000 SC models
2.2.3 IBM Elastic Storage System 3500
IBM ESS 3500 is a hybrid model offering high-performance local NVMe storage and capacity storage through SAS or SATA connected storage enclosures.
IBM ESS 3500 can include 12 or 24 drives of 3.84, 7.68, 15.36, or 30.74 TB NVMe drives for user data. Alternatively, four NVMe drives are used exclusively for log tip use. For more information, see IBM Elastic Storage System 3500.
IBM ESS 3500 can include 1 - 4 storage shelves of SAS or SATA drives. The first storage shelves can contain 52 or 102 drives. Extra shelves contain 102 drives. The drives are 10, 14, 18, or 20 TB. For more information, see IBM Elastic Storage System 5147-102 Storage Enclosure.
IBM ESS 3500 is based on IBM Spectrum Scale and IBM Spectrum Scale RAID.
IBM ESS 3500 supports non-disruptive capacity upgrade. You can add 5147-102 storage shelves. IBM ESS 3500 supports up to four storage shelves, but all HDDs must be the same size.
Figure 2-3 on page 21 shows the IBM ESS 3500 models.
Figure 2-3 IBM ESS 3500 models
2.2.4 Positioning various IBM Elastic Storage System models
The different models of IBM ESS storage building blocks provide varying levels of performance and capacity according to the needs of the IBM Spectrum Scale storage pool in which they are deployed. The various models of third-generation IBM ESS can provide NVMe or SAS or SATA storage in selected disk sizes.
Figure 2-4 shows a general positioning of the available models of IBM ESS.
Figure 2-4 General positioning of available IBM ESS models
IBM ESS models are available in different categories. The “x” in the models (see Figure 2-5) denotes the number of storage enclosures in that model.
Consider the following points:
•IBM ESS 3500 models are built on NVMe flash storage for high performance, and optionally high capacity 4U102 HDD storage enclosures. Each IBM ESS 3500 canister supports up to four PCIe4 cards. A single card either supports up to two high-speed networking interfaces or up to two high-capacity storage enclosures.
•The IBM ESS 5000 SL model is built on high-capacity 5U92 HDD storage enclosures.
•The IBM ESS 5000 SC model is built on high-capacity 4U106 HDD storage enclosures.
•Supporting IBM ESS components include the following ones:
 – IBM ESS protocol node (see 2.2.8, “Protocol nodes” on page 32).
The available IBM ESS models are shown in Figure 2-5.
Figure 2-5 IBM Elastic Storage System models
IBM ESS usable capacity is a percentage of the raw capacity. The usable capacity varies, depending on the IBM Spectrum Scale RAID parity that is selected for use. Generally, approximately 73% of the raw capacity is the usable capacity when the default 8+2P parity is used, which is the most commonly selected parity for most scenarios.
IBM or IBM Business Partners can provide the exact usage capacities of the various models on request by using the IBM File Object Solution Design Engine (FOS DE) or the Storage Modeler (StorM) to calculate the exact usable capacity. See your IBM or IBM Business Partner representative for assistance in determining the usable capacity that your IBM ESS model and implementation provides.
For more information about the various IBM ESS models specifications, see Appendix A, “IBM Elastic Storage System models” on page 65.
2.2.5 IBM Elastic Storage System Management Server
Released with the IBM ESS 3200 was IBM EMS. This management system can be used with the IBM ESS 3000, and it is required on IBM ESS 3500, IBM ESS 3200, and IBM ESS 5000. IBM EMS (IBM machine type 5105-22E) is based on the IBM POWER9 processor.
The POWER9 processor-based IBM EMS server features the following specifications:
•1x DD2.3 20 Small Cores, 190W/225W, 2.5 GHz or 2.9 GHz
•128 GB default memory, no NVDIMMs (nonvolatile dual inline memory module)
•No SAS HBAs
•The same network interface options as available on all third-generation IBM ESS models
Every IBM Spectrum Scale cluster with IBM ESS requires a minimum of one IBM EMS. This IBM POWER9 processor-based server is used to manage and deploy the IBM ESS I/O data server nodes. This server performs the following tasks:
•Runs an IBM ESS GUI server for managing IBM ESS in the cluster.
•Can optionally provide other IBM Spectrum Scale functions, such as a quorum node.
•Runs the newer management and ease of installation enhancements of third-generation IBM ESS by using Red Hat Ansible orchestration.
The initial IBM ESS storage building block in an IBM Spectrum Scale cluster requires an
IBM EMS. The purpose of the IBM EMS is a central control point to manage multiple
IBM ESS storage building blocks in the IBM Spectrum Scale cluster.
The third-generation IBM ESS 3500, IBM ESS 3200, and IBM ESS 5000 models require a POWER9 processor-based IBM EMS (IBM machine type 5105-22E) running RHEL Little Endian. This Little Endian IBM EMS can manage IBM ESS generation three and two.
The second-generation IBM ESS models use an IBM EMS (IBM machine type 5148-21L) running RHEL Little Endian. This Little Endian IBM EMS can manage second-generation
IBM ESS and IBM ESS 3000, but cannot manage first-generation IBM ESS or third-generation IBM ESS 3500, IBM ESS 3200, or IBM ESS 5000.
If you have a mix of generations of IBM ESS in the same IBM Spectrum Scale cluster, you must have a suitable number of IBM EMSs to manage the IBM ESS generations.
IBM ESS GUI and Call Home
The IBM EMS runs the IBM ESS GUI and is the central Call Home control point for the
IBM ESS hardware. IBM ESS Call Home monitors server and storage hardware. If a hardware failure event occurs, the IBM EMS can be configured to initiate a Call Home service action to IBM Service and Support.
2.2.6 Data Server hardware
The IBM ESS solution integrates a pair of IBM Spectrum Scale Data Servers that are cross-connected to all storage in the IBM ESS, which provides an IBM Spectrum Scale storage building block. Second-generation IBM ESS models use IBM Power servers for the data servers. The IBM ESS 3500, IBM ESS 3200, and IBM ESS 3000 use an x86-based pair of servers for data servers. The IBM ESS 5000 uses a POWER9 processor-based pair of data servers.
The server hardware, firmware, and drivers are all tested, integrated, and supported as a solution by IBM and as part of the overall IBM ESS solution.
In the following sections, we describe the server specifics for third-generation and second-generation IBM ESS models.
Second-generation IBM ESS Data Servers
All second-generation IBM ESS models use a pair of POWER8 processor-based servers for the integrated IBM Spectrum Scale Data Servers. The second-generation IBM ESS models use POWER8 processor-based (machine type 5148-22L) servers as
IBM Spectrum Scale Data servers. These 5148-22L servers all run the RHEL Little Endian operating system.
Third-generation IBM ESS Data Servers
With IBM ESS 5000, IBM introduced a new POWER9 processor-based IBM ESS server:
IBM Machine Type 5105-22E. This server is used in the IBM ESS 5000 only.
The 5105-22E is configured in three separate ways to provide of one of the following server functions in the IBM ESS 5000 system:
•IBM EMS
•Protocol node
•IBM ESS Data Server
All these servers are based on the IBM POWER9 processor-based architecture and include the same machine type and model (MTM) (5105-22E). In the IBM ESS 5000 configuration, they are differentiated by feature codes that specify which of the three possible roles this 5105-22E is performing.
The 5105-22E POWER9 processor-based servers feature increased memory and backplane bandwidth. In particular, the usage of PCI Gen4 bus allows each of the ports on the dual-port NICs to run at full rated speed.
The IBM ESS 3500, IBM ESS 3200, and IBM ESS 3000 2U24 form factor includes two x86 server canisters in the rear of the 2U24 storage enclosure, which provide a small footprint and a dense and integrated NVMe flash IBM Spectrum Scale storage building block. These two x86 server canisters run IBM Spectrum Scale RAID as a pair of IBM Spectrum Scale Data Servers, which are cross-connected to all storage in the 2U24 enclosure for high availability (HA).
IBM ESS 3500 Storage Enclosure 5147-102 is an enterprise-class, fully redundant SAS or SATA enclosure for IBM ESS 3500 5141-FN2. The enclosure supports up to one hundred and two 3.5-inch or 2.5-inch drives of any density and spindle speed, and up to four 5147-102 Storage Enclosures can be attached to IBM ESS 3500 (5141-FN2) to provide up to 8 PB of HDD capacity.
 
Note: The IBM ESS 3000 can be supported by a POWER9 processor-based 5105-22E IBM EMS or a POWER8 processor-based 5148-21L IBM EMS. This management server can be an IBM ESS 5148-21L that is used to manage other IBM ESS models in this same IBM Spectrum Scale cluster. If this cluster is a new IBM Spectrum Scale cluster, order a POWER9 processor-based 5105-22E IBM EMS to manage your third-generation IBM ESS IBM Spectrum Scale cluster.
Network interface cards
The IBM ESS solution features the following types of networks:
•Management and service networks that require basic, low-speed 1 GbE.
•All IBM ESS models support high-speed data networks (Extended Data Rate (EDR) InfiniBand, and 100 GbE, 40 GbE, 25 GbE, or 10 GbE) over which user data is read and written.
•Specific IBM ESS models support even higher speed data networks: IBM ESS 3500,
IBM ESS 3200, and IBM ESS 5000 support High Dynamic Range (HDR)
InfiniBand or 200 GbE.
The NICs on the IBM ESS Data Servers and the IBM EMS service these various network types.
Every IBM ESS node has a 1 GbE NIC for connection to the management and service networks.
Every IBM EMS, in addition to the 1 GbE NIC, must have one of each of the high-speed network NICs being used on the IBM ESS Data Servers. This configuration allows the
IBM EMS to monitor, manage, and measure the activity on the high-speed data networks.
 
Note: Specific IBM ESS models support NICs that provide Virtual Protocol Interconnect (VPI). VPI allows individual ports of the dual-port network card to be configured as InfiniBand or 200 GbE. If you use the VPI feature to reconfigure your NIC, make sure to configure or add suitable NICs to your IBM EMS so that the IBM EMS can attach to that network.
On the IBM ESS Data Server nodes, a sufficient quantity of high-speed data network NICs must be configured to provide the wanted IBM ESS bandwidth.
Various high-speed data network NICs are available:
•200 GbE
•100 GbE
•40 GbE
•25 GbE
•10 GbE
A VPI-enabled InfiniBand or 100 GbE network card is available on all third-generation
IBM ESS models. This dual-port InfiniBand and Ethernet VPI network adapter is based on the NVIDIA ConnectX technology. Each of the two ports on the card can be independently configured for 100 GbE or InfiniBand.
The HDR InfiniBand or 100 GbE network card is available on IBM ESS 5000. This dual-port InfiniBand-HDR and Ethernet VPI network adapter is based on the NVIDIA ConnectX-7 card with VPI enabled. Each of the two ports on the card can be independently configured for
100 GbE or HDR-100.
The HDR InfiniBand or 200 GbE network card is available on IBM ESS 3500. This dual-port InfiniBand-HDR and Ethernet VPI network adapter is based on the NVIDIA ConnectX-6 card with VPI enabled. Each of the two ports on the card can be independently configured for
100 GbE or HDR-200.
 
Note: Consider the following points:
•IBM ESS 3500 and IBM ESS 5000 use a PCIe Generation 4 bus, which fully supports both ports of the 200 GbE or InfiniBand NICs to run at full rated speed.
IBM and IBM Business Partners use the IBM File and Object Design Engine tool to estimate and configure an appropriate amount of NIC bandwidth.
Experience shows that throughput on IBM ESS is often limited to the amount of network NIC bandwidth that is available. The best performance is achieved over InfiniBand.
•Suitable network cables and connectors must be specified and ordered for your specific network requirements. If you use the VPI feature to reconfigure your high-speed network ports, make sure that your IBM EMS also can access the newly configured network.
•RDMA over Converged Ethernet (RoCE) might be available on third-generation
IBM ESS models for specific configurations, networks, and environments only. If you have this requirement, submit a Request for Price Quotation (RPQ) to IBM by contacting your IBM or IBM Business Partner representative.
•For more information about estimating and sizing your performance, or for specifying the suitable network cables and connectors, contact your IBM or IBM Business Partner representative.
You can intermix high-speed data network NIC types if enough data server slots are available. Each of the data servers in an IBM ESS must have same number and type of NICs.
Configure your IBM ESS to have the suitable type and number of NICs for your throughput requirements. It is supported and a common practice on IBM ESS Data Servers to bond multiple ports for increased availability and throughput.
2.2.7 Storage enclosures
A storage enclosure is a specialized storage drawer holds and powers flash or HDD storage, while providing the connections to allow them to communicate to one or more separate firmware service modules that run the entire enclosure.
First-generation IBM Elastic Storage Server
The first generation of IBM ESS was announced in October 2014. It was available in 2U24 storage enclosure (GS models) with SSD or HDD storage, or high capacity GL storage models that used IBM 1818-80E 4U60 HDD Storage Enclosures. New orders and storage enclosure upgrades are withdrawn from marketing and no longer available, effective
19 January 2018.
The announced EoS date for the first-generation IBM ESS models is 31 December 2021. For more information, contact your IBM or IBM Business Partner representative.
For more information about the first-generation IBM ESS Storage Enclosures, see Appendix A, “IBM Elastic Storage System models” on page 65.
Second-generation IBM Elastic Storage Server
The second generation of IBM ESS was announced in April 2017 and was available for order until the last order date of 20 December 2020.
The following components have been withdrawn from marketing:
•IBM 5147-084 HDD Storage Enclosures to add to a second-generation IBM ESS GLxS model until 30 June 2021.
•IBM 5147-024 SSD Storage Enclosures to add to a second-generation IBM ESS GSxS or GH model until 31 December 2021.
•IBM 5147-106 HDD Storage Enclosures to add to a second-generation IBM ESS GLxC model until 31 December 2021.
 
Note: The second-generation IBM ESS is still in full service and supported by IBM. No EoS date was announced at the time of this writing.
Third-generation IBM Elastic Storage Server
The following third-generation IBM ESS models were announced, and they use the associated storage enclosures:
•IBM ESS 3500 was announced and delivered in 2Q2022. It is a self-contained 2U24 form factor storage enclosure with NVMe flash storage in the front, and PCI Gen4 x86-based IBM Spectrum Scale Data Servers in the back. The IBM ESS 3500 was expanded in 3Q2022 to include support for up to four storage enclosures (IBM machine type 5147-102).
•IBM ESS 3200 was announced and delivered in 2Q2021. It is a self-contained 2U24 form factor storage enclosure with NVMe flash storage in the front, and PCI Gen4 x86-based IBM Spectrum Scale Data Servers in the back.
•IBM ESS 5000 was announced and delivered in 3Q2020. It uses 5U92 (IBM machine type 5147-092) or 4U106 (IBM machine type 5147-106) HDD storage enclosures.
•IBM ESS 3000 was announced and delivered in 4Q2019. It is a self-contained 2U24 form factor storage enclosure with NVMe flash storage in the front, and x86-based
IBM Spectrum Scale Data Servers in the back.
In the following sections, we describe storage enclosures that you are likely to encounter in working with second-or third-generation IBM ESS. Machine type 5147 is used for the
IBM ESS IBM Storage Enclosures. Machine type 5147 uniquely identifies these storage enclosure hardware components as part of an IBM ESS.
2U24 (5147-024)
The 5147-024 2U Storage Enclosure is used for SSDs in the second-generation IBM ESS GSxS and second-generation GH hybrid models. The following sizes of SSDs are available:
•3.84 TB
•15.36 TB
The 5147-024 in the IBM ESS is available in a fully populated 24-drive configuration only.
The IBM 5147-024 Storage Enclosure is shown in Figure 2-6.
Figure 2-6 IBM 5147-024 Storage Enclosure
5U84 (5147-084)
The 5147-084 5U Storage Enclosure is available to hold HDDs in the second-generation
IBM ESS GLxS models and the GH hybrid models. The following sizes of HDDs are or were available:
•4 TB
•8 TB
•10 TB
•14 TB
The IBM 5147-084 Storage Enclosure is shown in Figure 2-7.
Figure 2-7 IBM 5147-084 Storage Enclosure
The 5147-084 in the second-generation IBM ESS is available in a fully populated configuration only.
The first 5147-084 Storage Enclosure in an IBM ESS also contains two SSDs that hold internal IBM Spectrum Scale RAID metadata. The capacity of these two SSDs is not visible or available as user capacity for the IBM Spectrum Scale or IBM ESS file system.
4U106 (5147-106)
The 5147-106 4U Storage Enclosure is an extreme density storage enclosure that holds HDDs and available in second-generation IBM ESS GLxC and third-generation IBM ESS 5000 SC models. The following sizes of HDDs are or were available:
•10 TB
•14 TB
•16 TB (IBM ESS 5000 only)
The 5147-106 in the IBM ESS is available in a fully populated configuration only.
The first 5147-106 in an IBM ESS also contains two SSDs that hold internal IBM Spectrum Scale RAID metadata. The capacity of these two SSDs is not visible or available as user capacity for the IBM Spectrum Scale IBM ESS file system.
The IBM 5147-106 Storage Enclosure is shown in Figure 2-8.
Figure 2-8 IBM 5147-106 Storage Enclosure
The IBM 5147-106 Storage Enclosure is an unusually deep enclosure. Physical planning for this enclosure must be performed. Also, the racking, aisle widths, and rear clearances all must be reviewed and approved by the IBM Installation Planning representative.
 
Note: Because of the depth of this 5147-106 enclosure, the IBM ESS models that use 5147-106 must be installed in the IBM 7965-S42 rack. These enclosures (and associated IBM ESS SC models) cannot be installed in the IBM 7014-T42 rack.
Second-generation IBM ESS Hybrid models storage enclosures
Second-generation IBM ESS Hybrid models combined 5147-024 Storage Enclosures with flash SSD storage and 5147-084 enclosures with spinning HDDs and in a single IBM ESS building block. These IBM ESS Hybrid models provided a combination of drawers of 2U24 SSD and 5U84 HDDs, which combines the speed and low latency of flash SSD storage with the density and affordability of spinning HDD.
This configuration provides a flexible, smaller footprint than using a separate flash SSD storage building block and disk HDD storage building block. Where the IBM Spectrum Scale IBM ESS configuration indicates the usage of the IBM ESS GH hybrid models, these
IBM ESS hybrid storage building blocks were used to give better storage density and a smaller footprint at a lower cost.
Third-generation IBM ESS 3500, IBM ESS 3200, and IBM ESS 3000 Storage Enclosures
The third-generation IBM ESS 3500, IBM ESS 3200, or IBM ESS 3000 with NVMe is available with a 2U24 storage enclosure, with slots for 24 NVMe storage drives in the front of the enclosure. The IBM ESS 3500, IBM ESS 3200, and IBM ESS 3000 use different physical storage enclosures (they are not the same). Conversion from one model to another is not possible. These different 3x00 storage enclosure models are similar in that they provide front panel hot-swappable drive access. Rear access is provided to the two power supplies and two x86-based IBM Spectrum Scale Data Servers.
Figure 2-9 shows a front view of the IBM ESS 3500 and IBM ESS 3000 Storage Enclosures.
Figure 2-9 Sample IBM ESS 2U24 front panel accessible NVMe Storage Enclosure
 
Note: IBM ESS 3500 and IBM ESS 3000 use enterprise-class NVMe drives. The
IBM ESS 3200 supports either NVMe or IBM FlashCore®, depending on size.
IBM ESS 3x00s use a mirrored set of internal boot disks for the x86-based IBM Spectrum Scale Network Shared Disk (NSD) Data Servers.
5U92 (5147-092)
The 5147-092 5U Storage Enclosure is available to hold HDDs in the IBM ESS 5000 SL models. The following sizes of HDDs are available:
•6 TB
•10 TB
•14 TB
•16 TB
The IBM 5147-092 Storage Enclosure is shown in Figure 2-10.
Figure 2-10 IBM 5147-092 Storage Enclosure
The 5147-092 in the IBM ESS 5000 SL model is available in a fully populated configuration only. The first 5147-092 in an IBM ESS 5000 also contains two SSDs that hold internal
IBM Spectrum Scale RAID metadata. The capacity of these two SSDs is not visible or available as user capacity for the IBM Spectrum Scale or IBM ESS file system.
4U102 (5147-102)
The 5147-102 4U Storage Enclosure is available to hold HDDs in the IBM ESS 3500 models. The following HDD sizes are available:
•10 TB
•14 TB
•18 TB
•20 TB
The IBM 5147-102 Storage Enclosure is shown in Figure 2-11.
Figure 2-11 IBM 5147-102 Storage Enclosure
IBM ESS 3500 Storage Enclosure 5147-102 is an enterprise-class, fully redundant SAS or SATA enclosure for IBM ESS 3500 5141-FN2. The enclosure supports up to one hundred and two 3.5-inch or 2.5-inch drives of any density and spindle speed. Up to four 5147-102 Storage Enclosures can be attached to IBM ESS 3500 (5141-FN2), which provides up to 8 PB of HDD capacity.
2.2.8 Protocol nodes
The IBM ESS IBM Spectrum Scale protocol node feature code makes it possible to order POWER9 processor-based 5105-22E server nodes with a specific hardware configuration, which are tested and tuned by IBM to provide IBM Spectrum Scale protocol node (also known as Cluster Export Services (CES) services. The MTM for protocol nodes that are ordered with an IBM ESS is IBM machine type 5105-22E.
The default protocol node hardware features the following configuration:
•5105-22E POWER9 processor-based model with two 10 core POWER9 processors
•Runs RHEL Little Endian
•192 GB or greater memory in default configuration
•1 GbE 4-port network adapter that is required for management network
•Up to seven network adapters
All NICs that are available for other 5105-22E Data Servers are for 5105-22E servers that are ordered as protocol nodes.
 
Note: The 5105-22E protocol nodes are supported for management by the POWER9 processor-based 5105-22E IBM EMS node only.
5105-22E protocol nodes may coexist in an IBM Spectrum Scale cluster with previous generation IBM ESS 5148-22L protocol nodes and with other non-5148-22L IBM Power Little Endian servers that also are protocol nodes.
The IBM ESS 3500 can run embedded virtualized protocol nodes. The limitations for running CES require no more than 1024 Network File System (NFS) active connections or 512 Server Message Block (SMB) or Common Internet File System (CIFS) active connections. Object support is not supported on the embedded virtualized protocol nodes.
 
Note: IBM Spectrum Scale requires that in any one IBM Spectrum Scale cluster, all protocol nodes must be of the same architecture and of the same Endian. The following types of IBM Spectrum Scale protocol nodes architecture and Endian are supported:
•IBM Power servers running RHEL Little Endian, including the IBM 5105-22E POWER9 processor-based protocol node, and the previous generation POWER8 processor-based 5148-22L protocol node
•IBM Power servers running RHEL Big Endian (for example, IBM 8247-2xL servers)
•x86 servers running supported Linux operating systems
For more information, see the IBM Spectrum Scale Functional Support Matrices in the
IBM Spectrum Scale FAQ at IBM Documentation.
If you are ordering IBM Power 5105-22E servers as protocol nodes, you might have other IBM Power RHEL Little Endian protocol nodes in the IBM Spectrum Scale cluster that are not IBM 5105-22E protocol nodes or previous generation 5148-22L protocol nodes. These other protocol nodes are installed separately and deployed, upgraded, and managed by the customer. The IBM EMS tool sets are not supported and cannot be used for the management of operating system, kernel, network manager, systemd, OpenFabrics Enterprise Distribution (OFED), or firmware on the non-5105-22E or non-5148-22L protocol nodes.
2.2.9 Networking switches
The IBM ESS solution requires IP or InfiniBand networking switches to interconnect the
IBM ESS hardware components.
As a best practice, purchase and use NVIDIA networking switches. These network switches are tested by IBM to work with IBM ESS.
Clients also can provide and support their own Ethernet or InfiniBand networking switches.
Types of networks that are used by IBM ESS
The IBM ESS features the following types of networks that are used to interconnect the
IBM ESS components:
•Management and service networks, which must have separate 1 GbE networks for management and service traffic
•Data network, which is a high-speed network for the data that is written into and read out of the IBM Spectrum Scale IBM ESS file systems
Regardless of the source and type of network switches between the IBM ESS, the IBM Spectrum Scale cluster and the users, the networking must be solid, robust, reliable, and provide consistent latency and response time. Non-blocking, high-speed switches are highly recommended.
Experience shows that IBM Spectrum Scale and IBM ESS can easily generate throughput rates that can overwhelm an over-subscribed network. IBM Spectrum Scale provides a network load generation testing tool (nsdperf), which is available to anyone as an open source repository on GitHub. This tool should be used to test network performance, network latency, network topology, and network readiness level before IBM ESS is installed.
For more information about the nsdperf tool, see this GitHub repository.
Management network switches
IBM ESS components require a 1 GbE Ethernet IP connection for management and service network traffic. As part of an IBM ESS order, IBM can supply the NVIDIA 1 GbE management network switch for IBM machine type 8831-S52 or 8831-S48.
As a best practice, order and use this 1 GbE network management switch because it is integrated into the IBM ESS solution at the manufacturing stage. This integration allows the entire IBM ESS to be installed and initially tested without external network dependencies.
Alternatively, the client can provide a 1 GbE network switch management infrastructure. In this case, the client is responsible for providing the suitable management IP networking port counts and network switch configuration and definitions, including VLANs.
High-speed data network switches
IBM ESS components also require a high-speed data network IP or InfiniBand connection for the reading and writing of user data. As part of IBM ESS testing, IBM tests with NVIDIA data network switches.
As a best practice, order and use the NVIDIA data network switches where possible. These switches are tested and integrated with the IBM ESS solution.
NVIDIA high-speed network switch drivers and firmware is tested and delivered integrated within the IBM ESS software solution stack.
Alternatively, the client can provide their own 1 GbE management network and high-speed data network switch infrastructure. In this case, the client is responsible for provisioning networking port counts, firmware, network configuration, and definitions. The client also is responsible for maintaining and troubleshooting any networking issues.
2.2.10 Racking
The IBM ESS solution can be ordered with or without the building block being integrated into an IBM rack.
An IBM ESS solution that is ordered with feature codes that specify integration into the rack in IBM manufacturing result in a fully integrated, tested, and pre-cabled solution.
The following IBM racks are available for IBM ESS:
•IBM 7014-T42 Enterprise Rack
•IBM 7965-S42 Enterprise Slim Rack
Both racks provide 42 EIA (42U) of usable space for IBM ESS components and network switches. A choice of various electrical PDUs is available to be specified for inclusion in the rack side areas. The difference between the racks is the width and depth.
It is advantageous to order IBM ESS as a racked system because IBM ESS Manufacturing delivers an IBM ESS with the rack mostly pre-cabled and pre-tested. This configuration provides faster time to install with higher quality.
The IBM 7014-T42 is the default for most models of IBM ESS, except for the IBM ESS GLxC models and IBM ESS 5000 SC models.
For the IBM ESS models that use the 5147-106 Storage Enclosure, use the IBM 7965-S42 rack, which is designed to accommodate the deep IBM 5147-106 Storage Enclosure. The IBM 7965-S42 rack is narrower and fits on a 24-inch floor tile. This rack might better fit clients with tight racking requirements and space. It can be ordered for other IBM ESS models where the client wants an alternative to the IBM 7014-T42 rack.
It is also possible to order IBM ESS without a rack. In this case, the IBM ESS components are assembled and tested by IBM Manufacturing. Then, the IBM ESS components (servers, storage, networking switches, and cables) are uninstalled from the manufacturing rack, labeled, and shipped.
If installing the IBM ESS in a customer-supplied rack, it is the customer’s responsibility to correctly specify all physical planning for electrical power, connectors, cables, and racking locations.
Clients also must plan the physical installation and cabling of the IBM ESS components in the client-supplied rack. An IBM TSS Services contract is available and recommended to perform physical planning, specification, and physical installation of the IBM ESS components in a client rack on behalf of the client.
For the rackless IBM ESS order, the solution must be assembled and integrated into the rack at the client site. Extra IBM Technical Support Services (TSS) installation hours for installation might be required. For more information, see your IBM TSS representative.
For the purposes of cable management and serviceability, each IBM ESS building block is integrated and deployed within the same rack.
If you require to split the IBM ESS components between adjacent racks, ask your IBM or
IBM Business Partner representative to submit an RPQ special bid request. IBM examines the request and responds regarding whether IBM can grant and support the specific requested physical racking.
2.3 Software components
This section describes the software components that are used on the IBM ESS solution.
2.3.1 Software solution stack overview
The IBM ESS solution provides an integrated and tested stack of these operating system, adapter drivers, firmware, IBM Spectrum Scale software, management software, and installation scripts into a full IBM ESS software stack. This software solution stack is supported as an integrated IBM Spectrum Scale storage building block solution by
IBM Service and Support.
In the following sections, we briefly describe the following IBM ESS solution stack software components:
•RHEL operating system
•IBM Spectrum Scale high-performance parallel file system
•IBM Spectrum Scale RAID software-defined storage (SDS)
•IBM ESS solution installation and management software, which includes (but not limited to) the following components:
 – IBM ESS-specific documentation for installation and upgrade scripts
 – Other tools for the IBM System Services Representative (IBM SSR) to use while installing IBM ESS, such as essinstallcheck, essnetverify, and essutils.
•Third-generation IBM ESS systems deploy a container-oriented management software stack in the IBM EMS that includes Red Hat Ansible playbooks for enhanced ease of use for installation and orchestration. Installation is faster, and skill requirements are significantly lower than previous-generation IBM ESS models.
IBM preinstalls the complete integrated and tested IBM ESS solution stack on the IBM ESS Data Servers in IBM Manufacturing.
The IBM ESS solution stack levels are released as a version, release, and modification level.
The IBM ESS solution stack components are periodically upgraded to newer release levels, tested as an integrated solution, and released as a new level of the IBM ESS solution software. Additionally, generalized recommendations are made available to help clients implement a code update strategy. It is a full-field perspective, and as such, a customized recommendation that accounts for specifics such as business upgrade windows, length of time since last update, and decommission plans. You might require assistance from local support teams. For more information, see IBM Spectrum Scale Software Version Recommendation Preventive Service Planning.
For more information about the release levels of the IBM ESS software solution and the levels of the software components for that IBM ESS release level, see question 2.2 in the
IBM Spectrum Scale RAID FAQ for IBM ESS in IBM Documentation.
2.3.2 Operating system
The IBM ESS solution runs RHEL as the operating system on the IBM Spectrum Scale Data Servers.
Each IBM ESS solution release level integrates and tests a suitable current level of RHEL, including any necessary RHEL fixes and errata that are required for the successful operation of the IBM ESS solution stack. IBM periodically provides new IBM ESS solution release levels that incorporate newer levels of RHEL. These releases are provided often enough to assure that a current level of RHEL is always available.
Consider the following points:
•All generations of IBM ESS can coexist in the same IBM Spectrum Scale cluster.
•Second-generation and subsequent-generation IBM ESS models use RHEL.
Embedded RHEL licensing on IBM ESS 3500, IBM ESS 3200, IBM ESS 3000, and IBM ESS 5000
Starting with third-generation IBM ESS, IBM uses an embedded RHEL license. This license is ordered and included with every generation three+ order.
First- and second-generation IBM ESS Red Hat Enterprise Linux subscriptions
For all first- and second-generation IBM ESS models, Red Hat requires the following separate subscriptions to be in effect for each IBM Power server:
•RHEL Level 3 support subscription
•Red Hat Linux Extended Update Support (EUS) subscription
IBM includes the following required Red Hat subscriptions (with RHEL and EUS feature codes) in every IBM ESS order:
•First-generation IBM ESS models: IBM Program ID 5639-RH7 - Red Hat Enterprise Linux Big Endian
•Second-generation IBM ESS models: IBM Program ID 5639-RLE - Red Hat Enterprise Linux Little Endian
Client responsibilities for RHEL licensing on IBM ESS
For first- and second-generation IBM ESS models, specific customer responsibilities exist for registering the required IBM ESS RHEL subscriptions. For more information about the
IBM ESS customer’s responsibilities, see “How do I register the required Red Hat Enterprise subscription keys for my Elastic Storage Server (IBM ESS)?”
2.3.3 IBM Spectrum Scale
In this section, we describe IBM Spectrum Scale, the high-performance parallel clustered file system software that is used in IBM ESS in enterprise-level, high-performance computing environments, university and research environments, and commercial business environments worldwide.
Formerly known as General Parallel File System (GPFS), IBM Spectrum Scale is highly distributed, clustered file system software that provides high-speed concurrent data access to applications that run on multiple nodes and clusters. In addition to providing parallel high-performance file storage capabilities at petabyte scale, IBM Spectrum Scale provides tools for tiering, management, administration, and archiving of that enterprise-level data.
IBM Spectrum Scale is the IBM strategic SDS for enterprise big data, analytics, and AI applications.
In the most common IBM Spectrum Scale deployment architecture, IBM Spectrum Scale data is accessed by IBM Spectrum Scale clients and users over a LAN network, accessing disk volumes that are known as NSDs that are attached to IBM Spectrum Scale nodes that are known as NSD Data Servers. In this IBM Redpaper publication, these nodes also are referred to as Data Servers.
IBM Spectrum Scale and IBM ESS
IBM Elastic Storage System and IBM Elastic Storage Server are pre-integrated and pre-tested IBM Spectrum Scale storage building blocks.
IBM ESS 3500 uses a pair of x86-based servers, and IBM ESS 5000 uses a pair of POWER9 processor-based servers. These servers run RHEL and IBM Spectrum Scale, and they are defined as IBM Spectrum Scale NSD Data Servers. In all IBM ESS, the NSD Data Servers are cross-connected to provide failover and redundancy. If one of the NSD Data Servers fails, IBM ESS fails over the storage and data to the other NSD Data Server, which ensures continued availability of the data in the IBM Spectrum Scale cluster.
IBM Spectrum Scale for IBM ESS licensing
All nodes in an IBM Spectrum Scale cluster run a copy of the IBM Spectrum Scale software. IBM ESS typically is licensed for IBM Spectrum Scale by using the following specific
IBM Spectrum Scale for IBM ESS IBM program IDs:
•5765-DAE IBM Spectrum Scale for IBM ESS Data Access Edition
•5765-DME IBM Spectrum Scale for IBM ESS Data Management Edition
This specific IBM Program ID with a “Per Disk” metric is normally used for licensing
IBM Spectrum Scale for IBM ESS software on an IBM ESS. IBM Spectrum Scale for IBM ESS software licenses include IBM Spectrum Scale RAID license entitlement. The license price for IBM Spectrum Scale for IBM ESS is tiered as flash and HDDs have different list prices per TB. You count only the physical number of flash drives or HDDs for the “Per Disk” metric.
An advantage of the IBM Spectrum Scale for IBM ESS “Per Disk” metric is that the size of the SSD or HDD does not affect the license list price. For example, if your IBM ESS model has 550 HDDs, your IBM Spectrum Scale for IBM ESS license list price is the same, regardless of whether you are specifying 6 TB HDDs or 16 TB HDDs.
IBM Spectrum Scale for IBM ESS licensing helps to contribute in building a complete IBM hardware and software solution by integrating the IBM Spectrum Scale and IBM ESS solution.
For more information about IBM Spectrum Scale and IBM Spectrum Scale for IBM ESS software licensing, options, and considerations (such as the use of IBM ESS in a Socket-licenses IBM Spectrum Scale cluster, or for IBM Spectrum Scale capacity licenses, or for using IBM ESS in an IBM Spectrum Scale Enterprise License Agreement environment), see the following resources:
2.3.4 IBM Spectrum Scale RAID
IBM Spectrum Scale RAID is an SDS controller that performs all of the storage controller functions that are normally associated with hardware storage controllers. IBM Spectrum Scale RAID integrates all HA and features of an advanced storage server into IBM Spectrum Scale SDS.
IBM Spectrum Scale RAID runs on the IBM ESS NSD Data Servers. IBM Spectrum Scale RAID provides sophisticated data placement and error correction algorithms to deliver high levels of storage reliability, availability, and serviceability, and performance.
IBM Spectrum Scale RAID implements a declustered erasure code parity schema, distributing data, redundancy information, and spare space across all disks of the IBM ESS enclosures. With this approach, a significant improvement is realized on the application performance. Also, storage rebuild time overhead is reduced (disk failure recovery process) compared to conventional RAID controllers.
IBM Spectrum Scale RAID implements large cache for performance by using memory on the IBM ESS NSD Data Servers. The large cache intelligently improves read and write performance, particularly for small block I/O operations.
IBM Spectrum Scale RAID mitigates performance effects of storage rebuilds
If storage failures occur, IBM Spectrum Scale RAID reconstructs lost or erased stripes for I/O operations dynamically. By using the highly distributed erasure coding, IBM Spectrum Scale RAID mitigates the performance effect of storage media failures.
IBM Spectrum Scale RAID end-to-end checksums
IBM Spectrum Scale RAID includes integrated end-to-end checksums that detect data corruption that might otherwise go undetected by a conventional storage controller. Unlike conventional storage controllers, IBM Spectrum Scale RAID is integrated with the
IBM Spectrum Scale file system and performs end-to-end checksum comparison all the way out to the IBM Spectrum Scale client code on the workstations. This feature ensures data integrity at a file system level, detecting and automatically correcting data corruption errors that might occur in conventional storage environments.
In an environment where a customer experienced excessive file system checks and suffered downtime to repair file systems, the use of IBM Spectrum Scale RAID end-to-end checksums mitigates file system check problems. This feature assures availability of data and removes application outages that are caused by file system checks.
IBM Spectrum Scale RAID disk hospital
One of the key features of IBM Spectrum Scale RAID is th disk hospital. This powerful function asynchronously diagnoses errors and faults in the IBM ESS storage media, down to the level of the individual drive and the individual performance of each drive. IBM Spectrum Scale RAID is fully aware of and tracks the performance of each individual drives because all drives do not perform equally. IBM Spectrum Scale RAID uses the individual performance history of each drive to make intelligent data allocation and data retrieval decisions.
Extensive health metrics down to the level of the individual drive are maintained by the disk hospital. Performance variation is continually monitored. If or when a disk metric exceeds a threshold, the storage media is marked for replacement according to the disk maintenance replacement policy for the declustered array.
As an example, disk hospital features the following metrics:
•relativePerformance, which characterizes response times. Values are compared to the average speed. If the metric falls below a specific threshold, the hospital adds “slow” to the pdisk state, and the disk is prepared for replacement.
•dataBadness, which characterizes media errors (hard errors) and checksum errors.
For more information about IBM Spectrum Scale RAID implementation and best practices, see the Administering IBM Spectrum Scale RAID manual for your particular IBM ESS model. at IBM Documentation.
2.3.5 IBM Elastic Storage System solution installation and management scripts
In this section, we provide an overview of the components of the IBM ESS solution installation and management scripts. This overview includes the following information:
•IBM ESS-specific documentation for installation and upgrade scripts
•Ansible playbooks and container-based software installation
•IBM ESS specific tools and utilities for the IBM SSR and administrators to use while installing or maintaining IBM ESS
The third-generation IBM ESS models introduce container-based deployment methods, which use Red Hat Ansible playbooks for installation and orchestration.
For more information about these IBM ESS solution components, see the IBM ESS solution release-specific level information at IBM Documentation.
Installation, upgrade, and administering guides
IBM provides manuals and documentation for deploying and administering IBM ESS, including the following publications:
•Quick Deployment Guide (QDG), which documents IBM ESS-specific scripts for installing, deploying, and upgrading IBM ESS for experienced users.
•IBM Spectrum Scale RAID Administration, which focuses on administering IBM Spectrum Scale RAID on IBM ESS.
•Problem determination guide, which provides more information about monitoring, troubleshooting, and maintenance procedures.
The IBM ESS product documentation is available at IBM Documentation.
Ansible playbooks and container-based software installation
Third-generation IBM ESS models include a software update process and methodology that is based on Ansible playbooks. The Ansible playbooks provide powerful, flexible orchestration tools that drive containerized software deployment methodology and speed and simplify the process of installing IBM ESS.
The POWER9 processor-based IBM EMS is the control management node where the Ansible software and containers are installed. The POWER9 processor-based IBM EMS is used as the single point to perform system management for second- and third-generation IBM ESS models.
On this IBM EMS node, Ansible is configured to store the container and cluster configuration and definitions for all the managed IBM ESS Data Servers. Network services (such as Dynamic Host Configuration Protocol (DHCP) and SSH) are enabled to allow IBM EMS to deploy IBM ESS solution stack components to all IBM ESS nodes in the cluster.
essutils are IBM ESS installation and deployment toolkits that facilitate IBM SSR hardware setup, installation, deployment, and upgrade tasks. Any authorized IBM ESS system administrator also can use these tools.
For more information about essutils and gssutils, see the Quick Deployment Guide, ESS 3500 Hardware Planning and Installation Guide, and ESS software deployment preparation, which are available at IBM Elastic Storage System 3500.
..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset
18.191.61.38