IBM Elastic Storage System planning and integration
This chapter provides guidelines and considerations for the correct planning, installation, and configuration of the IBM Elastic Storage System (IBM ESS).
This chapter also describes configurations and integration considerations for a smooth
IBM ESS deployment into an existing or a new IT environment. In this chapter, we describe the following elements:
•Roles and responsibilities
•Planning
•Networking
•Installation
•Maintaining hardware
•Maintaining software
•Upgrades
This chapter includes the following topics:
3.1 IBM Elastic Storage System planning overview
In this section, we make recommendations and highlight best practices for various roles and responsibilities when an IBM ESS is deployed.
3.1.1 Roles and responsibilities
In this section, we discuss best practices were learned from successful IBM ESS installations regarding who is responsible for each of the areas of expertise.
Overall planning
An experienced project leader should be appointed to coordinate and manage the IBM ESS installation project. Deploying an IBM ESS requires coordination across different IT departments and involves various team members (IBM and client).
The various client and IBM teams that the project manager must coordinate include (but are not limited to):
•Physical planning for the data center, including the networking team
•Networking setup, switch installation, cabling, and verification
•Hardware setup, installation, and verification
•Software planning and integration of the file system into the environment
•Acceptance testing and criteria for moving into production
Networking
IBM ESS components are connected by using IP or InfiniBand networking. IBM ESS data users access IBM ESS by way of high-speed IP or InfiniBand networking.
The networking planning tasks that are required include (but are not limited to) include the following examples:
•Suitable networking switches must be ordered or allocated, including cable planning, ordering, racking, and proper connectors.
•Network addressing must be planned for IBM ESS and the integration of that network into existing client IP or InfiniBand network.
•After the network is designed and approved, IP addresses must be implemented and tested.
•Multiple VLANs must be designed, configured, and tested on the management network switches.
•Suitable networking authentication and firewall rules must be configured and tested to assure that they comply with client network audit and security requirements.
•Testing of the network before IBM ESS is installed (by using tools, such as nsdperf or Network Readiness Tools that are provided by IBM Spectrum Scale as no-charge, open-source tools) must be done to identify and verify the network’s ability to provide
IBM ESS with the required network bandwidth, latency, and consistent performance.
These tasks require the client IBM ESS infrastructure implementation team to complete the following tasks:
•Identify all IP or InfiniBand networking-related requirements.
•Engage early with the infrastructure networking team.
•Document and communicate all required IP or InfiniBand network provisioning, implementation, load-testing, and network implementations, including all QA timeframes, implementation lead times, testing windows, and acceptance criteria.
•Monitor these tasks closely to assure that the networking is in place and load-tested before IBM ESS is installed. Experience shows that networking addressing mismatches, VLAN issues, configuration issues, and a failure to remedy these issues are the major cause of unforeseen delays in IBM ESS installation progress.
The client IP or InfiniBand networking team is responsible for the following tasks:
•Receive the specified networking requirements.
•Consult and collaborate with the IBM ESS implementation team to design, optimize, and agree upon a suitable IP or InfiniBand networking design and implementation plan.
•Implement the agreed-upon networking changes.
•Run requested network load tests by using tools that are provided by IBM Spectrum Scale before IBM ESS is installed, including agreeing to and implementing any necessary quality assurance and acceptance criteria.
•Adjust and tune the network based on pre-installation tests or during the IBM ESS implementation.
The following best practices from successful IBM ESS installations assure success:
•Engage early with the client networking team.
•Obtain collaborative buy-in and well-defined ownership of networking-related task completions.
•Operatively running network performance tests by using IBM Spectrum Scale tools far enough in advance of the IBM ESS installation allows time for necessary tuning adjustments to be made.
•Ongoing IBM ESS implementation team and client networking team dialog and interaction during the IBM ESS installation process quickly remedies any networking-related installation issues.
Installation
In this phase, the following IBM and client teams must be involved and coordinated:
•Physical site administration and engineering
•Networking team and network engineering
•Hardware implementation team, including client IT administrators, IBM System Services Representative (IBM SSR) for IBM ESS physical installation and setup of physical hardware
•Software implementation team, potentially including IBM System Lab Services consultants, and client operating system and software administrators
These teams collaborate to implement the following phases:
•Physical site planning
This planning includes the receipt and staging of IBM ESS components after shipment, physical floor space planning and racking; electrical power planning and provisioning, including specifying suitable electrical power, cables, connectors, and power distribution units (PDUs); and assuring proper airflow and air conditioning.
•Software planning and integration
This planning includes defining the IBM Spectrum Scale file system implementation parameters, requirements, integration testing, and acceptance with the client applications and procedures.
•Hardware physical setup, installation, and verification
This planning and scheduling includes the IBM ESS hardware setup with the IBM SSR and the IT physical installation teams.
•Acceptance testing
Running well-defined criteria for acceptance testing and moving the IBM ESS system into production are featured in this phase.
Maintaining hardware
After the IBM ESS is installed and running in production, many parties are responsible for various IBM ESS hardware tasks, including the following examples:
•Client IBM ESS administrators:
 – Monitoring IBM ESS physical hardware by using provided tools and the GUI
 – Basic hardware problem determination
 – Optional basic maintenance, such as replacing a failed solid-state drive (SSD) or hard disk drive (HDD)
•IBM SSR: Responsible for physical IBM ESS hardware maintenance and repair.
Maintaining software
After the IBM ESS is installed and running in production, many parties are responsible for various IBM ESS software tasks, including the following examples:
•Client Red Hat Linux administrators:
 – Assuring Red Hat Enterprise Linux (RHEL) licensing and subscriptions are in place for first- and second-generation IBM ESS
 – Applying required RHEL security patches as recommended by IBM
•Client IBM Spectrum Scale administrators:
 – Monitoring and tuning IBM Spectrum Scale performance
 – Data management, which includes monitoring data and storage usage patterns, and writing and running policies that manage data, backups, and snapshots
Learning Services and Classes
IBM Learning Services or IBM Business Partners can provide education courses about the customer topics for maintaining IBM Spectrum Scale and IBM ESS hardware. More worldwide resources might be available.
Classes that are available for IBM Spectrum Scale and IBM ESS include (but are not limited to) the following offerings:
For more information, contact your IBM representative or IBM Business Partner.
Upgrades
The IBM ESS provides an integrated software solution stack, which is composed of specific software release levels and the following fix levels:
•RHEL kernel levels and components
•IBM Power server firmware
•NVIDIA network interface card (NIC) drivers
•IBM Spectrum Scale and IBM Spectrum Scale RAID software
 
Note: It is a client responsibility to plan and upgrade their IBM ESS solution software.
IBM recommends upgrading the IBM ESS software solution stack at least once a year and to keep the IBM ESS current on levels of RHEL, firmware, and IBM Spectrum Scale.
IBM System Lab Services is available and it is recommended to engage Lab Services to assist and perform these software upgrade actions on behalf of the client, if required.
Technical and Delivery Assessment and IBM File Object Solution Design Engine
An IBM Technical and Delivery Assessment (TDA) is a technical inspection of a completed solution design. Technical subject matter experts (SMEs) who were not involved in the solution participate to ensure the client’s satisfaction.
In a pre-sales TDA, which is performed before the IBM ESS solution is ordered, the IBM File Object Solution Design Engine (FOS DE) tool and process is used to examine and approve the IBM ESS solution design to determine the following information:
•Will the IBM ESS solution work as expected?
•Is the configuration and implementation plan sound?
•Will the IBM ESS solution meet client requirements and expectations?
In a pre-installation TDA, SMEs also evaluate the client’s environment for readiness to install, implement, and support the proposed solution.
IBM and IBM Business Partner teams are responsible to conduct these pre-sale and pre-installation processes. Contact your IBM representative or IBM Business Partner to schedule and perform these TDA and FOS DE processes.
For more information about the IMPACT and pre-installation TDA tools, which help your IBM account team, client technical specialist, and IBM Business Partner select the best IBM ESS and IBM Spectrum Scale solutions to match your business requirements and needs, see the following web pages:
 
These resources are available to IBMers or Business Partners after logging in by using their IBMid or IBM PartnerWorld® ID.
3.2 IBM Elastic Storage System hardware planning
In this section, we describe the necessary planning and implementation steps that must be followed as part of installing the IBM ESS hardware.
3.2.1 Planning for network switches and networking
Suitable network planning and configuration are one of the most important success factors for an IBM ESS installation. This planning is crucial because the IBM Spectrum Scale and
IBM ESS cluster components are connected by way of an extensive set of IP addresses and associated networking switches and cabling.
The following networks are required to install an IBM ESS:
•Service network
This network connects the flexible service processor (FSP) on the management server and I/O server nodes (with or without the Hardware Management Console (HMC), depending on the IBM ESS generation).
•Management and provisioning network
This network connects the management server to the I/O data server nodes (and HMC if the first-generation IBM ESS machine type 5146 models are used). The management server runs the Dynamic Host Configuration Protocol (DHCP) on the management and provisioning network. If an IBM EMS is not included in the solution order, it is assumed that IBM Power IBM EMS exists in the same IBM Spectrum Scale cluster that can be used.
•Clustering network
This high-speed data network is used for IBM ESS data transfer to and from the rest of the IBM Spectrum Scale cluster and client node access. High-speed networking is preferred (ideally 200 Gb InfiniBand-High Dynamic Range (HDR) or Ethernet), but other selected speeds are supported, such as 100 Gb InfiniBand-HDR or Ethernet.
 
Note: This network must provide consistent, congestion-free network packet delivery with consistent low latency. IBM can supply tools before the installation to test the network. The network might be provided by NVIDIA switches that are ordered through IBM at order time, or the network might be provided by the customer.
For more information about for estimating and sizing suitable network performance, contact IBM or your IBM Business Partner.
•External and campus management network
This public network is used for external and campus management of the IBM EMS, the HMC (if available), or both.
Figure 3-1 shows an example network topology for IBM ESS.
Figure 3-1 IBM ESS 3500 network topology example
An overview of the many networking steps that are required in the installation of an IBM ESS are described next. This example is representative (not all inclusive) of the necessary network planning and implementation steps that must be completed before installing the IBM ESS hardware. For more information about network planning, see the IBM ESS Quick Deployment Guide that is available at IBM Documentation.
Example: Network IP addressing, planning, and implementation for IBM ESS
In this section, we present an example of typical networking-related items for installing an
IBM ESS. The following template can be used for the installation process:
1. Confirm Private IP range for DHCP server.
2. Confirm Private Service network with defined IP addresses and private management network with defined IP addresses. Separate the networks by using switches or VLAN.
3. Confirm Public network connections for IBM EMS. Multiple IP addresses are needed.
4. Select Host → IP mappings to confirm the mappings for the following items:
 – IBM EMS
 – IBM ESS Data Servers: I/O server 1, I/O server 2, I/O server 3, I/O server 4, and so on
 – 200 Gb Ethernet (GbE), 100 GbE, 40 GbE, 25 GbE, 10 GbE, or InfiniBand switches
5. Set up domain names for management private network.
6. Set up domain names for high-speed interconnect.
7. Set up Data I/O partition and partition profile names.
8. Confirm Server names.
9. Confirm that the 10 GbE, 25 GbE, 40 GbE, 100 GbE, 200 GbE, or InfiniBand switches are in place and cabled.
10. Set up Bonding if it is used.
11. Set up Public network in place and cabled to EMS.
12. Confirm that all building block components are correctly cabled in the rack (Data I/O servers, EMS, and switches).
13. Set up and confirm dual feed power to racked IBM ESS components.
14. Set up the IBM EMS console or terminal, including the network connections.
15. Prepare for and install RHEL errata.
16. Register Red Hat subscription keys for all IBM ESS servers, as needed (if you have first or second-generation IBM ESS).
17. Define and specify IBM Spectrum Scale installation parameters and requirements. For example, how many file systems, what are the block sizes, how do we allocate and split metadata, and what replication factor is used.
18. Confirm all physical NVMe, SSD, or HDD disks are in place. IBM ESS installation checks with scripts.
19. Confirm that all physical cabling in place. IBM ESS installations are checked by scripts.
20. Confirm that wifi access exists in lab to set up Webex (for access to and by support teams).
This template shows the processes that IBM Systems Lab Services can perform as part of a contract to install an IBM ESS System. As part of that contract, IBM Systems Lab Services provides and works with the customer on a detailed IP networking planning document.
3.2.2 Planning for servers that are used in IBM Elastic Storage System
In this section, we review planning information for the physical servers that are used in an
IBM ESS. We also describe specific recommendations for memory, NICs, and cabling and connectors.
IBM EMS
Each IBM Spectrum Scale cluster that uses an IBM ESS requires at least one IBM EMS. The EMS acts as the central management focal point for all IBM ESS systems that are in the cluster; runs specific functions, such as the IBM ESS GUI; and stores and distributes the master boot image for all the IBM ESS systems that are in the cluster by using the xCAT software (first- and second-generation IBM ESS) or by using Red Hat Ansible playbooks (third-generation IBM ESS 3500, IBM ESS 3200, IBM ESS 3000, and IBM ESS 5000).
For first-generation IBM ESS (machine type 5146), an IBM POWER8 processor-based machine type 8247-21L server is used as the EMS. This EMS runs RHEL Big Endian. Only first-generation IBM ESS are supported by the 8247-21L EMS.
For the second-generation IBM ESS, a POWER8 processor-based server with machine type 5148-21L is used as the EMS. This EMS runs RHEL Little Endian. In an IBM Spectrum Scale cluster, a 5148-21L EMS can simultaneously support second-generation IBM ESS and
IBM ESS 3000.
For the third-generation IBM ESS (IBM ESS 3500, IBM ESS 5000, IBM ESS 3200, and
IBM ESS 3000), an IBM POWER9 processor-based server with machine type 5105-22E is used as the EMS. This EMS runs RHEL Little Endian. If the IBM ESS 3000 is deployed into a cluster that features a 5148-21L EMS, the IBM ESS 3000 can be supported by that 5148-21L EMS.
Third-generation IBM ESS introduces an EMS software deployment mechanism that is based on containers and Red Hat Ansible orchestration playbooks.
An IBM Spectrum Scale cluster can simultaneously support all three generations of IBM ESS.
All first-generation IBM ESS are supported by an POWER8 processor-based 8247-21L
IBM EMS running RHEL Big Endian. All second-generation and IBM ESS 3000 are supported by an POWER8 processor-based 5148-21L IBM EMS running RHEL.
All third-generation IBM ESS are supported by IBM POWER9 processor-based 5105-22E IBM EMS running RHEL Little Endian. Announced on 23 February 2021, this POWER9 processor-based 5105-22E IBM EMS can also support second-generation IBM ESS.
IBM ESS Data Servers
Every IBM ESS is an integrated IBM Spectrum Scale storage building block that consists of two IBM Spectrum Scale Data Servers, and each data server is cross-connected to all the storage for high availability (HA). In normal operation, half of the IBM ESS storage is running through the first data server, and the other half of the IBM ESS storage is running through the second data server. If a planned or unplanned outage occurs on one of the data servers, the storage is failed over to the remaining data server in the IBM ESS, which provides HA and resiliency.
Each first-generation IBM ESS (machine type 5146) storage building block uses two POWER8 processor-based servers (machine type 8247-22L) each acting as an
IBM Spectrum Scale Data Server. These 8247-22Ls run RHEL Big Endian. Only first-generation IBM ESS use the 8247-22L RHEL Big Endian-based data servers.
In the second-generation IBM ESS, two POWER8 processor-based servers with machine type 5148-22L are used as the IBM ESS Data Servers. These servers are physically identical to 8247-22L servers, but carry an IBM ESS-specific machine type of 5147 to assure that
IBM Service and Support always know that these servers are part of an overall IBM ESS integrated solution. These 5148-22L POWER8 processor-based servers run RHEL Little Endian.
An IBM Spectrum Scale cluster fully supports a mix of all generations of IBM ESS in the same cluster. A 5148-21L EMS can simultaneously support second-generation IBM ESS and
IBM ESS 3000. A 5148-21L EMS cannot support a first-generation IBM ESS, IBM ESS 3500, IBM ESS 3200, or IBM ESS 5000.
For the third-generation IBM ESS 3500, IBM ESS 3200, and IBM ESS 3000, the 2U24 form factor uses an integrated pair of x86-based servers as IBM Spectrum Scale Data Servers.
For the third-generation IBM ESS 5000, two POWER9 processor-based servers with machine type 5105-22E are used as the IBM ESS Data Servers. These servers are specific POWER9 processor-based servers for IBM ESS 5000. They carry an IBM ESS specific machine type of 5105-22E to assure that IBM Service and Support always know that these servers are part of an overall IBM ESS-integrated solution. These 5105-22E POWER9 processor-based servers run RHEL Little Endian.
An IBM Spectrum Scale cluster can simultaneously support all three generations of IBM ESS. First-generation IBM ESS in the cluster are supported by an 8247-21L IBM EMS running RHEL Big Endian.
All second-generation and IBM ESS 3000 systems in the cluster are supportable by the 5148-21L IBM EMS or 5105-22E IBM EMS, both of which run RHEL Little Endian.
All third-generation IBM ESS 3500, IBM ESS 3200, and IBM ESS 5000 require a 5105-22E POWER9 processor-based IBM EMS that runs RHEL Little Endian. Announced on
23 February 2021, the POWER9 processor-based 5105-22E IBM EMS also supports second-generation IBM ESS.
Memory
POWER9 processor-based DIMM memory cards (specified with POWER9 processor-based server memory feature codes) are used in the 5105-22E servers. The default memory chip is a 32 GB DIMM.
The default memory size for the 5105-22E when it is used as a data server is 384 GB, which consists of twelve 32 GB DIMMs. When a 5105-22E is used as an IBM EMS, the default memory is 128 GB, which consists of eight 16 GB DIMMs.
When 5105-22E is used as IBM ESS Protocol node with single socket, the default memory is 128 GB, consisting of 16 GB DIMMs with quantity of 8. When 5105-22E is used as IBM ESS Protocol node with two sockets, the default memory is 192 GB, consisting of 16 GB DIMMs with quantity of 12. This memory is enough for most IBM ESS installations.
POWER8 processor-based DIMM memory cards (specified with POWER8 processor-based server memory feature codes) are used in the 8247-21L, 8247-22L, 5148-21L, and 5148-22L servers. The memory size for the 8247-21L and the 5148-21L EMS was originally 32 GB.
IBM ESS 5148-21L IBM EMS default memory is now 64 GB.
The IBM EMS memory size is enough for most IBM ESS installations. If many IBM ESS systems are used in your IBM Spectrum Scale IBM ESS configuration, contact your IBM representative to see whether larger IBM EMS memory sizes might be required for your installation. More EMS memory can be specified at order time or added later as a field miscellaneous equipment specification (MES).
The standard memory for the IBM ESS Data Servers is enough for most IBM ESS usage cases. If you have a large workload configuration, or you have a specific workloads type, larger memory might be needed on your IBM ESS Data Servers. Check with your IBM representative to see whether you have specific workloads that might benefit from more main memory.
Standard IBM Power memory upgrades for IBM ESS Data Servers can be ordered and installed without disrupting the access to the IBM Spectrum Scale file system. In the installation process, the storage in one of the two IBM ESS Data Servers that are upgraded is failed over to the other IBM ESS Data Server.
This procedure allows the first IBM ESS Data Server to be gracefully brought down and then upgraded. Then, the same process is repeated for the other IBM ESS Data Server to be upgraded.
 
Note: Both data servers must have the same amount of memory in each IBM ESS building block. When possible, as a best practice, all IBM ESS systems in a specific IBM Spectrum Scale storage pool should have the same size and speed. If this configuration is not possible, keep the IBM ESS similar in size and speed. IBM Spectrum Scale parallelism means that in a specific storage pool, the speed of data reads/writes is gated by the slowest storage building block.
IBM ESS 3000 uses a pair of x86-based servers that are integrated into the back of the 2U24 storage enclosure. The standard memory for each of these IBM ESS 3000 Data Servers is 384 GB. This memory is enough for most of IBM ESS 3000 usage cases. You can specify another 384 GB of main storage per data server at order time (or later by field upgrade), which results in a total of 768 GB of memory per IBM ESS 3000 Data Server.
IBM ESS 3200 uses a pair of PCI Gen4 x86-based servers that are integrated into the back of the 2U24 storage enclosure. The standard memory for each of these IBM ESS 3200 Data Servers is 512 GB. This memory is enough for most of IBM ESS 3200 usage cases. As of this writing, no option is available to specify any extra main memory per IBM ESS 3200 Data Server.
Network interface cards
The IBM ESS features the following types of networks that are used to interconnect the
IBM ESS components:
•Management and service
Mandatory, separate 1 GbE network for management and service traffic.
•Data
A high-speed network for the user data being written into and read out of the
IBM Spectrum Scale and IBM ESS file system.
Standard NVIDIA server NICs are used to provide high-speed network connectivity in an
IBM ESS. The network cards that are available vary depending on the IBM ESS generation.
Make sure that the network bandwidth requirements are clearly studied, documented, and communicated. Specify a suitable number of NICs to meet expected IBM ESS performance requirements. Plan and specify the correct cables and connector types.
 
Note: IBM Power IBM ESS Data Servers do not support every possible IBM Power PCI adapter.
Cabling and connectors
In addition to specifying the correct number of NICs to provide bandwidth requirements, make sure that physical planning for cable lengths, adapter connector types, and switch connector types, are clearly studied, documented, and communicated.
Physical machine room placement of IBM ESS components relative to the physical location of the IP network switches must be clearly understood and documented. Physical cable orders, including wanted colors, connector types, and required cable routing must be studied, specified, and verified with network teams, server teams, physical site teams, and infrastructure teams.
Many network connector types cable types are available. Requirements must be identified for the required networking components, including the following examples:
•Transceivers
•Optical versus copper
•Connector form factor
•Cable lengths
3.2.3 Planning for storage configuration
IBM ESS storage configuration for ordering purposes is relatively straightforward.
First, determine your IBM Spectrum Scale storage requirements. Request that your IBM representative or IBM Business Partner use the FOS DE tool to help identify usable capacity requirements and to estimate wanted performance.
After storage capacity and performance requirements are known, identify which models of IBM ESS feature the suitable type and quantity of storage media available. Within each model, you choose the suitable drive size. Request that your IBM Representative or
IBM Business Partner use the FOS DE tool to verify the performance of your chosen
IBM ESS model with your requirements.
Part of the value of IBM ESS is that after storage performance and capacity requirements are known, IBM ESS models provide known performance and usable capacity characteristics because these factors were considered when IBM designed the IBM ESS model.
Each IBM ESS model includes predefined storage configurations and requirements, which are pre-tested and supported as a solution by IBM. Cable lengths, installation parameters, adapters, drive preparation, and location, all are pre-specified within the IBM ESS architecture. This situation demonstrates the value of the use of IBM ESS as an optimum and tested method to deploy IBM Spectrum Scale storage by using an efficient building block approach.
3.2.4 Planning for IBM Spectrum Scale protocol nodes
First announced in 2014, IBM Spectrum Scale protocol nodes provide external Server Message Block (SMB), Network File System (NFS), object, and Hadoop access to
IBM Spectrum Scale data.
From a planning perspective, IBM Spectrum Scale requires that all protocol nodes in an
IBM Spectrum Scale clusters are all x86, IBM Power Little Endian, or IBM Power Big Endian.
No protocol node intermixing of these nodes is allowed in the same IBM Spectrum Scale cluster.
IBM ESS coexists in an IBM Spectrum Scale cluster with protocol nodes. IBM ESS also has no effect on which type of protocol node can be used in an IBM Spectrum Scale cluster.
With the IBM ESS 3000, IBM ESS 3200, and IBM ESS 5000, IBM provided the option of ordering IBM Spectrum Scale protocol nodes based on the POWER9 processor-based 5105-22E server. These protocol nodes are supported by the IBM EMS and can be managed by the IBM ESS GUI and by IBM ESS installation tools, which provides greater ease of use and management capability for ordering a complete IBM Spectrum Scale or IBM ESS or protocol node solution from IBM.
The 5105-22E IBM Spectrum Scale protocol nodes can coexist with any other IBM Power Little Endian protocol nodes in the same IBM Spectrum Scale cluster.
For more information about 5105-22E protocol nodes, see Quick Deployment Guide, which is available at IBM Documentation.
3.2.5 Planning for physical, electrical, and cooling installation
Preparation for installation of your IBM ESS system is as important as the physical installation of the system. To have a smooth and successful installation, preparation processes, and TDAs must be followed.
FOS DE and pre-installation meetings with IBM TSS and IBM Service personnel must be completed. Also, IBM Elastic Storage Server (IBM ESS) environmental specifications provide detailed information about your system, including dimensions, electrical, power, temperature, environmental requirements, and noise emissions. By following the preparation process, IBM service personnel can efficiently install the physical hardware and also deployment of the software.
The specifications for IBM ESS can be found in IBM Documentation.
3.3 IBM Elastic Storage System software planning
In this section, we highlight key items to be aware of for planning the installation of the
IBM ESS software stack; in particular, IBM Spectrum Scale and IBM Spectrum Scale RAID.
3.3.1 Planning for IBM Spectrum Scale file system
The IBM Spectrum Scale file systems that are placed into the IBM ESS must be thoroughly planned, documented, and communicated. Make sure that user application, authentication, firewall, and backup and restore requirements are understood and included.
Before the arrival and installation, the IBM Documentation manual for IBM Spectrum Scale and IBM Spectrum Scale RAID Administration should be consulted. These manuals contain detailed instructions and best practices recommendations for implementing the requirements of the client’s environment.
3.3.2 Planning for IBM Spectrum Scale RAID
The IBM Spectrum Scale RAID Administration manual provides best practices and recommendations for setting up IBM Spectrum Scale RAID. Consult this manual and follow the recommendations.
The IBM Elastic Storage System and IBM Elastic Storage Server implementation of
IBM Spectrum Scale RAID uses just a bunch of disks (JBOD) arrays, provides the required redundancy protection, and delivers usable disk capacity, required spare capacity, and maintenance strategy.
In IBM ESS, the IBM Spectrum Scale RAID implementation best practices are enforced as de facto standards by the IBM ESS installation scripts, which provide configuration parameters that can achieve the best storage performance.
Each JBOD array is connected to each of the two Network Shared Disk (NSD) Data Servers in the IBM ESS to protect against server failure. Each NSD Data Server has two independent paths to each physical disk to protect against path failure and provide higher throughput to the individual disks.
3.4 Ordering IBM Elastic Storage System
IBM ESS is always part of a larger IBM Spectrum Scale system. Because IBM Spectrum Scale and IBM ESS are sophisticated technologies to assure high client satisfaction, IBM requires that a TDA and the FOS DE tools are used to elicit requirements, assure that designs are reviewed, assessed, and approved.
The IBM ESS solution process engages a pre-sale and a pre-delivery TDA. The purpose of these two assessments is to match the client’s requirements and needs. It also evaluates the client’s current IT environment to create the best proposed IBM ESS solution that matches the client’s needs.
Before confirming the IBM ESS order, the IBM team or the Business Partner uses the
IBM FOS DE tool. FOS DE is a question and answer tool that elicits and documents the various technical requirements for a solution that uses IBM ESS. Contact your IBM representative to request that a pre-order IBM ESS FOS DE design evaluation is performed.
IBM eConfig is used by the IBM or Business Partner team to configure the IBM ESS for ordering. eConfig is a tool that helps the IBM or Business Partner team to configure the correct combination of IBM Storage Enclosures, servers, NICs, and associated feature codes to make up the wanted IBM ESS.
 
Note: An approved pre-installation TDA and an approved FOS DE run must be performed by the IBM account team or IBM Business Partner to have an IBM ESS order to be entered and shipped.
As part of the order, various decisions regarding hardware must be made. Be prepared to supply requirement information to the IBM Business Partner. The following requirements often are included in the information that is sent:
•Storage requirements (total capacity, HDD or SSD sizes, and wanted performance)
•Server memory and NIC requirements
•Wanted network cables, cable lengths, connector types, and network switches
•Physical electrical power requirements (phase and voltage)
•Any other more requirements (Your IBM or Business Partner representative works with you to determine these requirements.)
IBM or the Business Partner then configures the IBM ESS. IBM ESS components can be installed in an IBM rack (preferred) or installed in customer-supplied racks.
Figure 3-2 on page 55 shows an example of some possible IBM ESS configurations in one rack. Shown on the left in Figure 3-2 on page 55 are the size of various IBM ESS components in rack units. Shown on the right of Figure 3-2 on page 55 are examples of four IBM ESS SL models. These configurations are examples only; your configuration is tailored to your requirements.
Figure 3-2 IBM Elastic Storage Server physical racking examples
If an IBM ESS 5000 is upgraded with more storage enclosures, it is not necessary to reposition the POWER9 processor-based 5105-22E Data Servers. IBM installs the additional storage enclosures above the POWER9 processor-based data servers in the rack.
3.5 IBM Elastic Storage System installation
In this section, we describe the process that is used to physically install the IBM ESS and deploy it into the client’s IBM Spectrum Scale environment.
The installation process starts with the client preparing their environment to be ready for the IBM ESS. After the client’s environment is prepared, the IBM expert service personnel install the hardware. Depending on the solution, the deployment of IBM ESS also is performed by IBM specialized service personnel.
3.5.1 Physical installation, racking, and cabling
IBM Technical Support Services (TSS) should be engaged to help you plan for the physical installation and racking of the IBM ESS components.
IBM TSS works with you to help you plan for the many items to consider for racking, installation, and servicing of the IBM ESS. Your TSS representative can help you plan for many factors, including the following examples:
•Electrical power, power connectors, PDU, and amperage
•Network adapter and connectors
•Electrical power cables, network cables, and cable lengths
•Airflow considerations (especially IBM ESS SCx models), which use the deep
IBM 5147-106 Storage Enclosures
•Shipping location of the system
•Location of systems that are installed in customer racks (if rackless; that is, not installed in IBM rack)
•IP addresses, networking, and switch configuration that is needed to install and connect the IBM ESS components
•Lift tools
•Ladders
•IBM SSR physical installation instructions (WCII) and IBM SSR scheduling and confirmation
If you have an IBM ESS HDD model, allow time for the physical installation of the many HDDs into the storage enclosures. HDD IBM ESS models are shipped with the storage enclosures empty to mitigate shipping damage. Each HDD is shipped in a sealed protected package. In a large IBM ESS, many HDDs are used and it takes time for the IBM SSR to install each HDD into its slot in the storage enclosures.
When are you planning for physical installation, remember to consider for future expansion. Some factors include the following examples:
•Whether to reserve space in a rack for future expansion
•Location of rack in data center for purposes of future cable lengths and electrical power supply expansion
•Location of network switches and the IBM EMS for purposes of cable lengths for networking
3.5.2 Integrating into client environment
Integrating the IBM ESS into the clients’ environment consists of many actions. In this section, we provide an overview of key best practices, practical advice, and experiences and lessons learned for a successful IBM ESS deployment.
Integrating into customer networking environment
In any IBM Spectrum Scale or IBM ESS solution, the network interconnect always plays a central role. The goal of this section is to guide you in lessons that are learned from an
IBM Spectrum Scale or IBM ESS network readiness discussion.
Networking is a vast expertise area, and you should not expect to need to become a networking expert. The following questions guide you through initial discussions you must have with your networking infrastructure team to assure that your IP network is ready for your IBM Spectrum Scale or IBM ESS solution to integrate IBM ESS into your environment.
Will a new or existing network be used
Document if you are installing IBM Spectrum Scale or IBM ESS on an existing network or if this installation is on a new network that is designed specifically for IBM Spectrum Scale or IBM ESS.
IBM Spectrum Scale or IBM ESS is a high-performance, high-data rate big data solution. Best practices include providing the IBM Spectrum Scale or IBM ESS cluster with a dedicated network infrastructure that is co-designed with the solution.
However, especially in a commercial environment, many (if not most) circumstances are such that you cannot implement a dedicated network for IBM Spectrum Scale or IBM ESS. Instead, the IBM Spectrum Scale or IBM ESS traffic must coexist on an IP network. Especially in this case, consider the following points to validate if the network can provide the high bandwidth and consistent low latency IP networking that IBM Spectrum Scale or IBM ESS solution requires:
•IP Network vendors and switch models.
Document and provide a list of the IP network switch vendors and switch models that are to be used to provide the network infrastructure for this IBM Spectrum Scale or IBM ESS cluster. IBM Spectrum Scale or IBM ESS depends upon IP networking for all of the
IBM Spectrum Scale nodes and users to connect and communicate with each other.
NVIDIA is the preferred high-performance IP networking switch vendor for IBM.
•Provide a network topology diagram.
Provide a copy of the network topology and document where the IBM ESS will be placed and how they will be connected.
•Total number of nodes in IBM Spectrum Scale or IBM ESS cluster.
Document the number and type of IBM Spectrum Scale and IBM ESS nodes that are planned for your solution. Provide the number of protocol nodes and the number of
IBM Spectrum Scale NSD Data Servers.
Are there sufficient port counts and port speeds that are available for the network switches
Document the number of ports and port speeds that are required by your IBM ESS and assure that number of ports and speeds are available within the networking infrastructure.
What are the interconnects between switches
Document the inter-switch links, where they are located, and what traffic traverses them.
Can existing networking links and switches be used
IBM Spectrum Scale or IBM ESS can generate unusually high workload on a network. If
IBM Spectrum Scale or IBM ESS is to be placed on an existing network, document how heavily used the network is, and what the anticipated usage will be after IBM ESS is added.
If it is necessary to validate whether the client's network can sustain the intended
IBM Spectrum Scale or IBM ESS workload, use the IBM network performance tool nsdperf (written by IBM Spectrum Scale Development). The nsdperf tool mimics IBM Spectrum Scale workloads and provides an open-source, no-charge method to test the network before installation.
For more information, see this GitHub repository.
What other network traffic types and amounts must coexist on these network links
Document what other traffic is to compete for bandwidth on the IP network. IBM Spectrum Scale depends on low latency, consistent delivery of many small metadata messages to work properly. If any congestion exists in the IP network, IBM Spectrum Scale can encounter problems. Therefore, the best practice is to have a dedicated network infrastructure for the IBM Spectrum Scale or IBM ESS, if possible.
Communicate this need for low latency and consistent, congestion-free delivery of
IBM Spectrum Scale metadata messages to the network infrastructure team.
What are the basic workload types
Document what other traffic IBM Spectrum Scale is to compete with for network bandwidth. Identify usage cases that are to be used, looking to check the workload mix (small block IOPs versus large block sequential, for example). Engage a qualified IBM Spectrum Scale architect with networking skills to project how the IBM Spectrum Scale or IBM ESS network infrastructure performs in your specific environment.
Sustained bandwidth required in GBps
Determine performance expectations. In addition to documenting GBps sequential streaming workload requirements, what other specific workloads are to be running? What tools or criteria are to be used to quantify that performance? What specific criteria determine “good” or “not so good” performance?
Is validation required to confirm that the existing network can provide necessary GBps and IOPS
If it is deemed necessary to examine the network further and validate that it is ready for an IBM Spectrum Scale or IBM ESS solution after reviewing these questions and answers, use the nsdperf tool. For more information about this tool, see this GitHub repository.
Integrating into a customer software environment
IBM Spectrum Scale and IBM ESS provide various file or object interfaces to the user applications. Determine, inventory, and document the user applications and their required interfaces that the applications require. From that information, design the file systems, directory structures, and data architecture that must be supported by the IBM Spectrum Scale or IBM ESS solution.
Design the IBM Spectrum Scale or IBM ESS cluster to deploy the wanted POSIX, Linux, or UNIX file systems. If necessary, design and deploy the IBM Spectrum Scale protocol nodes and their associated shares, mount points, and IP addresses that the software applications require.
Integrating into a customer security and authentication environment
IBM ESS is a part of an IBM Spectrum Scale cluster. IBM ESS and IBM Spectrum Scale support security and authentication by using various interfaces, including LDAP and Active Directory. Determine, document, and inventory the required security, authentication, and access control requirements. Then, use that information to design the IBM Spectrum Scale or IBM ESS solution.
For more information about IBM Spectrum Scale authentication and security, see IBM Spectrum Scale Security, REDP-5426.
3.5.3 Reliability, availability, and serviceability
IBM ESS uses servers and design that have a rich tradition of reliability, availability, and serviceability (RAS).
IBM ESS’s reliability (and the availability of the applications it supports) is a function of much more than just the reliability of the processors, or even of the entire system hardware. A full description of a system design for RAS must include all of the hardware, firmware, operating system, applications, operating environment, duty cycle, and so on.
Reliability in hardware is all about how often a hardware fault requires a system to be serviced; the less frequent the failures, the greater the reliability.
Availability is how infrequently such a failure affects the operation of the system or application. For high levels of availability, correct system operation must not be adversely affected by hardware faults. That is, a highly available (HA) system design ensures that most hardware failures do not result in an application outage.
Serviceability is concerned with identifying what fails and ensuring an efficient repair (of that component, firmware, or software).
IBM Call Home: IBM ESS hardware
The IBM EMS and I/O server hardware includes IBM Call Home capability.
In the first-generation IBM ESS, the Hardware Management Console (HMC) provided the
Call Home focal point for service receiving, logging, tracking system errors, and, if enabled, forwarding problem reports to IBM Service for hardware events.
When a serviceable event occurs on one of the monitored components (such as servers or storage), the Call Home generates a Call Home event.
In the second and subsequent generations of IBM ESS, the HMC is no longer required.
IBM Call Home: IBM Spectrum Scale software
IBM Spectrum Scale software also has its own separate Call Home capability for purposes of automatically collecting and securely uploading data that is related to IBM Spectrum Scale to IBM Service and Support. These functions are separate from the IBM ESS Call Home functions.
3.6 Monitoring IBM ESS
Monitoring the IBM ESS system includes system health, performance, and capacity monitoring. You can monitor the system through IBM ESS GUI or with the help of CLI.
For more information, search for the topic “Monitoring IBM Spectrum Scale RAID”
at IBM Documentation.
3.6.1 GUI
The IBM ESS GUI provides robust facilities to monitor and manage the system.
IBM Documentation provides more information about how you can configure the IBM ESS GUI to monitor the performance of the following functional areas in the system:
•Network
•System resources
•IBM Spectrum Scale RAID
•NSD server
•IBM Spectrum Scale client
•NFS
•SMB
•Object
•Clustered Trivial Data Base (CTDB), which is used for SMB
•Transparent cloud tiering
•Waiters
•Active File Management (AFM)
For more information, see Monitoring and Managing the IBM Elastic Storage Server Using the GUI, REDP-5471.
3.7 Maintaining and repairing IBM ESS
For IBM ESS features, a full library of documentation for maintaining the hardware and software of the IBM ESS solution is available at IBM Documentation. At the website, select the IBM ESS solution release level that you are using.
3.7.1 IBM ESS IBM Solution service and support
IBM provides service and support for IBM ESS as a solution. IBM ESS-specific keywords and procedures are in place so that you, as a client, when calling or electronically requesting
IBM ESS support, are routed to an IBM ESS cross-component solution support team that understands the interaction of all of the IBM ESS components. They also can diagnose and respond to your query in a holistic fashion, and where necessary, direct your query to suitable Level 2 or Level 3 resources.
3.8 Upgrading IBM ESS
Maintaining and upgrading your IBM ESS software solution stack is a customer’s responsibility.
You also can choose to engage IBM System Lab Services or an IBM Business Partner to perform the upgrades for you.
The IBM ESS solution is an integrated package of hardware, firmware, and software. All components of an IBM ESS are integrated, tested, and supported by IBM as a solution. IBM periodically provides a refreshed or up-leveled IBM ESS software solution stack.
Existing IBM ESS customers are entitled to download the newer IBM ESS solution stack from IBM Fix Central. IBM recommends that you plan to upgrade your IBM ESS solution stack at least once a year.
For hardware upgrades, you should contact and work with your IBM or Business Partner specialist to plan and run the IBM ESS upgrades when necessary.
3.8.1 IBM ESS hardware and firmware
IBM ESS hardware and firmware updates are included in the IBM ESS solution stack. Upgrading to a new level of IBM ESS solution stack also upgrades the server and storage firmware levels, if necessary.
Typically, the IBM ESS upgrades are done non-disruptively. During a planned maintenance window, fail over the storage to one of the I/O data servers, upgrade the I/O data server, and then, fail over the storage to this newly upgraded server and upgrade the other server.
Storage enclosure firmware can typically be applied non-disruptively.
IBM ESS also supports nondisruptive addition of more storage enclosures, which allows an IBM ESS to be non-disruptively upgraded to a larger capacity. For example, with IBM ESS 5000, this process supports the addition of a 5147-092 to an SLx model or a 5147-106 to a SCx model. Therefore, an installed SC1 to SC8 model, or SL1 to SL6 model, can be upgraded to any model configuration up to SC9 or SL7 respectively, without causing downtime or an interruption in service.
An IBM ESS 3000 or IBM ESS 3200 that was ordered half-populated with 12 NVM drives, can be upgraded to fully populated 24-drive IBM ESS 3000 in a similar manner, by adding the NVMe drives in the front of the IBM ESS 3000 Storage Enclosure. No downtime or interruption in service is required.
The client is responsible for keeping the IBM ESS operating system, software, firmware, and GUI current with the recommended levels of software. For more information about upgrade instructions, see IBM Documentation.
 
Note: IBM and IBM Business Partners might offer IBM Services® to assist or perform these upgrades on behalf of the client.
For all hardware upgrades, clients work with the IBM or their Business Partner to match their needs with the IBM ESS upgrades and services offerings that are available.
3.8.2 IBM ESS software
IBM ESS provides a fully integrated and tested solution stack of software that consists of the following components at a specific release and fix level (including all prerequisites) that were fully integrated and tested:
•RHEL
•IBM Spectrum Scale
•Cluster export services code (CTDB, Samba for SMB, Ganesha for NFS, and OpenStack Object Storage (Swift))
•xCAT (eXtreme Cloud Administration Tool) (if you use first- or second-generation
IBM ESS)
•IBM Power or x86 server firmware
•RHEL kernel at specific level (for example: 3.10.0-957.21.3)
•RHEL system errata at specific level (for example: 219-62.el7_6.6)
•RHEL Network Manager at specific level (for example: 1.12.0-10.el7_6)
•RHEL device drivers (IPR)
•NVIDIA OpenFabrics Enterprise Distribution (OFED) network adapter drivers
•Electronic Service Agent (ESA)
•Hardware Management Console (HMC) if this installation is first-generation IBM ESS
For IBM ESS 3000, IBM ESS 3200, and IBM ESS 5000, Red Hat Ansible playbooks and software is used to manage and provide significantly improved ease of use for upgrading IBM ESS software components
Keeping the IBM ESS software solution stack current is a customer responsibility. IBM recommends that clients plan to upgrade their IBM ESS software solution stack at least once a year. Typically, the IBM ESS hardware and software solution stack upgrades are performed at the same time.
For more information about the levels of the software components for each IBM ESS software solution release level, see IBM Documentation.
Upgrading the IBM ESS solution software is done by a full replacement of the software solution stack with a newer level.
New levels of the IBM ESS solution software stack are available at IBM Fix Central. For more information about these newer levels of IBM ESS software, log on to IBM Fix Central by using your IBMid. At the website, in the Product Selector field, enter IBM Elastic Storage System. Then, select your IBM ESS version or release level, your platform (for the latest IBM ESS, this platform often is Linux on IBM Power PC 64 Little Endian).
A list of available IBM ESS solution software fix packs for download is shown. When you start the download, IBM FixCentral checks your IBMid authentication profile to assure that you set up your IBMid with the suitable authentication and client customer number information. This information authenticates that you are authorized to download IBM ESS solution software.
 
Note: If you confirmed that you are authorized to download fix packs but receive a message that says, “No applicable support agreement found”, this message indicates that your IBMid profile does not match the IBM entitlement database. In this case, click Request Help in the download window. An IBM Help ticket is opened and IBM contacts you to help resolve the matter.
For more information about applying the IBM ESS Fix Pack, see the IBM ESS Quick Deployment Guide.
IBM System Lab Services is available to perform the IBM ESS software upgrade for you. As a best practice, take advantage of these services. For more information, contact your IBM representative.
3.9 IBM ESS hints and tips
Consider the following hints and tips:
•Physically plan ahead and think through the following issues:
 – Where the IBM ESS servers and components are to go in the rack.
 – Where to place the network switch and the server locations into the rack’s position.
 – Cabling and type of adapter connectors. Always follow the recommendations from IBM.
•IBM recommends that you use defaulted tuning parameters, unless your IBM or
IBM Business Partner SME reviewed your environment and recommended other suitable choices.
•Always test and qualify your IP network before installing IBM ESS. IBM Spectrum Scale and IBM ESS can achieve high performance and use all available network bandwidth. Tune your network specifically for optimum IBM Spectrum Scale and IBM ESS performance.
•Have all networks planned, implemented, and tested before IBM ESS hardware is installed. For example, identify and implement network configuration, VLANs, IP addresses, and authentication and security passwords. Make sure that your IP network environment meets IBM’s IBM ESS network criteria and is verified by an IBM SME.
•Define clear IBM ESS performance criteria expectations and test plans to verify performance.
•Use only IBM ESS and IBM Spectrum Scale certified machines and components in your IBM ESS hardware and software environments.
•Ensure that every IBM ESS cluster includes at least one IBM EMS and one management network switch.
•Plan your education and skill sets for IBM Spectrum Scale and IBM ESS so that you can understand and use this technology to its fullest capabilities.
•IBM recommends that each IBM ESS building block has the same IBM ESS code level. For migration or upgrade purposes, adhere to requirements for IBM ESS code levels that are allowed to coexist during the migration or upgrade within each cluster. For example, ensure that a code level must be no more than two different OFED levels apart.
•Ensure that each IBM ESS building block is integrated into the same rack; it cannot be split into multiple racks.
..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset
13.58.212.170