Break glass is an information technology term used to describe the solving of a catastrophic problem as if metaphorically smashing the glass of a fire alarm and instantly getting help. In the case of privileged password management solutions, it refers to retrieving sensitive credentials by a human identity when an emergency situation arises, and traditional access methods have failed. In other words, you need a special privileged credential to restore operations and there is no way to retrieve it due to some catastrophic event or outage. A break glass scenario, therefore, bypasses standard operating proceeds and access controls and should only be allowed during the most extreme situations. The method of getting these credentials can vary based on the outage and business ramifications of allowing a user out-of-band privileged access.
As a matter of process, a user performs a break glass checkout or reset of credentials when he or she needs immediate access, even if the user is not authorized to manage the system. This method is customarily used for the highest-level system accounts, such as root accounts for Unix and Linux, SYS or SA for a database, or administrator for Windows (local or domain). These highly privileged accounts are not usually assigned to specific identities, so instead, break glass restricts their access with various controls to limit retrieval. However, it is obvious that unrestricted user access to break glass credentials could create an unacceptable security risk if not implemented correctly.
Break glass scenarios can be caused by network outage, application fault, or natural disaster that disrupts the normal availability of your privileged access management solution. Therefore, factors like a backup power source and network redundancy should be considered when designing your break glass policy and real-world implementation. Also, a threat actor may consider your break glass process a target since it does contain credentials that can be leveraged in an attack. To that end, access and monitoring of credentials used in break glass processes should also be considered in your design.
- 1.
An emergency situation when direct access to a managed system is not possible and a break glass credential is retrieved as the enabler.
- 2.
Getting access outside of the standard operating process because mission-critical systems are down, or a required approver is unavailable. The goal is service recovery in as short of time as possible.
- 3.
Retrieving passwords or secrets from a physical safe or other offline backup on a physical device, such as USB drive, or other physically secured removable media.
Break Glass Process
For authorized break glass users (new or existing), consider creating pre-staged emergency user accounts that are managed and distributed in a way that can make them quickly available. This should occur without administrative delay, but have the appropriate restrictions from a threat actor. The break glass accounts and distribution procedures should be documented and tested as part of any implementation and carefully managed to provide timely access when needed. These can be stored in a highly available password manager or a secure physical location and have physical counterparts stored on other media and in a highly secure environment (physical fireproof safe).
To comply with auditing requirements, even if an approval is bypassed, the system should still fully log who has access and what actions were performed. Additionally, IT administrators should review the logs to ensure compliance with change management processes when a break glass process is used.
Break glass processes that are implemented outside of the password management technology, such as a physical safe and storage of printed passwords, should be routinely updated and manually tested for effectiveness and change control. Only select users should have access to the combination or keys to the physical safe, and they should be treated like any other sensitive information within the organization.
Break Glass Using a Password Manager
Information technology (IT) organizations often utilize a password manager as a break glass solution to provide access to their environment when the established processes for logon or authentication fail. IT teams might typically authenticate against LDAP or AD, and then the user would execute sudo or a least privilege solution to gain managed administrative privileges. When this method fails, the break glass process would require IT to provide a password for an account within established parameters (timeframe, privileges, scope, etc.) to access the application or system.
During normal operation, users who need access to privileged passwords will access a password manager to retrieve a password or establish a session so that they can perform whatever tasks or operations are assigned to their roles. This requires that the password management solution have the rights to fully manage, rotate, and secure the current password for the target resource. Relying on end users to diligently remember, rotate, and securely document all their passwords is invariably less reliable and riskier, especially if any one of them is deemed available for break glass.
- 1.The person who needs a managed password cannot log in to the solution.
- a.
Repair user access to the password manager.
- b.
Reset the managed credentials.
- c.
Reset the password for the user accessing the solution.
- a.
- 2.Fault authenticating to the password management solution.
- a.
Repair network connectivity for critical paths.
- b.
Restore password management connectivity to critical authentication services.
- c.
Repair authentication system.
- d.
Store a printed-out copy of the passwords in a highly secure location.
- a.
- 3.The password management solution is not available.
- a.
Repair network connectivity.
- b.
Access solution through fault-tolerant node.
- c.
Repair password management solution.
- d.
Retrieve passwords through a password cache.
- a.
- 4.Managed passwords are invalid.
- a.
Refresh the password by using the solution to generate a new one automatically.
- b.
Use the password history feature of the password manager to determine the last valid password.
- a.
- 5.Catastrophic, but selective, connectivity anomaly.
- a.
When critical services are not functioning, access may be required via iDRAC, management networks, or crash carts.
- b.
When network connectivity does not allow access, lateral connectivity, not subject to segmentation, can provide break glass access.
- a.
- 6.Processes and workflow prevent access.
- a.
No approver is available in the time period required.
- b.
User access is restricted due to system ownership, such as employee role, contractor, or vendor.
- c.
Time-of-day constraints or critical event requires immediate unrestricted access.
- a.
Session Management
- 1.Controlling third-party access to managed systems.
- a.
Open alternate access into the environment via backup connections.
- b.
Disable session management access to the primary systems (not recommended).
- a.
- 2.Access session management in an alternate data center.
- a.
Open network path around the session management device (not recommended).
- b.
Access session management device in an alternate data center or disaster recovery environment.
- c.
Operate session management independently for management networks to provide access.
- a.
Stale Passwords
There are many situations where a password stored in the password manager may be stale through no fault of the technology. Such cases could arise due to restoration of backup images, rollback of virtual snapshots, or even the deployment of a new instance or system based on a template. In these use cases, the break glass password manager has automated the rotation of passwords of human, service, or built-in accounts throughout the environment.
Consequently, no one knows the correct password, and the password is not written down for manual retrieval. During normal operation, password managers will randomize and change the passwords, update managed systems, and store and test the new password. This is why the conflict exists.
- 1.If the tool cannot change a single or small number of passwords:
- a.
Repair connectivity or verify the configuration of the system to make password changes based on the uniqueness of the targets.
- b.
Manually change the password using another account that has appropriate privileges. Most password management tools have an account assigned to perform such operational tasks, typically called the “functional account.” This will be discussed further in Chapter 24.
- a.
- 2.If the tool cannot change any passwords:
- a.
Repair network connectivity or system access.
- b.
Verify functional accounts have proper privileges to manage passwords remotely.
- c.
Verify supporting services from AD, NTP, DNS, and others are all working correctly.
- a.
- 3.If the password of a built-in account is not known:
- a.
Randomize the password of the built-in account using the functional account.
- b.
Repair system by booting to single-user mode and change password using a crash cart or similar via a known privileged account.
- a.
- 4.If the password of a service account is not known, so a service will no longer start:
- a.
Randomize the password of the service using the functional account.
- b.
Establish a privileged connection to the system using a stored credential and manually set the service account password before automating password management.
- a.
Application-to-Application Passwords
- 1.If automation jobs or applications develop a fault:
- a.
Repair the password management solution.
- b.
Enable fault tolerance for the API.
- c.
Add caching to the scripts, configuration, or application to be fault-tolerant for a network, connectivity, or password management outage.
- d.
Manually or automatically cycle the resources ensuring that all dependencies have been met for the retrieval of a credential (bounce the box or service).
- e.
Implement automatic retries for the application or job to refresh the cache with current credentials.
- a.
- 2.If automation jobs or applications require change control for password changes:
- a.
Schedule password changes during maintenance windows.
- b.
Develop applications that are fault-tolerant or can be resumed in the event of an API query failure for any reason.
- a.
Physical Storage
For any break glass plan, your environment should strongly consider a recovery policy that includes the ultimate break glass solution—retrieving physical copies of passwords. There are inherent risks with storing physical copies of privileged passwords. However, with the proper physical controls in place to securely store the credentials, physical storage of paper can serve as one of the best options in break glass scenarios.
Create a plain text copy of the credentials and automatically print them in a secure location or store them on reliable removable media. Regardless of the format type, ensure that final storage is highly secure.
If your processes require, re-encrypt the digital media with an offline encryption package before writing to a USB drive or CD. Remember to back up the password for the offline encryption in a secure location as well. Typically, this should be another physical safe with a different combination. This creates a form of two-factor authentication to protect the offline versions.
Fully document the process for creating and storing break glass passwords. Passwords should be rotated and tested periodically.
Finally, as with any disaster recovery process, the paper or removable media process must be implemented following any regulatory compliance mandates that oversee your organization.
Context-Aware
Who is trying to log on?
What system are they trying to access?
Where are they logging in from?
What day of the week is it?
What is the time of day?
Applying context allows you to incorporate privileged access management best practices to better protect your organization from a breach. For example, if your break glass account is strictly for emergency use and never used for anything else, only make it available during off-work hours. If it is expected that the account would be accessed via a remote employee working from home, verify that the request is coming in via an authorized secure remote access solution. Therefore, always apply context to any break glass request.
Architecture
If any component of a break glass process or password management system itself becomes unavailable (natural disaster or outage), multiple levels of redundancy mitigate the risk of data loss or degradation of access capabilities. Flexible high-availability deployment architectures ensure that passwords remain available whether everything is installed in a single data center, across multiple geographic locations, or hosted in the cloud. This is traditionally the top priority of an architecture and defense before utilizing a break glass process. Physical copies of credentials should also be considered for disaster recovery locations, but the architecture for any PAM solution should consider relying on break glass only when absolutely needed.
Finally, for short-term outages (planned or uncontrollable) of the entire PAM infrastructure, passwords may be stored and retrieved via cloud PAM solutions. These would need to be configured to cache or replicate the information off-premise and secured against external threats, but this is a viable architectural deployment model to ensure maximum availability.
Break Glass Recovery
What event occurred requiring the break glass process?
Can this event be avoided in the future?
Was the access to break glass credentials appropriate?
Were there any resources in the break glass process that did not have coverage?
Who was notified of the execution of the break glass process?
Did the process introduce any additional risk (data loss, resource exposure, etc.)?
Was the restoration process of services accurate after a break glass event? If not, how can it be improved or fixed?
Were all electronic credentials and passwords reset or rotated after the break glass event?
Was all physical storage of credentials reinstated and codes to physical storage reset?
Was all break glass session activity verified and audited for inappropriate activity?
Were break glass credentials fully locked down again after the incident?
If break glass scenarios repeatedly occur, then the entire process should be evaluated to prevent their invocation in the first place. This could be anything from faulty hardware, network anomalies, to the unavailability of key personnel in a critical-need situation. The restoration of normal services should always include the complete postmortem of the break glass event.
Break glass scenarios should be considered for any sensitive privileged account, even in the event of the stakeholder’s death. Using the technology to support itself and physical access as a backup ensures that the controls recommended do not become a liability to the organization and a gold mine for a threat actor.