Chapter 17. Agile ALM in the Cloud
Cloud-based computing promises, and often delivers, capabilities such as scalable, virtualized enterprise solutions, elastic infrastructures, robust services, and mature platforms. Cloud-based architecture presents the potential of limitless scalability, but it also presents many challenges and risks. The scope of cloud-based computing ranges from development tools to elastic infrastructures that make it possible for developers to use full-size test environments that are both inexpensive and easy to construct and tear down, as required. The first step to harnessing its potential is to understand how application lifecycle management (ALM) functions within the cloud.
Technology professionals often use cloud-based tools at each stage of the development lifecycle to manage the workflow and all of its required tasks. Cloud-based ALM tools commonly include source code management, workflow automation (including defect and task tracking), knowledge management, and community-based forums. Many organizations make good use of the technology by maintaining their own private cloud. Virtualized environments enable continuous delivery and robust testing environments. The main advantage of cloud-based computing is its ability to deliver enterprise architectures at low cost and then scale the architecture as demand increases. Companies that use cloud-based technologies can focus on operating expenditures (OPEX) instead of capital expenditures (CAPEX). Therefore, businesses can keep initial costs low and pay only for actual resources utilized, adjusted as system needs vary. Making the right choices requires that you understand the goals of the ALM in the cloud.
17.1 Goals of ALM in the Cloud
The goal of ALM in the cloud is to establish a comprehensive suite of automated tools to support your entire development effort at very little cost. We discussed automating the ALM using tools in Chapter 7, but this chapter makes the assumption that you have a limited budget, although you may need to scale up in the future. The agile ALM, and especially DevOps, relies heavily upon a robust tools infrastructure to support the entire software and systems lifecycle. We make the assumption that your environment will likely need to scale because it has become standard practice to begin with a small pilot project or proof-of-concept (POC) and add more resources once the project has been approved and funded.
We also have another related goal, which is using our ALM implementation in the cloud to help us learn more about cloud-based development, because we may be supporting the development of software and systems that themselves may very well be in the cloud. If the application being built will eventually run in the cloud, then it is a very good idea to manage its ALM in the cloud using the same service provider and the same infrastructure.
17.2 Why Is ALM in the Cloud Important?
ALM in the cloud is important because it is common to have a limited budget in the beginning of a project. This is especially important for startups, which may have very limited budgets or perhaps are even self-funded. We see many teams adopting low-end tools with limited functionality that cannot scale to meet the full long-term needs of the agile ALM. Starting in the cloud allows you to establish the right tools at a fraction of the cost and then scale up as the project demands. It is important to plan to use the best tools, ones that have the security and features you need to support agility at scale.
There is another reason why using the cloud is very important; it gives the team some practical experience working in the cloud, often with the same tools that they have to deploy when the code is ready for production. The cloud provides a flexible sandbox for development and operations to make the journey together as the agile ALM evolves to the meet the needs of the software and systems development lifecycle.
17.3 Where Do I Start?
It is common for development teams to begin with inexpensive or even completely free tools just to get started. Low-cost (or no-cost) tools are often sufficient in the beginning, and perhaps even complete the POC, thereby securing management approval for the pilot or POC and also your funding. Some teams choose to use cloud-based service providers who offer robust ALM tools as Software as a Service (SaaS) or Platform as a service (PaaS). The SaaS approach means that the service provider handles the software installation, configuration, and maintenance. You generally use the SaaS-based product via a Web browser, perhaps with the use of a browser plugin. The PaaS approach typically involves more of the configuration and administration. There are many agile ALM cloud-based solutions, from version control to project and workflow automation tools. If these solutions do not work for you, you may decide to use Infrastructure as a Service (IaaS), where your cloud-based provider simply provides virtual servers and you handle the software installation, configuration, and administration. Your total cost to get started will likely still be much lower compared with buying machines and having to provide space in a physical datacenter, a practice that is commonly referred to as “on premises” (on-prem).
Many commercial tools vendors offer no-cost “community editions”—robust products that may have some limitations compared with the commercial version yet can still be used by a small team, often up to ten users. This means that your scrum team can get started using products that can scale to thousands of users. We have worked with several teams where we implemented the most robust ALM tool suites using community editions and then efficiently converted up to the commercial versions once the funding and approvals for purchase and implementation had been attained.
We coach teams to avoid starting with tools that cannot scale to full production usage and instead start with the best tools, even if they are only community editions, implemented on cloud-based infrastructures. This approach allows you to start small and then grow to full production usage.
Most teams start by choosing an approach to version control and then a service provider for the infrastructure. Many teams begin with a tool to provide and configure infrastructure, followed by application build, package, and deployment.
17.4 Understanding the Cloud
Cloud-based providers are evolving their services into remarkably powerful (and complex) tools and solutions. Understanding cloud capabilities and then implementing these solutions can be a daunting task, requiring engineers with considerable training, skills, and experience. In many organizations, the DevOps engineer is routinely assumed to be an expert in managing complex cloud-based resources. Actually, although the cloud has enormous potential, DevOps is just as essential on physical servers (or, as many like to say, “bare metal”).
It is common for cloud systems engineers to spend considerable time learning how to work with the complex offerings from cloud vendors, who themselves are getting very good at turning almost anything into a scalable SaaS or PaaS offerings.
We have seen numerous instances where getting into a particular vendor’s offering was initially easy, but then sometimes the service became less than acceptable. It is often difficult to then move to another service provider when poor service actually affects your business. Using DevOps best practices, you should be able to structure your implementation steps so that you can switch providers as necessary. This may require a little extra work up-front to abstract the implementation details, but is well worth the effort to ensure that you can always choose the best cloud service provider or bring your operations back in-house on physical servers should you so choose. In fact, architecting your ALM so that you can pull back out at will is a must, as there are often times when cost, security, or concerns warrant such a move.
17.5 Developing in the Cloud
Software and systems development in the cloud can feel like being a kid in a candy store. There are so many powerful resources from which to choose, and the potential seems to be without limits. Many groups are discovering the value of developing in the cloud, using evolving technologies such as virtualization and container-based deployments. The DevOps approach of requiring that developers work in production-like environments is remarkably feasible when cloud-based resources are available, and the agile ALM is greatly enhanced by this limitless potential. We also feel that DevOps can help ensure you are not locked into the cloud, but retain the ability to bring your environments in-house when necessary, including deployment to physical servers. There is some concern regarding security and reliability in using powerful source code management and workflow automation solutions in the cloud.
17.5.1 Source Code Management in the Cloud
Many popular source code management cloud-based solutions also offer a framework for workflow automation and release coordination. Some of these providers allow you to use their resources and restrict access to only your own internal employees. We have been using a number of these services for quite some time and are impressed with their capabilities, delivered at a low cost. However, we are not without our concerns in terms of reliability and security. If your source code information contains valuable intellectual property (IP), then you need to make an informed decision as to whether or not you feel comfortable trusting a service provider to maintain security, or if you view this responsibility as something that you need to bring in-house. We are not saying that cloud-based providers are more or less secure per se, as some of them have a great deal of experience managing security and a large, experienced staff on-hand dealing with this responsibility on a daily basis. But if you are a contractor working on a healthcare system, you must be aware that outsourcing the management of your ALM in no way outsources your regulatory responsibilities, such as adhering to HIPAA regulations as well as other requirements.
17.5.2 Build Automation in the Cloud
We find that cloud-based build automation is more commonly performed on servers provisioned on elastic infrastructure providers. DevOps absolutely requires fast builds, so virtualization of build machines is an absolute must-have. That said, we currently see most folks managing the complexities of their application build, package, and deployment directly, so although cloud resources may be part of this effort, it is generally handled as infrastructure. We know that cloud-based providers are working on improving SaaS and PaaS build, package, and deployment solutions, and these may very well become more popular in the future.
17.5.3 Release Engineering in the Cloud
Release engineers are benefitting greatly from all that the cloud has to offer. Being able to harness unlimited resources and production-like environments makes our job much easier to accomplish. But there is also some danger out there, and release engineering needs to focus more on creating release packages, whether they be web packages (e.g., JARs, WARs, EARs)1 or simply GNU tar2 or zip3 files, that are cryptographically signed and contain a manifest with a full list of configuration items along with their own MAC-SHA1 or MD5 signatures. Cloud-based development values the procedures inherent in release engineering, but also requires the more advanced security-related techniques that some folks foolishly skip. Your release containers should always be tamper-proof (using cryptography) and be verifiable as coming from the trusted source, in a manner that is called nonrepudiation. Nonrepudiation means that you have signed the tamper-proof container with your private key, which is verifiable with your public key.
1. Java Archive (JAR), Web Archive (WAR), and Enterprise Archive (EAR) are packages used to simplify the deployment process.
2. GNU tarball.
3. ZIP file.
17.5.4 Deployment in the Cloud
Deployment in the cloud is getting interesting. Virtualization and container-based deployments have led many of us to reconsider how we approach the deployment of complex systems. Being able to rapidly build and provision a virtual machine and then deploy applications through a fully automated deployment pipeline have quickly become expected capabilities. The potential of applications running on multiple servers (via a load balancer or proxy) and updated during the working day without any impact to end users is even more exciting. This practical approach to continuous delivery is accomplished by upgrading a few machines and then proceeding on to the rest, if there are no problems, within a specified timeframe. If the upgrade appears to have some technical issue, then the modified servers are taken offline and reverted back to their original state. The idea is that having a couple of servers die is an acceptable price to pay to avoid impact to the larger system, and more importantly, to the customer.
17.6 Change Management in the Cloud
Cloud-based change management solutions help manage the automated workflow required for processing and tracking change requests (CRs). These solutions are often part of a comprehensive IT process automation solution. The change management approach should include support for the change control board (CCB), which manages the review and approval of CRs. The CCB maintains a change calendar to avoid scheduling conflicts, which could result in serious incidents. The change management process should also review and evaluate technical risk. Every change request incurs a risk of the change being made and also a risk of the change not being made. The change management process should review the technical risk of each change and its potential downstream impact. To accomplish this task, the CCB must rely upon subject matter experts (SMEs) who possess deep and thorough knowledge of what might happen if each change is approved. We recommend determining the assets affected by each change request and enlisting the help of the experts who understand the potential impact of each change on the asset itself. It is a best practice to appoint these SMEs to a change advisory board (CAB) to assist the CCB with reviewing and approving change requests.
One key concern is to understand the management of changes made by the service provider. In fact, it is essential to include service provider notifications in your change management function.
17.6.1 Service Provider Notification
Many cloud-based providers do a great job of ensuring reliable service. But sometimes outages cannot be avoided. Third-party notifications must be part of the change management process.
When you decide to use the cloud to manage your ALM, you should ensure that you take a wide and comprehensive view of managing the entire lifecycle.
17.7 Managing the Lifecycle with ALM
Managing the lifecycle with robust ALM tools must take a very broad view. We have discussed many examples of using the cloud to manage the ALM. You may find other aspects of the ALM that need to be handled as well. You should also consider whether or not management of the ALM will eventually be brought in-house and “on-prem,” as is often the case when the ALM has to scale to the enterprise. We will discuss this further in Chapter 19.
17.8 Cloud-based ALM Tools
Cloud-based ALM tools usually include workflow automation; support for agile development; and specialized tools to support application build, package, and deployment. In some cases, you may be able to implement large, comprehensive ALM solutions, such as those that exist for customer relationship management (CRM). In others, you may have to create toolchains to integrate one or more vertical solutions.
17.9 Achieving Seamless Integrations
Using vertical solutions involves tools that perform a specific task well, but that may need to be integrated with other tools. There is often a tradeoff between a comprehensive ALM solution and integrating vertical solutions. We discuss managing tools integration in Chapter 7, but the cloud does bring some unique challenges. For example, trying to integrate different cloud-based solutions may present specific challenges. Cloud-based ALM solutions are especially helpful with iterative development.
17.10 Iterative Development in the Cloud
Iterative development is well suited to the cloud. Supporting the ALM lifecycle may involve a fair amount of effort. Sometimes, the frameworks offered by cloud-based providers are sufficient and can therefore save time and work. But you may find that, over time, you need more robust tools support. Once again, this is exactly where cloud-based development benefits greatly from the elasticity and flexibility of the cloud.
17.10.1 Development Models in SaaS
We have seen some SaaS-based solutions that had established ALM lifecycle processes that were not only available, but actually required. In some ways, these solutions are mature and well supported. However, we have also seen that they are sometimes limited and fall far short of solutions we have implemented. In one instance, we were forced to use the provider’s user acceptance testing (UAT) environment, which did not fully match the production environment, which increased the risk of defects occurring in production that were not successfully identified in UAT. Development models in the SaaS can be quite mature, but you may still find yourself less than satisfied if you have become accustomed to robust ALM lifecycle solutions.
17.11 Interfacing with Your Customers
Keep in mind that your customers will hold you responsible for the success or failure of your application lifecycle methodology. Selecting a cloud-based provider may save you some time, but outsourcing this work does not outsource your responsibility. If your service provider fails to maintain acceptable security standards, you run the risk of failing to adhere to required regulatory compliance requirements. In other words, if you are supposed to adhere to HIPAA and your service provider has an incident that discloses personal identifiable information (PII), then you will still be left holding the bag for the breach caused by the service provider’s failure to secure the environment.
17.11.1 Fronting Service Providers
Many cloud-based providers allow you to brand their solutions with your own skin. Providing a seamless interface is important for many businesses and something that you may be able to implement.
17.12 Managing with SLAs
Service providers do typically offer a service-level agreement (SLA) with their customers, which specifies target uptimes and agreed-upon procedures for resolving issues. It is important to understand the SLA and evaluate whether or not it is acceptable for your business. There is usually a cost for a “premium” SLA, providing better service and response, but, of course, there is usually a cost to systems being down as well. Either way, you are always dependent upon your cloud-based provider, as it is the factor over which you have the least control.
17.12.1 Reliance upon Service Providers
You must rely upon your service provider to maintain an acceptable level of support. This is another reason why some projects start by using cloud-based providers in the beginning, but then choose to bring the ALM tools support in-house using on-premises machines.
17.13 Managing Cloud Risk
The cloud, despite all of its advantages and potential, also has a great deal of risk. As we have said previously, risk is not always bad. You need to understand the risks inherent in cloud-based computing and come up with a plan for communicating and mitigating them. In some cases, these risks are acceptable, or perhaps unavoidable, in the beginning of a project. You may be required to implement your ALM in the cloud with a long-term plan of bringing your ALM back in-house and on-premises as the project approaches completion. One area where the cloud clearly excels is its limitless capacity for robust development and test environments.
17.14 Development and Test Environments for All
Developers benefit greatly from access to robust development and test environments. Unfortunately, it is all too common for developers to be limited to their own laptop and the number of virtual machines (VMs) that they can squeeze onto their own machine. The cloud excels at providing low-cost development and test environments that can significantly improve programmer productivity and especially the quality of the code. With the cloud, developers can test on production-like environments as early and as often as they wish. This approach also implicitly promotes managing infrastructure as code, a key DevOps objective.
We also find that it is a very sound practice to start small and then grow as needed.
17.14.1 Starting Small
The cloud enables one to create environments both large and small. However, we have observed that it is often a wise choice to start with a small footprint and then grow. This approach keeps the work required to stand up the environment to a minimum.
17.15 Environment Management
Environment management is often overlooked, but is actually especially important in the cloud. You should not just rely upon your cloud provider, as they may not have the same sense of urgency for your business operations as you do. We find that most outages are actually related to a lack of good environment management. You should always understand and monitor your runtime dependencies. When you do this well, you will often get notifications in advance of actual outages and, more importantly, you will likely be able to take corrective action before customer impact occurs. One important consideration is ensuring that you have gold copies of all production baselines.
17.15.1 Gold Copies
Your ALM not only should, but in our opinion must, safeguard baselined production releases of your system created through the ALM. You need to have a storage facility, often called the definitive media library (DML), for these production baselines. Any and all changes to the production baselines need to be tracked and verified. This is best accomplished with the support of a CMDB.
17.15.2 CMDB in the Cloud
The configuration management database (CMDB) contains records that describe both the expected and current states of all configuration items (CIs). This information is maintained in a configuration management system (CMS), which is usually a relational database that provides supporting information for the CMDB. You should have automated procedures in place to verify that all configuration items are recorded and any discrepancies identified. Many CMDBs fail because the information cannot be updated in a timely manner using fully automated procedures.
17.16 DevOps in the Cloud
DevOps in the cloud has its own special set of challenges, often due to the dynamic nature of the cloud and the iterative development that accompanies agile development. You can manage these requirements by having a fully automated set of procedures to build your infrastructure and seamlessly deploy your applications. This approach is especially valuable in responding to a cyberattack or other unauthorized changes, which may threaten the reliability of your systems.
17.17 Controlling Costs and Planning
Controlling costs and planning can be especially challenging in cloud-based development. We find that some cloud providers are less than completely clear about what things cost, and their PaaS and SaaS solutions often have hidden costs that may be difficult to predict. We recommend that you invest the time to understand the vendor’s cost model and also set up a credit card that notifies you immediately when charges are incurred. We had one experience where we set up a small environment for testing and then deleted all of the servers and resources, but we kept getting charged small amounts. We contacted the support line, who were also unable to help us figure out what was still causing us to incur charges. Finally, we deleted the account.
17.18 Conclusion
Managing your ALM in the cloud has significant advantages, although not without some challenges, and certainly not without effort. If the application that you are building will eventually run in the cloud, using the same cloud provider to manage your ALM is usually a very wise choice. In some cases, using the cloud is only a temporary solution. In all cases, DevOps can help you manage the ALM for success.