Once the initial foundation for an RPA initiative has been established, it is time to begin work on the first project or series of projects. How that is actually done will depend on many factors in your organization.
RPA fits comfortably with traditional Waterfall methodology and Agile and its numerous hybrids and variations. How you proceed will depend on whether or not yours is an Agile shop or Waterfall.
The business case that was created earlier must be updated. Remember, the preliminary business case was based on limited information. By the time you are ready for development, additional information will have been obtained. This may change the specifics of what’s required, the anticipated savings, the other systems impacted by the proposed change, the tools used, etc. These changes must be incorporated into the updated business case.
Pause and Consider How is this information best gathered? What meetings have been held up to this point that have included information that can be used to refine and update the business case?
A readiness checklist will be required. This is used to assure that all the steps necessary during development to deploy the software to production have been completed. The readiness checklist follows the project through its life cycle, since there are requirements for deployment that must be accomplished at each stage.
- Roles: When work begins on an RPA project, several roles will be required.
Business analyst: This person has been involved from the start, working with the business to fully understand the need and translating that need to requirements.
System analyst: This person has also had some involvement from the start, working with the business analyst to identify the systems impacted by the proposed change. Early in the project life cycle, exactly what systems are impacted may be suspected but not confirmed. During this phase, the exact systems impacted must be identified.
Architect: The architect is also not new to the project, having worked with the business and system analysts to help determine a general timeline for the project and to evaluate the level of complexity. Like the business and system analysts, the architect’s work now becomes more specific, as the “unknowns” from early in the project become clear.
Developer(s): A developer may have had some input early on, but as long as there was an architect engaged, it is unlikely. Now the developers will read and understand the requirements and translate them to code.
Business owner: The business owner is the requirements expert. They know what is needed and assist in creating the iterations of the project.
If Agile (or one of its hybrids) is used, the previous roles will all be involved, but simultaneously rather than consecutively. RPA lends itself very well to Agile, due to the iterative nature of both.
Challenge with Differing Methodologies
RPA teams usually work with more than one business unit at a time, because they are working with different applications which may be owned by different business units. The RPA team cannot change its own methodology for every specific client or utilize multiple methodologies at one time. If there are different methodologies that other business units are working with, and RPA is using, for example, Agile, there are challenges: how can one methodology be used by RPA when the other business units are using a Waterfall methodology?
In the financial sector, for example, many business units are using a Waterfall methodology, while most RPA teams are using Agile or Lean. The business unit is accustomed to signing off on completed requirements; Agile and Lean use an iterative approach, wherein partial requirements are approved and then developed. The RPA team then has to wait until all requirements are approved to begin development. How can the wait time be minimized?
Instead of signing off the entire requirements document, the business must sign off on sections incrementally, noting in the document that the sign-off does not pertain to the entire requirements document, but only to the particular section. Upon full completion of the requirements (note that by this time, most of the development will have been completed), the business can then sign off on the entire document.
This requires training of members of the business unit. The concept of signing off only partial requirements will be new to them, and they may not be comfortable with the idea. They may say that they can’t possibly sign off on “partial” requirements, because they need to see the whole picture before they can approve any part of it. This is, certainly, for some, a new way of doing things. For analysts and for SMEs, they need to think about how the requirements document is divided. They will need to look only at the steps that are being automated, but even then, they may want to see the end-to-end requirements document before approving any one part of it. This is true of deployment as well; the Bot will be deployed incrementally. Having a product RPA incremental development roadmap or plan is critical to define small pieces of RPA. These are building blocks for the final automation. This roadmap might include epics, stories, or other artifacts. Regardless of what they are called, they constitute the step-by-step roadmap for the automation.
Each portion of requirements should be large enough to develop or be a complete functional portion of RPA, for example, accessing the email account to retrieve email content, or accessing the system’s/application front end by logging in under RPA credentials, etc. Each requirement should produce a “minimal viable product,” even if that small “chunk” is never delivered separately to the customer. How can you best work with the business to help them understand this division or “chunking” of requirements?
First iteration of requirements: “Bot will open the email, read only the phone number, and update the phone number in the web application.” The business will be asked to approve this first iteration of the requirements.
Second iteration of requirements: “Bot will open the email, read the phone number and address, and update the phone number and the address in the web application.” The business will be asked to approve this second iteration of the requirements.
Third iteration of requirements: “Bot will open the email, read the phone number and address, and update the phone number and address to the web app and the mainframe.”
In this situation, the first iteration of requirements, once approved, will be developed and tested. User acceptance testing will not occur until all components of the requirements are developed. The assumption here is that there will be incremental development, but only one release when all the products of all the requirements have been developed.
An option is that the product of the first requirements could be delivered and put into production, followed by the second being deployed, etc. This depends on the model that your organization selects.
With each new iteration of requirements, you are increasing the scope of automation. This may be a new concept for the business, which has been accustomed to only approving the entire requirements document.
In some organizations, the deployment team may be using a Waterfall methodology and the RPA team using an Agile methodology.
We will now discuss several common methodologies used with RPA.
Agile Methodology
Many RPA organizations select an Agile methodology or an Agile hybrid. The major aspect of Agile that is often incorporated into a new RPA team is the iterative nature of development, from start to finish.
In an Agile environment, some preliminary, very basic requirements are approved. This is the minimum viable product, meaning that the functionality works, but it may not be anything you’d want to give to a customer; it is too rudimentary. In our case study, a very preliminary requirement might be: “Bot will read the email, read the phone number, and update the phone number in the web application.” The business would approve that, knowing that it would be pointless to give it to the customer, since it is only part of what the customer wanted and was promised. But it is still a basic function that needs to be done. Once that is approved, the development team would begin working on it. The next requirement may be: “Bot will read the email, read the address, and update the address in the web application and the mainframe.” Again, this is only one aspect of what the customer requested and was promised, but absolutely required.
This, obviously, differs from the traditional, Waterfall methodology, in which all requirements are gathered and approved before any other work begins. But since, as has been shown in earlier chapters, RPA is best established following an iterative process (e.g., do some work; present to the Governance Committee; get approval to move ahead [or not]; do some more work; present again; etc.), using an Agile development methodology just makes sense.
One advantage is that changes can be made early in the project life cycle. For example, once the requirement “Bot must be able to read the incoming emails from mailbox XYZ” is approved, the business may decide that not only should address and phone number updates be done by the Bot, but name changes due to marriage, divorce, etc., should also be covered. In a traditional Waterfall methodology, this would be a major requirements change, requiring rework. In an Agile environment, since the requirements are captured and approved incrementally, such changes can be made on the fly.
Remember, the introduction of RPA in your organization can be revolutionary. If the methodology that the RPA team will use differs from what the rest of the organization is using, you will need to negotiate areas of difference. For example, how will you convince the business to sign off on partial requirements?
In comparing RPA initiatives using either an Agile or Waterfall methodology, Madan Kumar Divvela said this: “Typically, process automation with RPA takes a few weeks from inception to production. The traditional Waterfall methodology cannot keep up with the pace of RPA delivery.”1
Benefits: One of the benefits of using an Agile methodology is that changes can be easily made without costly delays and rework. Commenting on his own experience, Dominic Whaley, writing in Partnering Solutions, said that “Dividing the work into distinct packets enabled us to take stock at regular intervals. Changes or new requirements were noted and added to the next sprint.”2
Another benefit of delivering in increments is that lessons learned from the previous increment could be easily incorporated in the next set of requirements without needing to rewrite the whole set requirements and go through the reapproval process again. This way, something you think could have been done better in, for example, the gathering or documentation of the requirements for the first set of requirements can be done in the gathering or documentation of the second set.
This brings increased flexibility and reduces overall lead time of the development cycle.
Also, all the team members are busy all the time. Once the initial requirements have been documented, the developers and business analysts can begin working on the MVP. While this work is being done, the next set of requirements is being captured.
Agile also makes it easier to estimate the time it will take to develop each requirement, which makes scheduling deployment easier and more effective.
Another potential benefit was discussed in Blueprint, in an article entitled “How an Agile Approach to Automation Can Deliver RPA at Scale.”
In an Agile approach to Robotic Process Automation, business processes are designed and optimized before any development begins. This allows large organizations to fully standardize and optimize end-to-end, complex, and multi-layered business processes where the real ROI in automation resides. It also provides them with the invaluable opportunity to consider and tie those processes to larger business objectives and enterprise or regulatory constraints, policies, and controls.3
Possible disadvantages: Working in an Agile environment requires training. There is not only a new way of working and an entirely new vocabulary but also a new way of thinking. Simply changing the vocabulary but using the same methods does not make an organization Agile. It is a characteristic of human nature that people don’t like change, especially if they are very efficient and effective at the way they are currently working. Agile requires people on both the business and technical side to think about requirements, development, etc., very differently. While development teams often quickly get on board with Agile, the business side is frequently much more hesitant.
If your organization is not familiar with Agile, but you feel RPA would best be structured using this methodology, how would you work to convince people that it is? Do you see potential acceptance by the tech team, but resistance by the business? How do you get both groups to see the advantage of using Agile?
This is not meant to imply that only Agile is effective with RPA; that is not the case at all. While many organizations are moving away from the traditional Waterfall methodology, many are not. RPA can be established using this methodology as well.
Waterfall Methodology
Using a Waterfall (requirements, development, test, deployment) methodology with RPA simply replicates how projects are performed in any traditional environment. Once requirements are captured and approved, development begins. The process continues in a very traditional manner.
Let’s look again at our case study, and consider it in the context of a Waterfall methodology within RPA. It would follow the traditional pattern: requirements gathered and approved, followed by development, testing, and deployment. The manual process is still being automated, so the end remains the same; only the means are different. Governance meetings would be held after each phase, to gain approval (or not) to proceed.
Benefits: Most organizations are familiar and comfortable with this very traditional methodology. There are no surprises; it is tried and true (if not the most efficient way of developing projects today). For some organizations, this can be the safest way of introducing RPA to an organization; the introduction of RPA will rock the corporate boat a little; you don’t want to swamp it with an Agile tidal wave!
The Waterfall methodology works well for small and simple RPAs. If the process is very simple and straightforward, with little complexity, etc., the Waterfall methodology process will usually be successful. More complex manual processes are better automated in an Agile environment.
Possible disadvantages: This can be a much slower process. Work does not begin on development until all requirements are gathered and approved; testing does not begin until all development has been done. If, during testing, some error is found, or if the business decides it wants to add a requirement, much rework, with its inevitable accompanying delays, must be done.
It is only fair to note here that most of the literature comparing Agile and Waterfall usage in RPA programs comes down quite heavily on the side of using Agile. While that means you should certainly consider it, there are many factors you need to think about before making the decision for your organization.
Lean Software Development
Lean software development emerged as an alternative to the traditional Waterfall methodology with rigorous, document-driven, and traditional development approaches. Lean thinking principles are based on the Toyota Production System (TPS) and have been successfully applied in many manufacturing and product development organizations. Lean’s philosophy is based on reducing the development time by removing all non-value-adding wastes.
As Taiichi Ohno, one of the main architects of the Toyota Production System (TPS), explained the concept: “All we are doing is looking at the timeline from the moment the customer gives us an order to the point when we collect the cash. And we are reducing the timeline by removing the non-value added wastes.”4
It took Toyota 30 years to refine their methodology and practices to make them effective and efficient and what they are today. However, it did not get mainstream attention and recognition until MIT’s (Massachusetts Institute of Technology) five-year study of the automotive industry identified Lean as a source of huge productivity differences between the United States and Japan.5 The study resulted in defining a new term for production system – “Lean” (because by removing waste, it would require less of everything). The Lean production methodology and thinking is mainly described in the book The Machine That Changed the World by James Womack, Daniel Jones, and Daniel Roos.
During the last few years, Lean has also become popular within the software industry. It mainly originated from the book Lean Software Development: An Agile Toolkit for Software Development Managers by Mary and Tom Poppendieck.6 This book studies Lean production systems from a different perspective and adopts and tailors Lean principles and tools for software development. Its popularity is rapidly growing, and this growth is due to its effectiveness in identifying and eliminating waste and quickly responding to changing customer and market demands.
- 1.
Value is defined by the customer and is what the customer is willing to pay for. It is critical to have a clear understanding of what that is. The product owner can be seen as the “customer” in RPA.
- 2.
Value stream is a visual representation of every action (both value adding and non-value adding) currently required to bring a product or service through the flow from customer request to delivery. The process map, developed early in the life cycle, is a rudimentary but often adequate version of the value stream.
- 3.
Flow represents how people and materials or required items are moving from one step to another until the value is delivered to the customer. It is important to achieve a continuous flow by removing waste from the value stream so development or services can move smoothly toward the customer. That would lead to gains in productivity and efficiency, making processes leaner. That would dramatically reduce time to the customer; instead of taking months to deliver, it could be reduced to weeks or even days. The document that you will create to accompany the process map will capture this information.
- 4.
Pull means customers can order products or services when they need them, and those products or services could be delivered within a short time. It would ensure that nothing is built before it is needed, resulting in lower cost of inventory that potentially won’t be used by the customer; all that effort of creating inventory would be wasted. You determine what the next item to work on is.
- 5.
Perfection is probably the most important concept of Lean thinking. It requires making continuous process improvement as a core part of your organization’s culture. Striving for perfection is done through the repeatable process in which any waste is continuously identified and removed. It is a long journey which requires persistence and constant effort on all levels, from employees, supporting teams, and leaders. As Mark Crawford mentioned: “Lean experts often say that a process is not truly lean until it has been through value-stream mapping at least half a dozen times.”7
Extra features that the customer doesn’t use
Poorly designed functionality that failed testing and needs to be redone
Waiting for requirements or support
Partially completed work
Task switching, working on multiple development initiatives at the same time
Extra processing steps or lines of code
Unutilized or underutilized employee knowledge and creativity
Lean software development can be viewed as the application of the previous concepts and principles to the practice of developing software. Thus, the main aim and objective of Lean software development is to enhance customer value within budget and in the shortest possible delivery time.
Lean was created for manufacturing, but has evolved into a popular methodology for software development. How might it work in your organization? What do you see as the possible advantages? Do you think senior management might be impressed by some of the potential benefits?
Overlap with Agile Methodology
We would argue that Lean and Agile are two names for the same methodology. Looking closely at Agile software development practices, it is not too hard to notice close similarities with Lean concepts.
Code inspections and built-in quality in Agile support one of the cornerstones of what Lean methodology is founded on. The principle is not to push poor-quality to downstream processes; it means to find and fix defects early in the process flow and not to find them when the whole product is ready to be released.
Scrum uses a Lean “pull” principle to balance a workload and to achieve a smooth flow through the system. The team pulls stories from the list in each Sprint (iteration) based on team capacity and capability.
Agile (Scrum), like Lean, focuses on waste elimination by creating processes to identify and eliminate waste from the software development cycle.
Agile uses Kanban techniques to help manage the development of the product and to increase communication between team and stakeholders.
Agile rituals like morning standup, planning session, or retrospective session, all trace their origins to Lean.
We observed one team that faced heavy application-support tasks for multiple internal customers. The team used the Scrum framework for development of new features and solutions, but Kanban was used for tactical ad hoc-based support to fix breaks and address issues with their products. The team started with a simple pencil-and-paper board to track items on the fly and to track new, incoming requests. In this particular situation, there was a need to bring more visibility and workload management to the team. With time, the board evolved, bringing more complexity, with the ability to track the team’s performance and make it easier to identify and quantify the waste, especially in the area of waiting for items to develop and in the area of rework. The board was moved from pencil-paper based to a digital format and then was split into two different functions, one for the development of new products and services and the other to support the existing released solutions. The board evolved together with the team’s growing maturity in the Agile methodology.
There is no doubt that Agile methods and Lean software development go hand in hand in creating value for the customer. However, we believe that there are key differences between Lean and Agile.
Agile seems more tactical in nature where much focus is given to the technical practices, rituals, and techniques for effective and efficient software development.
In our opinion, little is said about what enterprise-wide operating framework and procedures must be in place to make Agile adoption successful. Therefore, to become an Agile enterprise organization, the major changes required must be initiated and driven from the top. H. Smits (2007) states that the “experience gathered during large scale implementation of Agile concepts in software development projects, teaches us that the currently popular Agile software development methods (like Scrum) do not scale to program, product and organization level without change. The fundamentals for changes to these methods are found in Lean principles….”8 At this point, an organizational strategy becomes the main driving force within which Agile processes could operate effectively.
Instead, Lean is applied from an upper management perspective, with the objective of optimizing all aspects of their activities across the entire organization.
From this viewpoint, Lean could be seen as a top-down approach while Agile is bottom up.
Without the strategic piece, Agile adoption is depressed by the organizational forces that are resistant to change. By having Lean introduced from the top down, the tension between developers working in an Agile environment and their stakeholders due to conflict of practices and principles would ease.
We believe that the Agile framework does not generally concern itself with the surrounding business environment in which the software development is taking place. It primarily provides a set of practices designed for use by developers and can be viewed as supportive to Lean methodology. Lean principles, on the other hand, can be applied to any scope, from the specific practice of developing software to the entire organization in which software development is just one small part.9
Poppendieck and Poppendieck (2003, 2006) viewed Lean principles as core truth that do not change over time, while software development tools and practices are the application of those principles to a particular situation. Those tools and practices should be tailored accordingly, depending on the environment, and change as a situation evolves. Consequently, they suggest that Lean thinking should be viewed as guiding principles to develop and adapt Agile practices.10
Starting down the Lean/Agile road can be difficult. We believe it would be very useful to look at developing an adoption or scale-up roadmap for companies to trial Lean and Agile practices within their business environment while minimizing operational risk. Just keep in mind that by creating a Lean/Agile island in the organization would not bring a huge breakthrough. However, it would bring enough evidence to learn while scaling up across the organization. Remember, Toyota spent 30 years to refine their production system; it is a journey that would require persistence and lots of effort.
There is a lot to consider between Agile and Lean and a lot of crossover. How do you think either would fit in your organization? Who in your organization could you talk to about Agile and Lean principles, to get additional input?
DevOps
The term DevOps combines development and operations practices and tools into one framework, which is designed to increase an organization’s ability to define, develop, and deploy IT solutions and services faster than traditional software development methodologies would do.
The core principle of DevOps lies in removing barriers between siloed development and operations teams, to bring their work into one framework to work together throughout the entire IT solution life cycle, from development and testing to deployment and operations. In other words, DevOps brings people, process, and technology together to constantly deliver value to the customer. In that framework, both teams communicate frequently by setting strong cultural ethics around information sharing through the set of chats, recurring design, problem-solving sessions, and huddles. This creates an environment in which communication flows rapidly across all engaged stakeholders such as developers, operations, and supporting teams.
Silos sometimes occur because of individuals or managers “hoarding” information. Tread softly as you work to eliminate them.
To avoid the long solution deployment lead times, developers work on small increments that are released independently of each other. To achieve that, DevOps uses continuous integration (CI) and continuous delivery (CD) pipelines in combination with automation to move code from one step of development and deployment to another. For CI, the key goal is to find and fix bugs as soon as possible to improve quality and time of delivery. The CD practice ensures that workable software can be built in short periods of time and is ready to be reliably released at any time without manual intervention. To deliver software quickly, DevOps tries to rely on automation tools as much as possible. A great level of automation would ensure testing and preparation for release to production with higher speed and frequency. When CD is properly implemented, developers would always have available built, tested, and ready-to-production items.
DevOps infinity loop11
DevOps can perfectly coexist with Agile software development and is a great framework for large IT-driven transformative projects to meet business needs. DevOps can address the challenges that limit business growth, including delays in the release of new software; sometimes the delay is so long that by the time it’s released, it doesn’t meet business needs. In today’s fast-paced environment, where technology changes occur at a rapid pace, an organization whose revenue is heavily dependent on IT capabilities must be flexible and be able to adjust quickly. Time is critical and any delays may shift an organization’s market position from leader to follower in a matter of a few months.
Higher speed of innovation.
Better adaptiveness to constantly changing market needs.
Rapid value delivery to build or maintain a competitive advantage.
Higher product/service reliability, much easier to ensure quality within smaller releases.
Better collaboration between technology and operations; development and operations work as one team by combining their workflows and by sharing accountabilities.
In addition, DevOps becomes popular among Lean and Agile practitioners and continues to evolve as artificial-intelligence-powered solutions come to aid in almost everything we do, from reviewing inquiries, searching for and analyzing data, performing digital tasks, and proposing decisions to incident management. It means smarter solutions would require even shorter development and deployment cycles, with higher solution reliability and built-in quality, and with seamless transition from business needs identification to viable IT solution.
First iteration of development and release: “Bot will read the email, read the phone number, and update the phone number in the web application.” Those steps are going to be automated; the address changes on the web application and mainframe and phone changes on mainframe will still be processed manually. The business will be asked to approve and accept this first iteration.
Second iteration: “Bot will read the email, read the address, and update the address in the web application.” The mainframe phone and address change will still be updated manually. The business will be asked to approve and accept this second iteration.
Third iteration: “Bot will read the email, read the phone, and update the phone number in the mainframe system the address change on mainframe will still be changed manually.”
Fourth iteration: “Bot will read the email, read the phone number and address, and update the phone number and address in the mainframe system.”
With each new iteration, you are increasing the scope of automation. Without lengthy requirement gathering, the steps of approval, RPA development, and testing the solution move from conceptual solution to live solution faster. Those shorter cycles could prevent shifting requirements and solution scope creep, both of which are common when teams are working on complex and large solutions. This incremental and continuous delivery approach may be a new concept for the business, which has been accustomed to only approving the entire requirements document, testing, and accepting end-to-end automation at once.
Think about hybrid versions. For example, RPA may do development, but some other team may gather requirements, do the deployment, etc. Would this be practical in your organization? If it is required, what steps will you take to help ensure success?
Regardless of what methodology you select, the main challenge for RPA is that, most of the time, the teams are new in the organization and usually follow an Agile or Lean methodology. Existing business partners and tech teams may still be in Waterfall mode, or in transformation, and Agile or Lean, with their own vocabulary, practices, rules, guidelines, etc., may be very foreign, and thus intimidating, to them. How do you build bridges between existing methodologies and the new methodologies that RPA will introduce?
We discussed four methodologies earlier in this chapter. Now we need to see how to incorporate the new RPA methodologies into whatever the existing corporate methodologies are.
The RPA team will be discussing Sprints, backlog items, product increments, and other terms that the business may not be at all familiar with.
The RPA team will present the business with a very limited requirements document, the MVP (minimum viable product), as discussed earlier. But the business may be expecting the complete requirements.
The business may allocate a resource to review the requirements a month or two after the project starts, depending on how complex it is, with the belief that it will take that long to document the requirements. Yet that resource will be needed throughout most of the project life cycle.
Let’s consider that RPA teams are more role based than team based. By that, we mean that while there is a core group of developers, a business analyst and a system analyst (other or different roles depending on the organization), there are other roles that will work on specific RPA projects, but not others. For example, in our case study, we would need someone from customer service to be part of the team from start to finish. That person (or persons) would help draft requirements in an interactive manner, and so would be needed throughout most of the project. For a different project, perhaps one that assures that taxes on merchandise adhere to government regulations, no one from customer service would be needed, but someone from the company’s regulatory team would be.
So with this model, the permanent team members (e.g. developer, process analyst) remain on that team; other needed resources are shared as required. After the project completes, the shared resources are released back to their original team.
Shared resources may not be required full time on the team. Determine the need on a project-by-project basis.
One of the potential downfalls is that, after the shared resources return full-time to their original teams, they won’t be available to address issues that arise. You must plan for how you will handle issues that those resources were expert in.
What specific steps do you need to take in your organization to determine the best methodology as you introduce RPA? Who do you need to confer with? Who needs to buy in? How will you get that buy-in?
As indicated, there are a variety of development methodologies for you to choose from. But you needn’t be overwhelmed; review what is current in your organization, determine which methodology you think would be most successful with your new RPA initiative, and work to implement it. But remember, what may appear to have great potential for your organization may wind up having to be changed. That is fine! Ideally, RPA is iterative; you learn as you go along. If something doesn’t work, understand why, and either remove the roadblocks, or go in a different direction.
In the next chapter, we will discuss how RPA is constantly evolving and provide details on the most important trends for you to consider.