ALM Assessments
Often in our work we need to perform an assessment of a given situation. It could be describing a system’s present architecture and then coming up with a plan for improving it, or it could be assessing how a system scales. When we start working with people instead of technology, the focus of the assessments is on a different level: suddenly there is a need to consider human factors and not only technological topics. Before TFS entered the scene, we conducted surveys of how a department functioned, for instance. We did such assessments by interviewing people from the organizations and thereby getting a picture of the situation.
Interviews are complex and result in a lot of information that you need to process afterward. Mostly we have prepared by writing questions on a form used for the interviews. Any follow-up questions that came up were carefully documented and included in later assessments. So this form started to become our tool, even though we did not have it in an application. Instead, we used a Word document that was updated and printed for each interview.
However, we have been a bit reluctant to use only tools. We thought about digitizing the questions into a web application or the like and letting the subjects answer the questions themselves, but we didn’t want to abandon the interview part. Tools can help, but they can also hinder because you can become too dependent on them. Another aspect of using tools for this purpose is that if we let a single person answer our questions in, let’s say a web form, we wouldn’t be around to ask the follow-up questions. These questions enable us to learn so much more than we would if we only looked at the answers to the original bunch of questions.
We started working with TFS a few years back when the product was new. We felt that it was a good foundation for taking control of an ALM process. It lacked (and still does in some cases) some of the support we wanted, but it was a good start, and nothing else was offered on the Microsoft platform to compete with it. The more we dived into it, the more we started thinking of how we best could evaluate an organization to implement TFS at their site. We realized very quickly that ALM was an important part of an organization’s ability to improve their software development cycle, and TFS was an excellent tool to help our customers. We could see that we had tools for visibility, traceability, and automation of high-level processes that fit nicely into the ALM concept.
Microsoft released their ALM assessments on the Web some years ago (www.microsoft.com/almassessment), including the Application Platform Optimization (APO) model. Microsoft provided its APO model to help IT organizations understand and adopt a more flexible and agile application platform. These felt like a good start, so we set off to figure out how to best use these tools to help our customers implement TFS. Previously, we found that many of our customers used only a fragment of the true potential of TFS (most often the version-control system). If we could show them why they should use more features, they would get so much more out of it and at the same time be more successful in running their projects.
Microsoft’s assessments were tools pretty much like web questionnaires, which is why we set off to work on creating an assessment based on Microsoft’s Application Platform Capability Assessments, but mixed with an interview part as well. The reason for this is that a tool cannot read between the lines. It cannot hear what a person says apart from what is actually answered. It is important to have the capability to ask follow-up questions, to clarify and discuss where needed.
One great advantage of the Microsoft assessments is that after everything is entered into the tool, the tool can handle much of the data processing automatically. This decreases the labor attached to the manual processing of the large amount of data an interview gives you. We “only” need to make sure that we enter the most realistic values into the system, and that’s where the interview part can help us out. By using the questions from the online assessment as a form for our interviews, we have a good foundation for interviews. Then we can use the results of the interviews, including follow-up questions and observations, to enter data into the assessments form, giving us the capability to reflect on each question and choose the best answer for the organization we are working with.
Microsoft Application Platform Optimization (APO) Model
APO is part of Microsoft’s Dynamic IT initiative that also includes the Infrastructure Optimization model and the Business Productivity Infrastructure model. There are four primary goals of the Dynamic IT initiative:
- Manage complexity and achieve agility
- Protect information and control access
- Advance the business via IT solutions
- Amplify the impact of your people
These models are aimed at helping customers better understand the current IT capabilities in their organizations and based on these results take the capabilities to a higher level of maturity.
In this section, we’ll give you a brief overview of the Infrastructure Optimization model and the Business Productivity Infrastructure model before focusing on APO.
Infrastructure Optimization Model
With the Infrastructure Optimization model, Microsoft focuses on four areas: the desktop infrastructure, the server infrastructure, the remote infrastructure (which covers how to handle remote offices or locations), and virtualization. Based on best practices internal to Microsoft as well as on feedback from customers, Microsoft has provided an approach they say will do three things:
- Control costs by looking over hardware, utilities, and space expenses in the data center to see where we can reduce costs. The costs also can be controlled or reduced by optimizing deployment testing and training expenses as well as reducing security breaches (in addition to other strategies not covered in this book).
- Improve service levels. This can be done, for example, by reducing service interruptions caused by security breaches, having a robust disaster recovery strategy, and avoiding desktop configuration conflicts.
- Drive agility. Here we find topics familiar to us by now, such as increasing the ability to adapt to business changes.
Business Productivity Infrastructure Model
Microsoft provides a definition for what optimizing our Business Productivity Infrastructure means (http://microsoftio.partnersalesresources.com/bpio.aspx). Microsoft defines it as follows:
“The Business Productivity Infrastructure Optimization (IO) model includes a complete set of technologies that helps streamline the management and control of content, data, and processes across all areas of your business. It helps simplify how people work together, makes processes and content management more efficient, and improves the quality of business insight while enabling IT to increase responsiveness and have a strategic impact on the business.
The Business Productivity IO Model defines five capabilities that are required to build a more agile infrastructure”:
- Collaboration
- Unified Communications
- Enterprise Content Management
- Reporting & Analysis
- Content Creation
Let’s now move our focus to the APO model. Before we explain what this really is, we will spend some time with the maturity levels Microsoft has identified for the assessment of the APO model. There are four optimization levels (see Figure 4-1).
Figure 4-1. The four optimization levels for categorizing an IT organization (as stated by Microsoft)
Basic
When a company is classified as a basic organization, it is characterized by brittle, disconnected applications and platforms. This fact hinders rapid adjustments to business changes and also hinders the rapid development and interoperability of business-critical applications. The organization makes no real use of business processes, or these processes (if they exist) are often ill-defined. The processes are definitely not automated in any way. Such an organization probably has no tool for collaboration between teams and team members, and definitely lacks the clear connection between IT and business that is crucial for a company to have.
The development process is probably quite rigid, which makes development hard to control. All in all, this leads to higher costs, application backlogs, and lower IT productivity. The IT department is probably seen as just a cost to management and its true potential as a strategic asset is clouded by all problems.
Standardized
The standardized organization has begun to use industry standards broadly across departments, as well as with business partners. These standards could be as simple as starting to use XML, for instance. Furthermore, such an organization has started to take control of their development and data infrastructure, enabling the use of business intelligence reports and analytics. They have also started to automate some of their business processes. The IT department has slowly begun to be seen as a business enabler that could provide help in building more-adaptive systems quickly.
Advanced
At an advanced level, IT is truly seen as a business enabler and partner. Now infrastructure and systems are more easily managed throughout their lifecycles. The business processes are well-defined and well-known. The business side has begun to truly take advantage of IT and can rely on the IT department to quickly make adjustments when changes in the business occur. Such a company has standardized a robust and flexible application platform for the most critical systems and processes.
Dynamic
A dynamic organization is fully aware of the strategic value of its infrastructure. The organization knows IT can help run the business efficiently and stay ahead of market competitors. Costs are controlled to the company’s maximum ability. All processes are automated and integrated into the technology, enabling IT to adjust to business changes and needs. The collaboration between the business and the IT department is working smoothly, as well. It is also possible to measure the effects of business benefits of IT investments, which is a strong argument when showing the value of IT. This kind of organization has also used SOA to the fullest so that cost-effective systems can be developed.
APO Capabilities
With this background, we are ready to have a look at the capabilities included in the APO model. Microsoft defines five capabilities, as shown in Figure 4-2.
Figure 4-2. Microsoft has defined five capabilities for their APO model (https://partner.microsoft.com/40029360)
User Experience
User experience (UX) is an unappreciated area. UX is important, but most often this capability is not included in projects as a special field. We far too often rely on developers to solve UX problems without thinking of the effects bad UX design could have. Not many developers are skilled in this area and the importance and value of this field is included in this capability. Usability should be a higher priority in most cases. We have seen projects that were considered failures because the user experience was too technical. The developers had a technical view on how to work in the application (or system) and had designed it with this as the primary viewpoint. The design was not in line with how the end user really worked, so the user interface needed a lot of rework, resulting in higher costs and a delayed project.
Business Intelligence
Microsoft also identifies business intelligence (BI) as a capability. Microsoft and many others have a vision that business insight should be provided to all employees. This leads to faster, more reliable, and more relevant decisions in the organization, because all members of the organization have access to the right information to support good decision making. We find areas such as data mining, reporting, data warehousing, data integration, analysis, and more here.
SOA and business process is another capability. SOA can be a great thing to implement in our organizations, as you might have heard over the years. But SOA in our opinion has turned a little cold lately, and we don’t hear much about it. This capability focuses on the integration between Business Process Management (BPM) and SOA.
This is an immature market according to some surveys as only around 30 percent of respondents said they had a combined strategy for SOA and BPM. A significant two-thirds of the organizations had no such strategy, in other words. This might be good for us as consultants because the market exists for helping out, but could be disastrous for some companies if they don’t change this situation.
Having effective business processes that we are able to quickly adjust to new or changed business needs is essential for an organization these days. We need ways to manage our processes and then automate them in our IT infrastructure. BPM will help with managing the processes, and SOA will help with implementing them in our IT environment.
Data Management
Data management covers what an organization should consider when integrating data management and analysis software. How is the data storage handled? Will it support the business-critical systems reliably? This capability also covers how database development is being carried out, how well the database team is integrated into the development projects, and so on. The main focus is to determine how best to build an integrated, well-managed, and always-connected data infrastructure to support our most demanding and mission-critical applications.
Development
Let’s look at the development capability. Here we find the things that can enable an organization to develop applications that connect business processes to meet business needs. It covers areas such as what kind of development platform the organization uses, whether a development process is in place, how the development team and projects are organized, how visibility into the process of a development project is going, and so on.
Application Platform Capability Assessment
Next we will take a look at what an assessment can look like.
This assessment is called the Application Platform Capability Assessment and exists in three versions:
- Application Lifecycle Management
- Business Intelligence
- SOA and Business Processes
We will use Application Lifecycle Management for our discussion because it is the most relevant for covering the development process and the ALM process. It covers all aspects of the ALM process and is very extensive. So in order to get good coverage on what parts of the ALM process you can improve, this is the assessment you should use.
When starting this assessment you can see that we have two options:
We will use the Team Assessment. One big difference with this assessment compared to the Individual Assessment, aside from the number of questions and the detail level in them, is that it is intended to be filled out by more than one person. Microsoft also encourages the use of a partner when gathering information about your organization. The best thing about using a partner for such an assignment is that you get an independent view on the answers and the state of the organization.
The ALM Assessment for Teams includes many areas; it has eight practice areas, all divided further into a various number of practices. The assessment has about 200 questions (this figure is subject to change), so it covers a great deal of material.
The following list shows the eight practice areas and their practices:
- Architecture framework
- Analysis and design
- Database modeling
- Elicitation
- Requirements analysis
- Requirements management
- Traceability
- UX research
- UI design and prototyping
- UI implementation
- End-user documentation
- Code writing
- Code analysis
- Code reuse
- Code reviews
- Quality metrics
- Database development
- Collaborative development
- Database change management
- Version-control repository
- Release management
- Build management
- Change management
- IT governance maturity
- Application portfolio management
- Compliance management
- Designed for operations
- Deployment
- Environment management
- Operations
- Customer support
- Database deployment
- Project initiation
- Project planning
- Project monitoring and control
- Risk management
- Stakeholder management
- Project close
- Test resource management
- Test planning
- Test management
- Test types
- Database testing
Starting the Assessment
When you start an assessment, you begin by filling in some information about the company. You can see in Figure 4-3 that you also set a time frame indicating the period that you’ll allow people to add information into the assessment. You also can fill out the name of the partner you work with during the process.
Figure 4-3. Starting an Application Platform Capability Assessment as an owner
The creator, or owner, of the assessment sends an e-mail to all contributors containing the URL of the assessment. Once a participant opens the URL, they are welcomed with a page as seen in Figure 4-4.
Figure 4-4. Starting an Application Platform Capability Assessment as a contributor
When you enter the assessment, you’ll have one page for each of the practice areas. As you know, these areas are divided into practices, and these are displayed as sections on each practice area page. Each section displays the questions for each practice (see Figure 4-5). Some practices have only one question, and others have more, so variance is great.
Figure 4-5. Answering an Application Platform Capability Assessment as a contributor
So what kinds of questions can you expect from this assessment? Some are detailed next so you can get an idea of how the practices are examined. Just as you would expect, the questions are more detailed in each area compared to the shorter APO assessment. Let’s take a look at two examples.
Example 1: Assessing Requirements Handling
This first example question asks about how you handle requirements in the organization. It tries to find out whether you update the original requirements when changes happen.
- Requirements Engineering and User Experience—Requirements Management practice
- Q: Are requirements updated as requirements change?
- A: 1. Rarely, 2. Infrequently, 3. Sometimes, 4. Most times, 5. Always, 6. Don’t know
In many organizations, we have seen requirements remain fixed in the requirements specification no matter what happens to the requirements themselves. What usually happens is that the developers go ahead and change the functionality to reflect the requirement change (that might have come as an e-mail or by phone), without changing the documentation.
So discussing this question at the interview will tell you more than the question itself would. Try to find out whether the organization has a requirements system or application, and if they do, whether it is used. You can also ask about how their change requests are handled, whether they have a process for that. If they have such a process, a developer would not implement the change. Instead, the developer would redirect the person initiating the change to the correct step of the change request process—usually by sending a formal change request to the project manager or whoever is in charge of this.
Example 2: Assessing Code Analysis
The next question we will show you covers code analysis. Code analysis enables us to make sure that developers follow a set of rules indicating how code must be written. Code analysis can include everything from naming conventions to more-specific coding practices. This analysis can be automated.
- Development—Code Analysis practice
- Q: Is there good static code analysis?
- A: 1. Rarely, 2. Infrequently, 3. Sometimes, 4. Most times, 5. Always, 6. Don’t know
There is no chance of having good code analysis without having it automated in some way. In Chapter 6, you will see that this is built into TFS, so we have access to it from there.
We use this question to find out more about the company’s use of tools for automating the development process. It’s a good starting point to dive into this subject, and to see whether the company has other areas where automation is or is not used. The answers will help you better understand how the organization can benefit from implementing TFS (or any other ALM tool).
Viewing the Results
When all participants have answered their assessments, the assessment owner closes the assessment so that no one can enter any more information. To see the results, the owner then clicks the Generate Report button (see Figure 4-6).
Figure 4-6. Generating a report for an Application Platform Capability Assessment
The questions are rated on a five-degree scale (1, 2, 3, 4, and 5) with a sixth choice being the possibility to answer “Don’t know.” The best score is 5, and the lowest is 1. (This is exactly like the scoring system we had in the Swedish schools a few years ago!)
The system calculates the medium score for each capability, for each capability area, and for the whole assessment and presents it graphically to the user (see Figure 4-7).
Figure 4-7. The report for an Application Platform Capability Assessment
You will see a text overview of the whole assessment. Our demo here shows a pretty good score of 3.56, which puts this organization at the Advanced level. In the table below this score, you can see the individual practice area scores. You can see the maturity level of each area as well. This information is a pretty good summary for management to look at. But if you want to see more detailed information, you can scroll down the web page to see the score and maturity level for each practice, as seen in Figure 4-8.
Figure 4-8. Report detail from the score for each practice
Now you can pinpoint any problem practices, which are practices with lower scores. You can look for the color red or yellow in the right column (not shown in Figure 4-9) to quickly identify such a practice. A manager might want to dive even deeper into the results, so farther down you will find the score for each question (see Figure 4-9). This setup gives you the capability to identify exactly where you have problems, in which practice—and then to use this information for planning corrective measures.
Figure 4-9. You can see the score for each question in the assessment
You probably want to download this report to use it internally, and Microsoft allows you to do this. You can save the report in different file formats so that you can process it any way you want.
Note You do not get any financial information in the Application Platform Capability Assessment report, only maturity scores at different levels of detail.
How to Use the Results
When we assess an organization’s ALM process, we need to gather as much information as possible about the client and the client’s organization. This is hard work if done manually. One would say it is close to impossible to collect that amount of information from so many people in the organization in a cost-effective way, without the use of a tool.
We used the questions from Microsoft’s assessment tools as a basis for conducting interviews with people from ALM organizations. We gathered people from all aspects of the ALM process, making sure both the business side and the IT side were represented. Then we spent 30 to 90 minutes discussing these questions in one-on-one meetings. After conducting all interviews, we completed the assessment ourselves and used the interview result as a basis for answering the questions. This way, we obtained a pretty realistic view of the organizations and their ALM processes. The results have also been better and have been more accepted by the organizations when we have done it this way as compared to when we have let only one person complete an assessment.
There are several ways to use the technique described in this book. We have tried it a few ways. Let’s first start with a few comments on the assessments themselves. Tools are good in most cases. Tools can help us with many tasks and simplify our lives greatly, but tools for assessing a complete ALM process? Could that work?
Using the Application Platform Capability Assessment
The team assessment dives down deep. Keep in mind that a tool cannot elicit all the nuances that an observer can. We use the Application Platform Capability Assessment questions as a basis for interviews, and make sure that we interview people about their special fields only. Architects answer architect questions, project managers answer project management questions, and so on. This approach has worked very well.
We strongly recommend using an external partner for the assessments. It is often easier to look at an organization when there are no strings attached, and no manager to answer to.
Why Do an Assessment?
Why should you spend the time and money doing an ALM assessment? The best reason is that before implementing TFS (or any other ALM tool for that matter), you need to know what the potential pitfalls in the ALM process really are. Every process has room for improvement, and the assessment is a very good way of finding out where improvements are most needed. You need to have as clear a picture as possible of the organization’s maturity level so that you can better anticipate what actions are needed for the organization to improve and thus be more effective.
The value of an assessment can be summarized in terms of what it provides. The assessment:
- Gives an analysis of the strengths and weaknesses with the current way of working
- Gives a foundation for prioritizing the ALM effort in the company
- Gives input to creating a roadmap for the improvement process
- Gives a baseline for follow-up assessments to measure the impact of an improvement project
An ALM Assessment will help your organization understand the current situation and make informed choices on the way to improve the ALM process. Often we think we know where the problem is, but before doing a proper analysis, it is hard to say. Making changes to the wrong thing(s) ultimately costs a lot of money—money better spent on correcting the real problem, which in the end could save money.
An ALM process is not something to implement all at once. You do it best little by little, piece by piece, starting with the lowest hanging fruit. If you just can show the decision makers the improvements of smaller actions, it becomes easier to get them to fund the rest as well.
We come back to this assessment throughout the book and tie together the topics with corresponding parts of the assessment.
Summary
This chapter has discussed the value of doing an assessment of the ALM process before implementing TFS (or any other ALM tool of your liking). The ALM assessments Microsoft offers are good but are best used in collaboration with an external partner carrying out the process in the form of interviews.
Use the assessment as a baseline for evaluating the impact of an improvement project. After a change in your process, you can perform the assessment again and again to measure and make sure you are moving in the right direction.