CHAPTER 5: ASSESS AND RECOMMEND
Once the project information is gathered, it needs to be reviewed and assessed to pull out relevant content, observations and insights. This information will then be used to document the project assessment findings, determine a set of recommendations to optimize the project delivery, and take action on the risks and challenges. This chapter will explore some methods for assessing the project information, identify some key items to look for, and review the approach for determining a set of recommendations.
5.1 Assessment framework
The “art” of the project assessment is taking all of the disparate information gathered from interviews, document reviews and observations, and then synthesizing it into relevant findings. This is where the experience of the assessment team becomes important but there are a few different methods that could be used for assessing the information. These are highlighted in Table 5.1, along with some suggestions on when to use each method, as well as some “pros” and “cons” of each.
Table 5.1: Assessment approach comparison
Compliance to a corporation standard framework
The most basic way to assess project information would be to start with a checklist from a corporation framework and review whether or not the project has created all of the relevant project documentation. Many corporations have a standard set of project deliverables and processes which are expected to be completed for each phase, such as the project charter during the planning phase, or use cases during the requirements phase.
This list of mandatory documents can be used to set expectations for projects including what deliverables should be in their plans, and what they will be assessed for during quality reviews or phase gate meetings. The assessment can then review each item in the checklist and identify gaps between the checklist and the project documentation.
This approach can determine if key project documents exist, but often does not get very detailed information on how the documents are used, or how well the project is being managed. The approach also does not provide the perspectives, observations and concerns from the project team members. For these reasons it could be good to assess framework compliance but it may not be a very effective approach for improving the probability of project success.
Inventory and organize the risks
During the information collection phase of the assessment, many risks and challenges get identified. The assessment team may want to capture and build an inventory of the identified risks, and then review them for similarities which could allow them to be grouped together into categories. For example, the risks could be organized by the project phase (e.g. requirements or testing), or by the project management area (e.g. scope management or schedule management). This approach enables the insights from the team members to be organized, along with the documentation findings, into logical categories of work.
This approach is useful for organizing the content and seeing the breadth of challenges on the project. The assessment team should look to identify the impacts of these risks beyond just collecting them. For example, if there are risks with delays in the project schedule, then the impact may be a delay in the overall project timeline, or a reduction in the realization of business benefits.
This approach is good at aggregating the project risks but is not very quantitative. Just collecting and documenting the risks does not let stakeholders understand the magnitude of the risks, or the relative priority and impact between them. This is a good starting point but the assessment could be more effective (and valuable) by looking into scoring or benchmarks.
Scoring framework
Another assessment approach is to use a framework to “score” risk categories or project areas. In this model, each category gets a score based on a defined set of criteria and the findings of the assessment. The areas which can be scored include the project management functions (e.g. schedule, resources and cost), project phases (e.g. requirements, build and test), or other defined project dimensions.
There are several different models which can be used to score the project assessment findings. Three of these include a relative scoring model, a numbering scale model, and a graphical display model.
1. Relative scoring – in this assessment model a score is assigned to each category relative to each other, based on a defined criteria set. For example, the scale could be impact of risk and the ratings could be “high”, “medium” and “low.” Figure 5.1 shows a scale of function adequacy where the relative scores include “adequate”, “identified opportunities or risks” and “needs improvement.” In this example, the project scope management was found to be adequate, whereas the cost management function needs improvement, and the other areas have some risks identified. There is also a high level description for each function which justifies the score for that function.
2. Numbering scale – another assessment model uses numbers to score each component, based on a defined scale. This approach can also color code the scores based on the numbers, which is otherwise known as a “heat map” of colors, going from green (best score) to red (worst score). Figure 5.2 demonstrates an example where the numbering scale is measuring how well defined each of the key project management deliverables are on a particular project, with the lowest number being the best score and the highest number being the worst. In this case, the project charter exists and is current (score of 1) but the communication plan does not exist (score of 5).
3. Graphical display model – instead of using relative terms or numbers, graphical icons could be used to visually represent the scores. One example of this approach is the use of “Harvey Balls” which are round ideograms used for visual communication of qualitative information. Figure 5.3 demonstrates the use of Harvey Balls for how well a project capability has been demonstrated during the assessment. In this example, the schedule management capability has some improvements recommended, whereas the scope management capability is found to be demonstrated and adequate.
Each of these types of assessment scoring models can be used to convey different information and messages. Table 5.2 outlines some of those differences and considerations for using each of the scoring models.
Table 5.2: Assessment scoring model comparison
Scoring Model |
Considerations |
Relative Scoring |
• Uses a small number of score categories so it is easy to see risk area differentiation • The tolerances for each score need to be well defined so that there is clarity in the differences between a low score and high score |
Numbering Scale |
• Uses several score values so may be better for distinguishing subtle differences in scores • Clear definition of scale is important |
Graphical Display |
• Good for demonstrating differences in scores and comparing them • More visually appealing so the presentation may appear more professional |
Regardless of what type of scoring model is used for assessing information, this approach attempts to quantify, or show, the relative scores for each area of the project review. One challenge of this model is that sometimes it is hard to measure and score values for every risk, so the results may appear subjective and be questioned. The assessment team should ensure that there is a clear explanation for each score, and that there is a well defined set of criteria for determining each scored value.
Comparison to benchmark
A fourth approach for assessing information would be to compare the project findings to industry benchmarks for certain information. Robert C. Camp, in his book, “Benchmarking: The Search for Industry Best Practices that Lead to Superior Performance”, states that benchmarking can best be described as “the search for industry best practices that lead to superior performance” (Camp 1989). The assessment approach can include finding industry best practices for project management and delivery, and using those as a basis for comparing and scoring the project assessment against.
These best practices could include successful approaches used by other corporations, or metrics collected from industry research corporations that demonstrate top performing projects. There are several industry benchmarks which could be used to measure the project assessment findings, with some examples identified below:
• Process maturity models including models, such as Capability Maturity Model Integration (CMMI).
• Traditional project performance measures of schedule, cost and resources against project baselines.
• Quality measures, such as the Cost of Quality which measures the impact of delivering poor quality solutions.
• Advanced cost metrics, such as the Cost Performance Index which is a measure of project cost efficiency.
• Advanced schedule metrics, such as the Schedule Performance Index which is a measure of project schedule efficiency.
• Delivery quality metrics, such as defect identification rates and average defect resolution times.
Instead of the scoring for the assessment being based on assessor judgment or relative to subjective criteria, this scoring model allows for comparison to specific and recognized industry metrics. Therefore, this approach comes with a high level of credibility and the appearance of objectivity.
This subsection has identified four ways of approaching how to analyze the information collected from the project review. There are other approaches and these different aspects of these models can be leveraged and used together. The assessment team should consider the best way to approach the analysis based on scope, objectives and approach for the project review. Note that all of these assessment models should also allow for the recognition of where the project is performing well; the assessment should not just focus on deficiencies but rather be balanced in its findings.
5.2 What to look for
Once the assessment approach is determined, the team should review the findings to look for insights that can be used to populate the assessment framework. This can include specific themes in the findings, observations for how the project is being managed, or team member morale and perceptions.
Themes
There will be a lot of collected information to sort through and much of it will highlight common topics and appear to be repetitive, so the assessment team should look for these themes or trends. Table 5.3 provides a set of examples of themes which could be identified on a project assessment.
Table 5.3: Types of themes that may be observed
Type |
Description |
Examples |
Project Structure |
Challenges with how the project is structured and teams are organized |
• Unclear roles or responsibilities • Gaps in project accountabilities |
Team Dynamics |
Interpersonal or cultural challenges with team members |
• Lack of stakeholder engagement • Interpersonal conflicts • Lack of trust |
Process Deficiencies |
Gaps or lack of clarity in delivery approach, documents or roles |
• Documents not reviewed and signed off • Missed or incomplete deliverables |
Lack of Diligence |
Project management deliverables not being maintained properly |
• Plans not being maintained • Open risks, issues and actions past due dates |
Information Transparency |
Areas where transparency of project information may not be sufficient |
• Insufficient reporting of tracking to plan dates or finances • Lack of reporting on test cases of defects |
Resource Competency |
There may be some resources who are struggling in their roles |
• Challenges with performing activities • Inability to meet project commitments |
Stakeholder Engagement |
Challenges with sponsorship or leadership engagement |
• Lack of decisions • Escalations of issues not being effective |
Ineffective Communications |
Not all stakeholders are receiving ample communications |
• Confusion on the team • Conflicting understanding of status or risks |
There are certainly many other possible themes and the assessment team should look to aggregate the findings into common results. Generally, when there are significant problems on a project, they reveal themselves in several different places but all tie back to the same root cause. The specific examples and symptoms that demonstrate the themes should be aligned to them, as this will become important when the team is packaging the report and will want to justify the findings.
How the project is being managed
Many project assessments investigate the “what” of the project including what documents are developed and what processes are followed. The assessment team should also consider looking at “how” the project is being managed. The project team could have the best defined PM and delivery processes but if they are not managing them properly, then the project can still have challenges. I wrote a book titled “Applying Guiding Principles of Effective Program Delivery” which focused on the key guiding principles of how to effectively manage a program (Wills 2013). Eight primary guiding principles were identified which are also areas that may need evaluating when conducting a project assessment.
1. Diligence
This guiding principle means that the project manager is staying on top of all of the moving parts of the project including keeping plans, risk logs, issue logs, action item logs, financials and decisions current, and ensuring that dates get met. Diligence can also include proactively managing activities, such as looking ahead at vendor contract expirations, resource start dates, future forecasts and upcoming purchases, to make sure that they are all tracking to the expected plan. Usually it is the project manager who has to think and track all of these items, so if we are not staying on top of all of them, then most likely they will not get the attention needed and possibly will not be completed on time or accurately.
This principle can be assessed by reviewing the key project deliverables to see how well they are maintained by the project team. Project plans which are not updated with progress or project RAID logs, with dates past the due date, could indicate a challenge with the diligence of the project team.
2. Attention to detail
Accompanying diligence is the principle of “paying attention to the details.” If diligence is about providing structure and staying on top of all of the activities, then attention to detail is about understanding the specifics of those activities and making sure that the quality of the work is accurate and complete. This could include knowing the details of the scope and plan, or looking at the document specifics, such as confirming that all action items, issues, risks and plan activities have target dates and named owners associated with them, so that they can be assigned and tracked.
The assessment team should look for quality in the deliverables, such as financial information adding up, accurate content in status reports and maintaining detailed project plans. Consistent themes with not having proper details could indicate other problems on the project which need to be addressed.
3. Transparency
This principle is focused on gathering relevant project information and organizing it in a way that allows project stakeholders to understand themes, trends and risks to the project as early indicators of progress. This is important to have so that project managers have as much time as possible to take remediating action when a risk or issue arises. Transparency could include proper tracking and reporting of schedule progress, accomplishments, risks, issues and financial progress.
The assessment team should look to understand how project information is collected and how it is used to manage the project, with progress being reported out to stakeholders. Also, during the interviews they should pay attention to how well the stakeholders feel that they understand progress and risks, or can access that information to their satisfaction.
4. Single sources of truth
There is a lot of information on projects to manage and, therefore, the management of key project content becomes critical. As projects grow in size and as time passes, information gets spread in so many places that it becomes almost impossible to manage, let alone communicate to the team. The guiding principle of having “single sources of truth” means storing all relevant project information in one location. This could mean having a single location for project scope, schedule, risks, issues, finances or resource information.
Each of the project management functions being reviewed should have authoritative sources of information and the assessment team should be looking for these in the documents. Team member interviews should also reveal clues as to what the sources of truth are for key project information and how well it is maintained.
This principle means being able to use facts and data to enable timely decisions and actions on a project. There are many decision points on a project related to scope, issue resolution, risk mitigation and solution recommendations. This guiding principle allows for decisions to be made effectively and in an informed way because there is a fact base to justify a particular option or recommendation.
The assessment should look at how decisions get made and issues or risks get mitigated, and what data is used to support the process. Some project documents may identity the supporting facts for decisions which may provide some insight. Other insights may have to be facilitated during the team member interviews.
6. The “ships” in the fleet of accountability
Accountability, as it relates to managing projects, can be grouped into three buckets that have words which all end in the letters “ship”, so I refer to them as the ships in the “fleet of accountability” and the project manager is the captain of this fleet. They are ownership which means to truly take accountability for project outcomes, stewardship which is caring for the project and corporation, and leadership which is being a champion for the team and a leader on the project.
Each of these areas should be sought out in the project assessment. Because these are all characteristics of project leaders, they will have to be gathered through team member interviewing to look for relevant themes.
7. Simplicity
This principle is about keeping processes, technology solutions and organizations as simple as possible to avoid complexity, cost and confusion. On projects, complexity in any of these areas easily translates into more cost, more areas for issues to occur, and more risk to the ultimate delivery of the project. Consider the advice of Antoine de Saint-Exupery when he says that “It seems that perfection is achieved not when there is nothing more to add, but when there is nothing more to take away.” He said this in the early 1900s and it is still an applicable approach today.
The assessment team should look to understand areas of complexity and highlight them as places which may require additional attention for risk, or even to seek to simplify. Some of these areas may be broader than the project, such as organizational or process complexity, but the review team should still identify them as areas to consider, even if they are broader enterprise recommendations.
8. Taking a customer focused approach
The final guiding principle is to take a customer focused approach to all project work and recognize where there are customer relationships with project stakeholders (e.g. the developer is a customer of the design documents and requirements). The assessment team should try to understand what roles have a customer relationship and, through the interviews, determine how well they are working together.
Evaluating the “how” components can be challenging to discover as part of the assessment, since they are not as easy to find as looking to see if a particular document exists or not. However, they are important because they give insight into how well the project is being managed which has direct impact on the outcomes and success of the project. They also require the assessment team to be experienced in project management techniques so they can recognize if these principles are being applied.
Other considerations to look for
There may be other understandings that come out of the assessment of project information which would be relevant for the report. These additional areas should be recognized and contemplated as part of the assessment.
Intangibles
There can be intangibles on the project which are difficult to measure or quantify but which definitely have an impact on the project. The assessment team should be aware of these items when they are reviewing the project and seek to understand the impacts of them on the project, as they may be significant and worth highlighting.
• Corporation cultures and norms – corporations and organizations have distinctive cultures, practices and norms which may influence how team members behave and how the project work is managed. For example, in some corporation cultures showing yellow on a status report may be reflected as not being able to manage the project well which leads project managers to report on status as green when, in fact, it may not be.
• Cultural conflicts – cultures should also be considered when there is a project comprised of several different corporations who have different cultures which may be interacting with each other. For example, one corporation may have strong rigor around plans and process, whereas another may be more verbal and “loose” with process. In this case, the first corporation would be frustrated with the lack of planning for the second corporation, and the second corporation may feel overwhelmed with the process from the first.
• Resource and team dynamics – there may be undercurrents between team members that are possibly impacting project work. This could include fighting, different personalities and working styles, and a prior history of challenges between team members. Any of these items could impede the team’s ability to work together, collaborate and communicate. The assessment should be sensitive to notice any team dynamics and point them out, especially if they are causing problems on the project.
• Perception and reputations – sometimes projects can have a reputation in the corporation beyond the project team. This can include a reputation of being a troubled project, working team members too hard, or not meeting commitments. The assessment may want to consider seeking to understand if the project being reviewed has a reputation around the corporation and then look to determine the causes of these perceptions.
• Morale – projects are critically dependent on the team members to be successful. Therefore, projects which have a low morale on the team should be considered to have significant risk, as team members may leave, feel overwhelmed and produce poor quality work, or just give up altogether. The project assessment should consider the morale and energy of the team as an indicator of success or problems.
Conflicting information
The document reviews and interviews will yield a significant amount of information regarding the project. At times, some of this information may appear to be in conflict with other information gathered. The assessment team should identify where this happens and investigate further to understand the conflicts and determine what the correct information is. In some cases, this conflict may be a notable item to call out in the report if it demonstrates a key point, such as confusion of information or lack of transparency. Some examples of conflicting information are noted below:
• Team member interviews identifying information which is conflicting with another team member.
• Team members saying one thing but documents saying another. The team could think there are no challenges in a particular area, and yet the risk log or issues log may demonstrate otherwise.
• Stakeholders getting one update and believing one set of information but project documents and team members having a different view.
• Several documents having conflicting information, such as different dates for the same milestone.
• Status report for milestones may indicate that an activity is complete but the project plan is not marked as 100% for that activity.
• Status may show that a project is in green status but other project deliverables (e.g. risk log, issues log, financial reporting and project plan milestone progress) may indicate additional challenge areas which would suggest that the project is not in green status.
Looking for these conflicts is useful for the assessment because it demonstrates some of the challenges which the team may be having, but then uses the facts and documents to support the findings and determine the true story.
Since the assessment will be reviewing many aspects of the project, it may reveal existing challenges with how the corporation operates, beyond the specifics of the individual project. These more broad observations should be captured separately, with some recommended areas to consider for the corporation. Some examples could include the following types of recommendations:
• Interdivision operating model – projects are usually comprised of personnel from several different organizations who have to work together. The project assessment may reveal some challenges in the operating model between these divisions and could highlight these, with some recommendations to improve those interactions.
• Process improvements – most corporations have a software delivery lifecycle with a standard set of processes and deliverables. The assessment may reveal opportunities to improve some of these processes in the standard framework.
• Stewardship opportunities – the project being reviewed may have developed new management tools or deliverables which would be good to share with other projects, and should be recommended for the corporation standard framework. The project may also have very good examples of standard deliverables which should be submitted for framework as well.
Since the objective of the assessment is most likely focused on the project and not the broader corporation, the assessment team should determine the best way to present any enterprise findings or recommendations. It may make sense to have a separate set of findings that get documented and presented to a different team of stakeholders, instead of grouping these findings with the project specific ones. This could be collected as part of a corporation wide initiative to capture project lessons learned, and the spirit of continuous improvement of the corporation standard delivery methodology.
5.3 Determine recommendations
After the collected information has been analyzed and assessed for themes, gaps, key findings and insights, the team should identify a set of recommendations. This is where the experience of the assessment team members and the industry best practices should be used to propose actions that the project can take to remediate risks and resolve issues. There are many different categories of recommendations and next steps that the assessment team could identify; some of which are documented below:
• Additional structure – the assessment may find that the program structure, roles or responsibilities require additional clarification, or changes, to become more effective. For example, a certain troubled area of the project may require additional oversight, or resources, to help manage that work.
• Resource changes – the project may have resources who are not the right “fit” for the role that they are in, and it is having an impact on the project. In these cases, resources may have to be changed. This may also include changing the allocations to resources, where some critical resources may need to focus more on the project and have an increased allocation to the project.
• Planning or replanning – the assessment may find that the current plans are not realistic and may suggest revisiting the plans, validating the scope, and possibly recasting the schedule and milestones. This may also include reviewing the scope and looking for opportunities to remove or defer scope, as a means of helping a troubled project get back on schedule or budget.
• Improved communications – in cases where information was not clear to the project team, there may be recommendations about better team member engagement and an updated communication plan.
• End to end reviews – there may be some requirements or solutions which are at risk, or are very complex which could require a thorough end to end review of scenarios, design and testing solutions, to ensure that any gaps, or risks, get identified and acted upon.
• Focus on quality – in cases where there are findings regarding the quality of deliverables or outputs, the assessment could recommend additional actions to improve quality, such as document reviews or formal gates and sign offs.
• Ensure standards and consistency – if the project review was looking at adherence to corporation standards, there may be some recommendations around improving the alignment of the deliverables to the standards and better consistency.
• Productivity tools – the team may have some additional tools or processes that they would recommend to improve its productivity. For example, more robust financial or schedule tracking.
• Actions to close – there will probably also be several outstanding actions which require attention, such as missed plan dates and outstanding issues, risks, or actions that need to be closed.
• Simplification – complexity drives risk, so if the assessment finds that the processes or technology solutions are too complex, they may want to propose some recommendations around simplifying those items. This will allow the team to focus better on the important areas, and will also remove some of the challenges associated with complex solutions or processes.
• Make decisions – the review may highlight some outstanding decisions which need to be made, and there could be recommendations to do so, which would allow the project to progress past a stated challenge.
• Further analysis – even with the document reviews and team member interviews, there may still be follow ups for further understanding and assessment. The assessment team should identify when there are additional review areas or analysis suggested, and the reasoning for them. For example, the team may suggest a technology deep dive review into a solution that appears to be risky.
These identified categories represent just a few samples of possible recommended actions which could arise from the project review. The assessment team will need to carefully think through what actions they would recommend against each of the project findings. To that end, there are some considerations that the assessment team should contemplate when determining which recommendations they will be proposing to the project.
Prioritize the recommendations
There will most likely be many recommendations in the assessment and they will not all have equal impact on the project, so it may make sense to identify the impact and somehow prioritize or weight the recommendations accordingly. The suggested actions should be aligned to specific findings, with a priority assigned to them in terms of criticality of impact, or immediacy of need. This identification will allow the team to focus on those actions which have the most immediate and highest possible impact on project success. For example, a risk in meeting project commitments should have higher priority actions than recommendations to align document formats to corporation standards.
Identify a sense of effort
The recommended actions should also come with some measure of effort to implement which could be identified in effort hours, duration, additional resources and possibly even cost. This would give the stakeholders a sense of what it would take to implement the suggestion, and they could then decide whether or not to pursue them. Not all recommendations can be easily estimated, so the assessment team should determine when it is appropriate to provide effort and when they should not include this information. Instead of specific estimates, the team may also want to consider having a “high”, “medium” and “low” value of effort assigned to each of the actions.
Demonstrate and justify the value
The recommendations should be treated as a set of proposals which the assessment team is trying to convince the stakeholders to approve. Therefore, the recommendations should clearly state the value and show how they solve problems, and why the project team should perform them. For example, suggested focus on diligence with managing a project plan should call out that proper tracking of the plan provides early indicators of problems which allows time for management to take action and avoid further delays. This statement would be much more appealing to stakeholders than just “track your plan.”
Beyond stating the description and value of the recommendation, the team should also look to somehow justify them. This could include referencing industry standard best practices or material. This can bring credibility to the recommended action and substantiate the suggested value.
Keep them practical
Project assessment recommendations should be practical which includes considering the phase of the project, the culture of the organization, and the impact or value. For example, suggesting for a project to update requirement templates when it is in the testing phase and nearly complete, would not be a sound recommendation, even if the finding was valid that the standard formats were not used. Also, given that the time when projects are assessed is usually when the project is under way, the team is probably busy, so the recommendations should be considerate of their time.
Determine how they will be tracked
The project review should not end with handing off a set of recommendations and a hearty handshake. To close the loop on the assessment, there should be a method and plan for tracking progress against the recommended actions, to ensure that they get implemented. This approach should be agreed to as part of the process, as well as confirmed with the final report and presentation.
In conclusion, each finding and theme documented in the project assessment should have a recommendation, or proposed action, to take against it. Once the team has identified the recommendations and aligned them to the discoveries, they are ready to package them into the report.