Change Management and DR
“The one unchangeable certainty is that nothing is unchangeable or certain”
—John F. Kennedy
In this chapter I will describe two processes: one for managing nondevelopment change in SharePoint and one for approving and rejecting development changes to your SharePoint environment. The processes for assessing which adjustments should be made to your SharePoint system are referred to as change management. The continuous process of managing the life of an application through governance, development, and maintenance is referred to as application lifecycle management. I will provide you with templates for both of these processes. I’ll also describe some of the reasons why change is resisted by IT departments, why change is necessary, but also why too much change is dangerous in a way that can be more sudden and destructive than too little.
Things change. Organizations reorganize. People move to different roles or employers. Business requirements change to realign with the changing needs of customers. IT systems are patched and changed to grow with greater demand and usage from the business. Changing SharePoint is not simple. The SharePoint platform is complex and interdependent with other systems. As a result, changing it can have unpredictable results in SharePoint and in the interdependent systems. Trying to resist change by seeing it as merely a risk to the system, however, only leads to it being forced upon the system eventually by something beyond your control. It is fair to say there are risks inherent in change, such as the following:
- Unavailability
- Poorer performance
- Weaker security
- Nonadoption by users
- Greater cost
- Need for more training
There are potential gains, too, which are based on the fact that better technology should make your tasks and processes better. These gains include the following:
- Higher availability
- Faster performance
- Stronger security
- More adoption by users
- Lower costs
- A simplified user experience (UX) means less training
The focus in relation to change by those who own SharePoint should be proactive—a process of refinement and improvement. For this to be possible, you must first have processes in place to manage change in your SharePoint environment. If change can be done properly, it will be the lifeblood of your system, increasing its value to the organization constantly. If change is not managed properly, it leads to system stagnation and lack of use, and eventually it becomes a cost with no benefit to your organization. In terms of availability, the biggest threat to your SharePoint farm is the system owners’ inability to manage change. In a sense, change is the engine of SharePoint that needs constant fuel to keep the whole system moving forward and moving toward the organization’s goals.
Entropy
“Any method involving the notion of entropy , the very existence of which depends on the second law of thermodynamics, will doubtless seem to many far-fetched, and may repel beginners as obscure and difficult of comprehension.”
—Willard Gibbs, Graphical Methods in the Thermodynamics of Fluids (1873)
Like everything else in our universe, SharePoint is a system prone to entropy. I’m defining it as “the tendency for all matter and energy in the universe to evolve toward a state of inert uniformity.” Too much entropy is the enemy of SharePoint: if entropy takes control, SharePoint loses energy and eventually ceases to be useful to the organization. Ultimately, SharePoint is subject to what are known as the laws of thermodynamics: the relationship between energy and motion, or in a sense, change.
The second law of thermodynamics states that in general the total entropy of any system will not decrease other than by increasing the entropy of some other system. Your IT owner is acting on this idea: if SharePoint isolated from the users, the entropy of SharePoint will tend not to decrease. If you keep the users from changing it, SharePoint will not change. But keeping SharePoint useful takes work; it can’t simply be left unchanged to stay effective. Adding content doesn’t really change the system. The structure and processes in the system have to be changed and become more efficient. Users and the IT owners must be constantly looking for ways to use SharePoint to make their tasks easier, faster, cheaper, or at least more pleasant.
Also, to keep something running, you have to keep adding energy. You can’t make SharePoint “run itself” by one big shove at the beginning. This is an assumption in many businesses: put a large amount of effort into launching SharePoint and it will create its own momentum and run itself. It eventually runs out of momentum just by following the laws of entropy. The business then complains that it is SharePoint’s fault when it gradually becomes more inert and useless. Entropy takes over.
It follows that a reduction in the increase of entropy in a specified process, such as a chemical reaction, means that it is more efficient. This is good news for us if we put this in a SharePoint context: if we can make changing SharePoint less work, we not only make it more efficient, we reduce the risk of entropy in the system, which will destroy it.
There is an unspoken assumption among infrastructure people that system stability and reliability are best maintained by keeping the business away from the technology. They are trying to reduce entropy. The users’ needs are perceived as a threat to the integrity of the SharePoint farm. For the farm to continue to be stable, the IT department’s role is to resist change. IT think in the long run they are acting in the best interests of the users because the SharePoint farm is not being compromised by non–IT literates who don’t understand it. This assumption is fundamentally wrong because it ignores the fact that by not allowing users to apply their energy to the system, it will inevitably slow and stop.
Application Lifecycle Management
As indicated, the continuous process of managing the life of an application through governance, development, and maintenance is referred to as application lifecycle management. An important element that informs an organization’s approach to this process is its philosophy of application development and change management. In some cases, a formal, somewhat tightly controlled approach is preferred. In such an organization, custom development or packaged solutions are often the first course of action. Although understandable in many situations, applications like SharePoint offer organizations an alternative to this approach.
Complex developed solutions are a bigger commitment in terms of cost and maintenance, but are sometimes the only way to meet business needs. In this section one of the things I’ll consider is the actual cost of developed solutions.
Later I will describe the processes within the business for managing change (RFCs, BIAs, and CABs) and change testing and review. First, however, let’s look at several different types of development models and consider what makes the most sense for SharePoint changes. Of course, there will be different solutions depending on the scale of your SharePoint environment, the number of developers you have working on the same code, and the number of different pieces of code that have to be developed and deployed at the same time. However, some basic considerations are often overlooked or misunderstood, and they are important for flexible and manageable change in a SharePoint environment. They apply to many of the changes most organizations need.
Development Models
Bringing user requirements all the way to being used in production can be done in a number of ways. You will likely have heard of development project philosophies like “waterfall,” “Scrum,” and “lean.” These provide models for application development; understanding them provides important context for the variation I think works most effectively in many SharePoint situations. I’ll also describe that variation in this section.
Waterfall
Waterfall gathers requirements at the beginning. Users don’t see the solution until they are testing it at the end, close to deployment (see Figure 9-1). This model was developed over the past 20 to 30 years, starting at the time when developers had to book time on mainframes to compile their code. It has not advanced with the times, as more complex solutions can be developed more quickly now due to better programming object models. The model requires developers to plan, build, test, review, and deploy in one linear sequence. There is only one compilation of the code as part of deployment. If there are major mismatches between the solution and the requirements, it’s too late to fix them.
Figure 9-1. Waterfall is linear, like an assembly line
Iterative Waterfall
The iterative waterfall model was developed in response to the limitations of the classical waterfall model. This model is somewhat better, but it still requires a lot of risky up-front development, some testing, and finally deployment (see Figure 9-2). This is how software like Office is created. Approximately every two years a new version is completed. From a user point of view, two years is a long time to wait for new functionality.
Figure 9-2. The iterative waterfall process simply repeats the waterfall process
Scrum
Scrum has short phases where the solution is made usable and tested as quickly as possible, and features are gradually added until it is finally reviewed and deployed (see Figure 9-3), the benefit being that users can provide input sooner and the solution can evolve with them.
Figure 9-3. Multiple Scrums of independent features lead to one final deployment
The reason Scrum evolved is that users generally don’t know what they want until you show them something. For this reason, the hardest solution to write is a reporting solution—not because the code is difficult, but because the business tends only to know that it would be a good idea to look more deeply at the data they are producing, because they know there is information of value in there they are not seeing. The problem is they won’t know what’s valuable until they see it. A Scrum approach allows the solution to evolve and the business to learn what they potentially could do with the solution, and refine their requirements accordingly.
Lean
Lean takes the Scrum approach one step further: the product is constantly added to by separately creating independent features (see Figure 9-4). If a feature is added and users don’t like it, it can be removed. This is similar to how Gmail was developed. Features were incrementally deployed, but it functioned from the start.
Figure 9-4. Lean constantly adds separate new features
The main difference between waterfall, iterative waterfall, Scrum, and lean is how often the users are brought back in to assess the solution. It helps users to have a solution evolve so they can see what can be done. As a result, the lean approach is now the most common. The main benefit of all these approaches is they control the process of change in software, which reduces risk.
SharePoint is software in the sense that it has parts that are compiled and installed. Like all Microsoft software, it has already gone through some kind of build process. But while it has already been built, it is also a platform composed of a raft of features that allow people other than developers to change the relationship between them and their content. There has to be a process to manage these changes too, but it can’t be something as old-fashioned and simplistic as the waterfall model. What approach is needed for deploying SharePoint?
SharePoint Variation: Build/Plan/Test/Review/Deploy
The plan/build/test/review/deploy model doesn’t fit comfortably with SharePoint because it is an application that already has so many functions built in for creating solutions. Unfortunately, users rarely know about them, so developers are employed to step in and write code that may not be needed, but that has the added advantage from the business’s point of view of adding security because the changes can be put through rigorous change management processes.
The better solution with SharePoint is to remove the build step, which already is part of the process Microsoft uses to create SharePoint. Microsoft goes through the entire process before you ever install it, which simplifies the change process for you. Because most of the functionality an organization needs from SharePoint has already been created, the change process becomes more one of implementation than development. However, changes should still be researched and planned carefully. They should be tested in a nonproduction site or farm, reviewed by users, and only then deployed to production (Figure 9-5). For example, say you have a live InfoPath form tied to a workflow for approval, and the users want new functionality added to the form that fills in the values of two fields based on the contents of another. Achieving this can be planned with some research, then built into a copy of the InfoPath template deployed to a test SharePoint site. Once the business reviews and tests the change, it can be deployed into production with confidence it won’t destroy existing live data and will function correctly. It is a mixture of waterfall and lean. Let’s call it a lean waterfall!
Figure 9-5. Microsoft builds, but owners of SharePoint should still plan, test, review, and deploy
There is one simple reason why leaving development to Microsoft (most of the time) makes sense: cost. SharePoint probably had thousands of developers working on it over years at a cost of millions of dollars. This level of expense is not something most companies want to take on. That is why your approach to changing SharePoint should be different.
Sometimes organizations have change requirements that are only possible to make at a disproportionate cost. The most common example in my experience is where the business contracts a developer to meet their requirements rather than requesting the change through the owners of the infrastructure. The developer listens to the users’ requirements and fulfills them using a combination of AJAX, C#, and custom controls so that the work is done in a matter of weeks. The business is happy and the developer goes on his way. Months, weeks, or even days later, the users decide they want to make a small change to the pages the developer created. They think it is a small thing so they ask their internal IT department. This is the first time IT is notified of these changes. They don’t even know how the business got the access rights to make them; maybe from someone in the IT department who didn’t think it would cause any harm since the developer knew what he was doing. Regardless, now IT is expected to own this solution, and they have no documentation for what was done, how, or why. They spend time untangling the developer’s code and eventually make the change. They realize the developer could have simply used an OOB Web Part to do what they did, but likely knew very little about the application itself—only how to develop on the platform—so he wrote something that is actually very expensive to maintain.
The users now blame IT for the fact that every small change takes hours or days. They also resent the fact that even the smallest change has to be made by someone in IT. Eventually, the solution succumbs to entropy and is no longer used. It cost a lot to create and a lot to maintain, but it never stayed current with the business’s needs.
What the business didn’t do was consider the cost of ownership of the solution. They only thought of the initial cost of hiring the developer for a few weeks. Had they taken the time to learn what SharePoint can do OOB, they could have created something they could manage themselves. The initial cost to learn the skills might have been higher than paying a developer, but the ongoing costs would have been lower. This is like my example of a thermodynamic system that gets more efficient. IT could have provided guidance and support to the business at a lower cost to them, too. The conclusion: change can be costly, but in the long run, the returns can be higher if the energy expended at the start is expended wisely. Change must be managed in a collaborative way within the business for everyone’s benefit.
Solution/Problem
Solution packages have made deploying and activating code in SharePoint much more streamlined and automated. As a result, less is likely to go wrong. However, users and developers tend to use them as a substitute for the more controlled plan/build/test/review/deploy process. Solutions only aid with deployment. Furthermore, they don’t automatically provide for a process of incremental updating. The important questions are rarely asked. What if we want to change this later on? What will happen? How will we change the existing live system as seamlessly as possible? Solutions can cause problems. They are also subject to entropy, but now they have added greater complexity and risk into your environment. Users are dependent on the solution as it is, but it’s slowly getting less beneficial because it isn’t evolving with their needs.
For example, I was recently asked to help the owners of a training page by making some small changes to their SharePoint site. They required a link to a new forum they wanted created, but they also wanted to add a column to a Web Part on the home page. I quickly added the forum and the link, but the Web Part had been created using AJAX rather than using the OOB Web Parts or even SharePoint Designer. There was no documentation or comments in the code, and the developer was unreachable. The only solution was to hire another developer for a week to come in and make a change that should have been a few minutes’ work. The solution looked great, but the cost of owning it and changing it was too high to be practical. So the question should always be asked: what will it take to change this code later on?
Development Developments
Virtualization makes creating an environment for writing code against a development SharePoint instance much simpler now than even a few years ago. This means organizations can have multiple development, test, and UAT environments much more cheaply.
A standard development cycle goes like this:
- Users report a bug or request a change.
- Developers can run multiple builds in their development environments in parallel before deploying them to a collective test environment.
- Some of these builds are deployed to a UAT environment that users can access and report if the bug is fixed or the new functionality meets their requirements.
- Changes to production can now be put through the RFC/CAB process before they are made.
Production changes may be tested in a staging/preproduction environment.
SharePoint as a platform should make this kind of complex change management unnecessary. An ideal situation is where all evolution in the SharePoint environment is made at the browser or application level. I have seen many SharePoint environments where all development is stopped as a matter of policy, so users and the business are forced to use SharePoint to its maximum potential. That makes the platform easier to manage, but ultimately it limits the value of SharePoint as a platform to the business, which is regrettable. SharePoint can sometimes only be used to its fullest potential with custom code.
Virtualization does make having multiple environments cheaper, but where does it end? You can end up with multiple virtual farms all running different builds of different developed solutions. One dev/test/prod environment is only fine if you only have one change being made at a time. If you have 50, do you really want to have 50 virtual SharePoint farms? The best way to keep SharePoint vital and changing is for the users themselves to instigate and make changes. This may require effort on their part to learn SharePoint, but it’s a process that will yield more benefit to them directly than engaging external developers or trainers to whom they can delegate the responsibility.
Evolution
Everything in our universe follows the laws of thermodynamics. (Note I say “our” universe, because for all we know there are other universes with different laws.) But is change always good? Too much change in a SharePoint system can lead to a breakdown of the system by accelerating the entropy. That is what IT infrastructure owners are resisting. Changing SharePoint without testing and doing a proper business impact assessment of the change, and without review by the IT staff and the business, can lead to rapid failure of the system. Think of too much change as being like the pressure the captain of the Titanic was under by the passengers and the owners of the ship to get to New York in record time. By doing so, he would increase the publicity for the ship, increase the number of passengers, and ultimately increase the success of the whole operation. However, the big chunks of entropic ice floating down from higher and colder latitudes put a sudden stop to that plan. Entropy is not merely the risk to SharePoint of gradual inertia, it’s also the risk of sudden, catastrophic change caused by moving too fast.
The solution is to evolve the SharePoint platform in a gradual and tested way. This is how evolution works. Multiple variations occur in the environment and the fittest survive and continue while the unfit die off. Survival means your successful genes are passed on to your offspring and the cycle starts again. The environment inevitably changes and other species (indeed other members of your species) compete for the finite resources, but because there are multiple generations, change is always accommodated because the species as a whole can evolve to suit the change.
Who Controls Change in SharePoint?
Determining who should have what rights to change what is a balancing act between the needs of the owners to maintain the integrity, security, and manageability of the system and the needs of the business. This is a role that has to be taken by strong leaders who see the needs of the business and IT as a whole as opposed to different sides. Sometimes the solution to why a user or group is receiving an access denied error would take longer to find than just giving the user or group higher privileges. This solves the immediate business need, but in the longer term, the granting of excessive rights can lead to security or stability problems. From the business point of view, security is only used to control access to content, but from an infrastructure point of view, it’s to prevent users from making changes to the system that have negative repercussions—ones that IT must ultimately fix.
It’s also true that some organizations have overly strict policies. These just lead to users finding ways to break or sneak around the rules. There has to be communication between IT and the business. It is like any relationship: both sides have to see they are part of the same organization and have to act as the same team, not opposing teams. In my experience, this comes down to great managers, people who listen to both sides and make decisions in everyone’s best interest when others can’t do this for themselves. It’s true to say that leadership is not something you can formulate in a book. It is a process of synthesis that is more art than science. But the benefits are clear to see: businesses that get the best from technology.
Not all changes are equal. From a technical standpoint, to understand the scope of a change you have to be able to understand which category it belongs in. There is also the business impact of a change, which can have nothing to do with the technical impact. A change may be technically very simple to make but can have a very large business impact. An example would be unplugging the cooling system of a server room. But if a business person has to make a decision about a change, they have to have some basic understanding of the technical level. SharePoint application changes can be placed into three categories. (This list does not include platform patches or service packs, as they come from Microsoft or a third party rather than inside the organization.)
- Configuration
- Design
- Development
The category a change belongs in depends on the tool used to make the change and the rights required to make it. Changes are fundamentally assessed by their potential impact on the integrity of the overall system. The lower the impact, the lower the rights required to make them and the less restricted the tool required to make the change. Here are the three tools required to make changes in each category:
- SharePoint
- SharePoint Designer
- Visual Studio
When assessing a change, you must understand the scope of the change and its potential impact on the integrity of your SharePoint environment. The larger the impact of the change, the higher the rights required to make it. Granting these rights to people who don’t know the impact of their changes is poor governance. The principle of least privilege should always be followed: never give users more rights than they know how to use. These categories help you classify the change’s severity level and who should be trusted to make the change.
Configuration
Changes that require only the browser and the pages of the SharePoint sites are configuration changes. These can be subdivided depending on the level of access required to reach the pages, and these depend on the scope of the changes to the farm as a whole.
- Farm
- Site collection
- Site
- Page
- List/library
Configuration changes are made directly into your production environment and they are not automated unless a PowerShell script is used to make the change and minimize the risk of human error. Changes can be first made in a development or testing environment but then have to be manually redone in production. The changes generally can’t be replicated or migrated in one batch into production unless you are able to back up and restore from development or test to production. This would only be possible if there has been no change in production content. Making manual changes, no matter how well documented, adds the risk that the steps made in development or testing will not be replicated accurately. The person making these changes must fully understand them; otherwise there is room for misinterpretation.
Farm
Membership in the Farm Administrators group is granted in SharePoint Central Administration (see Figure 9-6). This is a site for managing the SharePoint farm as a whole. Changes made here impact the entire farm. For example, you can add a new Crawler server, or change when Search crawls content. To make changes here, you need to understand the impact not just on users but also on SQL Server, the network, Windows Server, and SharePoint itself. In other words, a change here is high risk and should be managed carefully.
Figure 9-6. Central Administration for a SharePoint farm
Site Collection
Changes at the site collection level impact that parent site and all the sites below it. For example, if you deleted an existing site template from the gallery, it would no longer be available to any site in the site collection. Users with site collection owner rights should take time to thoroughly understand the impact of changing any of the many somewhat obscure settings on the Site Collection Settings page (see Figure 9-7). Until they do understand, they have to listen to IT infrastructure’s point of view, which is always to preserve the system. This may initially be at odds with the needs of the business, but as the site collection owner’s knowledge grows, they will act in the interests of the organization as a whole.
Figure 9-7. Site Collection Administration options
Site
Changes at the site level impact that site and all the pages on it. For example, if you changed the site theme, that change would be seen on the pages of all the lists and libraries on the site. Users with site owner rights should thoroughly understand the impact of changing any of the many settings on the Site Settings page (see Figure 9-8). Learning about these settings is a process of experiment and discovery. You can also find many online and written resources. No one source is complete, however. The following is my process:
- How can I achieve this business need?
- From my research of written and online sources, how have others met this need?
- Can I test potential solutions in a sandboxed, nonproduction site?
- Can I show it to users for their feedback?
Applying a process like this may not achieve a perfect result every time, but it is better than a shortsighted jab at the first seeming solution.
Figure 9-8. Site Administration options
Page
The Web Part framework was conceived as a way to change page properties simply through a browser, such as the properties to view a Content Search Web Part (see Figure 9-9). Changes at the page level impact that page. For example, if you move a Web Part to the bottom of the page, all users who can access that page will see the change. It is also possible to specify that a change be only seen by you; this is a personal view of the page. As with the other changes, users with page edit rights should thoroughly understand the impact of their changes. Pages usually inherit as their administrator the administrator of the overall site; individual pages rarely have a different person as their owner. Users with specialized skills can even use content editor Web Parts to make very interactive changes to the page using technologies like AJAX and web services, but if they fail their impact is still only page level.
Figure 9-9. Changing a Web Part page
List/Library
Users can be given the rights to add content to lists/libraries, create or modify views within the list, or add/delete/modify columns to the list. Even these small changes can have a major impact on other users because if a column is deleted, it’s not stored in the Recycle Bin. As another example, metadata navigation settings make it possible to browse the content of libraries using the metadata as if they were folders (see Figure 9-10). This may confuse users unless it serves a prior need to make content easier to find. Just because a feature is available doesn’t mean you should enable it.
Figure 9-10. List settings
Importance of Managed Change
Just making configuration and structural changes through the SharePoint UI straight into production by users, site, or farm administrators introduces a complication: it breaks the cycle that is part of all these approaches, that of plan/build/test/review/deploy. With SharePoint, via the web browser only, you can make very impactful changes to your SharePoint environment and users. The higher the credentials the user has, the more impactful the change. Sometimes a change can have unforeseen consequences because the user is unaware of exactly who is dependent on what they have ownership of.
For example, I once had to troubleshoot a situation where 15 levels of sites in a site collection suddenly had SharePoint groups disappear overnight. As a result, teams all over the organization couldn’t access SharePoint, and there were hundreds of calls to support from aggrieved users who thought they had been personally locked out. Before I got there, Active Directory and SQL Server were checked, a Microsoft support call was opened, and SharePoint thoroughly checked for errors. Then foul play was suspected: had hackers deleted the groups? Finally the branch manager came in; I immediately suspected him because he had a history of doing little DIY SharePoint projects in the evening.
Previously, he’d tried to create a Web Part with SharePoint Designer and made the portal’s home page inaccessible. I spoke to him and he insisted he’d done nothing that could have caused this issue. Eventually, he remembered removing some user groups from “his site” that shouldn’t have been there. But he’d not done this on any other site. Therefore, there was a bug in SharePoint, et cetera, et cetera. It turned out that he had navigated from his site to the Site Collection Settings pages without realizing it. Because permission inheritance was turned on, he had not only removed the access of these groups to “his” site, he had deleted them completely from the whole site collection, thus removing the access rights of all the users in those groups across the whole 15 levels of sites in the organization!
To this day, that manager still has those rights; unfortunately, he insists he needs them, but also insists he doesn’t need to be trained to understand them. But the point of this example is that a change management process is dependent on good governance, which is dependent on users not having more rights than they understand how to use. It also requires that users who have rights take responsibility for using them.
Design
Design-level changes require SharePoint Designer (see Figure 9-11). SharePoint Designer allows you to make changes that impact the structure and visual presentation of a site, so although they can be made with Designer rights, they really should only be made with the approval of the site owner as they can impact all the content and users. For example, SharePoint Designer allows you to change page layout templates for a site. This change would impact every publishing page on the portal using that page layout.
Figure 9-11. SharePoint Designer
Like configuration changes, SharePoint Designer makes changes directly onto the production environment. Like configuration changes, they can be made in a development or testing environment, but the changes have to be redone in production. In fact, the changes generally can’t be replicated or migrated in one batch into production. This adds the risk that the steps made in development or testing will not be replicated accurately. As before, the person making the changes must fully understand them to avoid misinterpretation.
Development
While I have indicated that SharePoint should be seen as a complete application that is mainly configured rather that developed, SharePoint also has a rich environment for extending it and developing on it. This is not something always to be avoided—it should just be taken as something where the cost of ownership is understood and accepted. Development on SharePoint can automate configuration by adding content types, site definitions, Web Parts, and workflows, which can’t be done as well, or at all, through the web UI.
Development-level changes can be made directly onto the server, but it should be policy that only changes packaged as features or solutions should be deployed. A SharePoint solution is a deployable package that can contain features and assemblies. The solution package is a .cab-based file with a .wsp extension. The Visual Studio 2010 SharePoint project templates create .wsp package files for you as part of your development project. A solution might contain a number of SharePoint features, and these features provide functionality such as Web Parts, list definitions, modules, and event receivers. Features and solutions require PowerShell to deploy.
Like configuration and design changes, PowerShell will deploy the features or solutions directly onto the production environment. Like configuration changes, they can be made in a development or testing environment first and then redeployed in production. Because they are packaged, there are no complex steps to deployment. As before, the person making the changes must fully understand them to avoid misinterpretation.
Testing in Production
Knowing what can be done in production without risk comes with experience. But if you are not sure of the consequences of a change, how can you mitigate the risk? The principle that should always be followed when you are making any change in SharePoint is this: can I roll this change back if it doesn’t work as predicted? If the answer is no, take the plan/test/review/deploy approach. Remember that the goal is never to lose content it has taken a lot of time and effort to produce. For example, if you want to change an InfoPath form library template, you must understand that when you change it, all the forms already created are linked to that template and thus could be affected. Make a copy of the template, publish it to another site, copy in some test forms, and make your changes there first. You can never make a change and know exactly what the results will be, but the bigger the potential impact, the more important a test replica is.
No change management process is exactly the same, but they will all have some essential elements. When you construct your change management plan, you can incorporate these elements. They all help contribute to making change constructive and not destructive. They allow your SharePoint system to evolve just quickly enough not to succumb to entropy. I’ve illustrated these elements in a process flowchart in Figure 9-12. Here are the pieces:
- Some kind of form to capture the details required for the change request.
- Someone to review the initial request and ensure it has all the appropriate details correctly provided.
- An assessment of the business impact of the change.
- A Change Advisory Board (CAB) will approve/reject the request.
- The change must be tested and scheduled.
- The change is reviewed and marked complete or withdrawn.
- All comments, approvals, and supporting documentation for the change request are stored for later reuse.
Figure 9-12. A change management process
Impact Assessment
Executing these steps correctly is your most proactive method to achieve high availability. The core of this process is the change impact assessment. Anyone familiar with the Information Technology Infrastructure Library (ITIL) has heard of an impact assessment. A change impact assessment is a systematic approach that seeks to discover possible risks associated with a Request for Change (RFC).
Most failed changes are simply the result of not taking into account current activities and a failure to communicate anticipated activities. This includes any situation that proposes to alter system configurations, operating practices, policies, or procedures, and any new or different activities to be performed.
For change impact assessment, or simply an impact assessment, to be effective, the system owners have to explore all of the differences between normal operations and any conditions that introduce risks or may contribute to a failed change. Used effectively, impact assessment can proactively manage risk. It is a simple idea that can be implemented quickly using a word processor or spreadsheet. You can even use a SharePoint list. It requires no complex software. It does require careful adherence to a formal procedure, however, and teamwork.
Impact assessment is a fairly mature and formal activity in most organizations outside of IT. In particular, military branches around the world have deep experience in planning changes. Yet in many organizations most IT impact assessments consists of e-mailing around an RFC and waiting for someone to comment about it, or worse, one developer with too many rights working in isolation on “one little change.” Another popular method for impact assessment is a meeting, usually a day or two before the change is to occur, where department heads talk about the work upcoming.
It’s clear that existing impact assessments don’t work. Every year all the big think tanks report that IT is its own worst enemy. Gartner has documented that about 80% of all incidents occur because of failed change management activities (http://www.gartner.com/id=334197). It doesn’t have to be this way. Change impact assessment is well known outside of IT, and there are models for performing impact assessment.
The main change request process (shown in Figure 9-12) also calls a subprocess called “perform impact assessment” (see Figure 9-13). The subprocess contains tasks for risk management, effort estimation, and testing recommendations that can be executed in parallel. Once completed, information is passed back into the main process.
Figure 9-13. A change management subprocess: a business or change impact assessment
While effective and easily implemented, impact assessment is not a panacea and doesn’t totally replace existing change procedures. Instead, it’s simply part of the change management process. The procedure for performing an impact assessment consists of the following steps, and I have detailed how they should be used in the following example, Sheep Watch. I will also show how this subprocess fits into the overall change management process.
Applied Scenario: Sheep Watch Impact Assessment
Figure 9-13 shows the steps of the process as well as steps off to the side that denote side processes. To describe the change management process, I will use the example of a Web Part being added to the home page of our fictional university in Newbridge, County Kildare, Ireland. This Web Part was mandated by senior management. Its purpose is to display a map of the roads approaching the campus to warn staff where sheep are blocking traffic. Staff can add updates based on their observations from the windows of the building on the campus. The goal is to reduce traffic bottlenecks around the campus during the day and when staff members are driving home.
The Web Part is called Sheep Watch. The developer who created it in IT on her own development environment submits the RFC to have it added to the production portal home page. This RFC is a list item on a SharePoint site (see Figure 9-14).
Figure 9-14. Initial submission
The Change Coordinator receives an e-mail alert when the RFC is added. She reviews it and notices the developer has not said whether the Web Part will be deployed as a solution. She adds a comment to that effect and the developer reviews it and adds her own comment to confirm it is indeed packaged as a solution (Figure 9-15). It is the University’s policy to only deploy code in this way, so this had to be confirmed.
Figure 9-15. Review by Change Coordinator
The Change Coordinator decides this is not an emergency change and approves it for movement to the next phase. An “emergency” is any service interruption that is classed as high impact, either because of the number of users affected or because systems or services that are critical to the organization are involved, and so it must be responded to immediately. It should not be the result of poor planning! Every Wednesday, representatives from the IT department including the SharePoint Administrator, the SQL Server Administrator, and the Network Administrator meet. Their task to perform the next step in the process: the business impact assessment, or BIA (Figure 9-16). To do this they perform the following tasks:
- Define the extent of the change proposed.
- Determine key differences in the changed state (proposed) from the original state.
- Focus on the possible effects of the key differences from step 2.
- Sort and prioritize the possible effects (step 3) from the key differences (step 2).
- Make a decision using the results.
Figure 9-16. The Business Impact Assessment
Define the Extent of the Change Proposed
In the RFC, the developer writes that the Web Part is installed as a solution in the root site collection of the portal. It installs a Web Part that should then be added to the portal home page. It also creates a list to store the sheep reports and photos submitted by users. She describes the functionality of the Web Part as a dynamic map that displays the locations and sizes of the herds of sheep as submitted by users. She estimates that there will be 10 to 20 reports submitted per month and that the capacity required to store the list and photos will grow to 500MB in 12 months. She anticipates the Web Part will not delay the loading of the home page much.
Determine Key Differences Between the Proposed State and the Original State
The IT team discusses that the main difference will be the increased capacity to store the images, the performance effect on the home page, and the downtime to install the Web Part.
Focus on the Possible Effects of the Key Differences
They factor 500MB into their capacity planning for the year and decide it can be accommodated. One of the team has spoken to the developer and seen the Web Part and says it looks like it won’t slow the portal down much. The performance goal is to have the home page load in 2 seconds, and it loads really fast on the developer’s laptop. The solution can be installed outside of work hours to minimize the impact on users.
Sort and Prioritize the Possible Effects from the Key Differences
The team determines the key risk is that the capacity required for the list could grow faster than expected. However, since only current images have to be preserved, it is determined that older images can be deleted if the list grows too quickly. Risk and effort estimation are determined to be low, and testing recommendations are that the Web Part be installed on the test system first to verify that the install works.
Make a Decision Using the Results
There is an SQL Server update scheduled for the coming Saturday, and they decide to install the Web Part at the same time, subject to approval at the CAB approval meeting scheduled for that Friday.
Change Advisory Board (CAB) Meetings
A Change Manager should always act as the Chair of any CAB meetings, either virtual or face to face. CAB meetings should be called by the Change Manager at appropriate times to ensure the prompt and efficient handling of all changes. During high levels of change, this could potentially be daily, but usually they are every Friday. For complex, high-risk, or high-impact changes, or when major projects are due to deliver products, a formal CAB meeting would be necessary. The meetings can then be used to provide a formal review and authorization of changes, a review of outstanding changes, and to discuss any impending major changes. Where face-to-face meetings are appropriate, they should have a standard agenda.
Relevant change information should be circulated in advance to allow CAB members to conduct impact and resource assessments prior to the CAB meeting. The CAB called by the Change Manager should consist of attendees who are relevant to RFCs being considered. This could potentially include attendees for other groups and parts of the business outside IT. Authorization at the CAB for each change must be given by appropriate representatives from all areas the change will affect. In our example, representatives from the users who own the SharePoint content and the management team who requested the Sheep Watch web part should be present.
A Standard CAB Agenda
A standard Change Advisory Board meeting should be structured and well chaired to make it as efficient as possible. It could include the following as appropriate:
- A review of all failed changes
- A review of all backed-out changes
- A list of RFCs to be assessed by CAB members
- A review of all implemented changes
- The change management process, including any amendments made to the process and proposed changes to the process (as appropriate)
- Change management successes for the period under discussion, such as a review of the business benefits seen as a result of the change management process (as appropriate)
CAB Considerations for Each Change (Prior to Authorization)
These considerations will include the information prepared at the BIA stage, but presented in a clearer and more concise way. The members of the CAB may not be very technical, so the technical aspects of the assessment of the change should be offered in a clear, jargon-free way.
- Risk/impact assessment (on the business)
- Effect upon the infrastructure and customer service, as defined in the SLA, and upon the capacity and performance, reliability and resilience, contingency plans, and security
- Impact on other services that run on the same infrastructure (or on software development projects)
- Resource assessment—the IT, business, and other resources required to implement the change, covering the likely costs, the number and availability of people required, the elapsed time, and any new infrastructure elements required
- The impact on non-IT infrastructures within the organization
- Effect/risk/impact of not implementing the change
- Other changes being implemented on the schedule of change
- Technical capability and technical approval
- Financial approval (if required)
- Third party/supplier involvement in the implementation of the change
- Business approval (if required)
- Review/assessment of the change priority
All CAB comments on each change and any issues that have been discussed must be documented by the Change Manager within the CAB meeting minutes. These will not be things that directly affect the process; they are just a record of individual points of view if there are differences of perspective. The issues may not lead to a rejection, but they will be noted and incorporated into the testing or the change itself.
CAB Recommendations/Decisions
All CAB recommendations and decisions that have been discussed must be documented by the Change Manager within the CAB meeting minutes. Some decisions and ultimately approval may have to go beyond the CAB. This would be unusual, but if the decision is escalated, it is still recorded and implemented through the CAB.
Once all aspects of the change have been considered (as per the CAB considerations outlined previously) the CAB will then give authorization for the change to be scheduled and moved into the change build stage of the process (Figure 9-17).
Figure 9-17. Outcome of CAB
If a change has been authorized by the CAB, the RFC can then be scheduled. If the CAB can’t make a final decision on authorizing a change, then the change escalation needs to be initiated by the Change Manager to ensure that authorization is given at a higher level. This could be the head of the University in our example, but this will rarely be the case. The escalation of change authorization is documented in the request for change—the Change Manager will detail to whom the change was escalated and the final decision that was made, either authorized or rejected.
Rejection of a Change by the Change Advisory Board
If the CAB rejects the change, the Change Manager must document in full the reasons for the rejection and ensure that the decision is communicated to the person who requested the change. In our example, the Sheep Watch Web Part was approved. If it is rejected completely, the proposer can begin the process again with an improved version of the solution.
Schedule RFC
After the CAB meeting, the approval is communicated to the infrastructure owners. This communication could be in the form of an e-mail, document, or spreadsheet. Before testing of the change beings, the best time to make the change can be set (Figure 9-18). This will be a time that fits in with other scheduled changes in a way that prevents them impacting each other. It will also be a time that doesn’t impact other processes such as backups. Finally, it will be at a time that has minimal impact on users while still meeting their needs. For example, a change to the payroll system might be best done a few days before the end of the month to ensure it is fully and properly deployed before the payroll is done.
Figure 9-18. Schedule the change
Test Change
Testing (see Figure 9-19) has to be done in as close an approximation of live as possible. With the advent of virtualization, it’s easier to have replica builds of farms that can be rolled back. Even without virtualization, a staging or preproduction environment can be used to perform a test. This environment should be as close to a copy of production as possible. The more differences, the more risk that the test could miss a problem with the change. The test process should therefore include testing the rollback process for the change; now is the time to do this, not after an unexpected event in the live environment. Define at this point the criteria for the successful deployment of the change. Has its impact on resources been within planned parameters? Does it function as expected? Is it producing any errors in the system logs or within the application?
Figure 9-19. Test the change, implement the change, assess the implementation, and maybe roll back the change
Implement and Assess, Perhaps Roll Back
In some ways the process of implementing and assessing the change is the same as testing it. The deployment process and the checks to ensure it went smoothly will be the same: Has its impact on resources been within planned parameters. Does it function as expected. Is it producing any errors in the system logs or within the application? If it has met these criteria, it’s a success; if not, it has to be rolled back. This is also a process you should test (see Figure 9-19). Sometimes a change doesn’t go completely to plan but may be left in production. This is unusual, though. Usually the previous state is restored, and the change request process starts again.
At this point, any lessons learned during the deployment should be noted as part of the process of formally closing the RFC. Its criteria for success should have been met, and if not, this information should be noted and communicated to the appropriate party. For example, if the code is using more resources than were planned, it can be reverted to the developers for a fix or to Infrastructure to add more resources. In either case, the process is complete (see Figure 9-20).
Figure 9-20. Review and close
Summary
Change management is a collaborative process where the impact of change has to be assessed from a business and a technical perspective. Change is the lifeblood of SharePoint; without it the system succumbs to entropy, becomes less and less relevant to user needs, and becomes a burden rather than a boon to the business. SharePoint does require change management. The best approach is one where building the software is Microsoft’s task, but planning, testing, reviewing, and deploying configuration changes are done by the users.
The exception is custom code, and because of its potential impact on the platform it must have a more stringent change management process. Solution packages make deployment more automated, but they are not a substitute for full application lifecycle management. By policy, custom development should be the exception, not the rule, because SharePoint at its best is a user-driven tool. Custom code puts the platform at greater risk and takes the control out of the hands of the users and infrastructure owners.
SharePoint is successful because changing it is not something that relies on arcane, hard-won specialist knowledge. It can be done by learning which changes can just be made and which require planning and testing before deployment. If a change can’t be immediately undone with no impact on the system or users, it needs planning and testing.
If changes can be kept within the scope of what users can do, change can be faster without resorting to a complex RFC/CAB process. Without change, SharePoint dies, but it doesn’t have to stay static either: managed change, or the “lean waterfall,” is the solution. With it, SharePoint can grow in a way that ensures its continued benefit to the organization.