An Admin’s Guide to Doing DevOps

If your company has invested in a click-friendly release management tool like Copado, consider yourself lucky. You can become a DevOps superstar with clicks not code. If your company is taking a more traditional approach of directly using version control and general-purpose CI tools like Jenkins, don’t worry. You have an opportunity to learn powerful and flexible tools, and you can reuse those skills for as long as you work in IT.

A common adage is that DevOps tools are far less important than the culture shift involved. What matters is that everyone whose work impacts your production org is able to collaborate on common systems and that there is a shared sense of responsibility for enabling innovation and reducing risk and confusion.

DevOps implies a process (moving from Dev to production), so it’s important to appreciate the benefits of not making changes directly in production. Your work then is to get access to a development environment, make and capture your changes from that environment so that they can be sent through the deployment and testing process, and participate in progressively improving that process.

Getting access to a development environment might mean using a shared Dev sandbox or running commands to clone a short-lived sandbox or scratch org. As soon as you begin working in a development environment, you will understand why developers make such a fuss when those environments are out of sync with production.

Making your own changes (creating fields, using App Builder, etc.) is the part you’re already good at. This is the creative part of your job, and it’s natural if you initially resent all of the overhead involved in capturing your changes to version control. But if you’ve not spent much (or any) time writing code, it’s extremely empowering to begin tapping into the process of seeing your configuration represented as XML and to track and compare your work in version control alongside developers.

There’s an enormous mystique surrounding programming. Popular culture perpetuates this, as do developers, who enjoy the air of mystery and exclusivity of doing work that seems so alien to other people. But writing code is just writing. Code editors are mostly just text editors. And the gap between coders and noncoders is just the gap between those who are literate in a particular language and those who are not. Think of getting comfortable with code as developing basic code literacy. Just as illiteracy rates are falling globally (see Figure 12-1), so too code literacy will serve you well in the modern world.

Amazingly, code functions as a universal language. But it’s one where computers give you feedback about whether you’ve written it correctly or not. This makes it well suited to introverts who are happy to learn without human interaction. But even experienced coders struggle and fail like tiny children when first learning a new programming language or technique. Their peers may think they have superhuman skills, but, in private, even the most brilliant developers use “Hello World” examples and painstakingly pore through code snippets to get started with new technologies.

In my view, what distinguishes programmers is only that they have the patience, confidence, and motivation to work through this learning process again, and again, and again. Even a little experience using Git and running command-line scripts can give you the confidence that code is not a strange and inaccessible world. The best Salesforce admins I know have taken delight and pride in getting a little experience with code and the command line. Such experience unlocks the possibility of diving deeper into coding, even if you choose not to.

If you get over that hump, and get comfortable tracking your config changes in version control, you are more or less home free. If your team has set up a CI engine and delivery pipeline, tracking your work in version control is the only obscure and manual process you have to do. Computers will do the rest of the work, running deployments and automated testing.

You’ll have to face deployment errors. But you would face those even if you worked with change sets. And you might have to face Git merge conflicts. Merge conflicts for most metadata types are not hard to resolve, especially using a code editor like VS Code. But merge conflicts for Profiles and some other metadata types can be very ugly. This will help you understand why Salesforce is increasingly promoting the use of Permission Sets instead.

Finally, once you’re participating in this delivery pipeline, you share responsibility for keeping it running and making it better. Making it better principally means improving the automated tests. The testing tools your team is using may vary from developer-focused to admin-friendly. UI testing tools like Provar, Selenium, and Tosca are designed to be more accessible than code-based tests. If you have the opportunity to help build or specify those tests, you are helping ensure that the changes you make will be protected over time.

The logic you embed in a Process or Flow may be every bit as complex as the logic in a piece of code. Salesforce has begun to enforce code coverage requirements even on Flows and Processes, which is an interesting decision that affirms Salesforce’s view that automated testing is essential. Encourage your developers to write their unit tests in a flexible way so that the expected inputs and outputs can be tuned easily. Behavior Driven Development (BDD) style tests are meant to have human readable inputs and outputs that make it easy to create multiple variations of a test to examine business logic against different edge cases. If developers write tests in a modular way, you can easily copy and paste a single test and adjust the inputs and expected outputs to help validate the logic of your Processes, Flows, Workflows, or Validation Rules. Dabbling in unit tests is one of the easiest and lowest-risk ways to start to code, since they have no impact on the org’s behavior.

The point of adopting this approach is to bring admins and developers together in a common system that lets them experiment with less risk of impacting production users. Even Salesforce experts will occasionally screw up. If you’re not familiar with working in a development org, it means that every experiment you do puts your data, your users, and their affection for you at risk. Working on a common delivery pipeline is an investment in collaboration and governance that gives you more reason to talk to the development teams, and ensures you can build together with speed and deliver to production with confidence.

Locking Everybody Out

My favorite story about the adoption of DevOps for Salesforce comes from Lex Williams, formerly a lead member of technical staff on Salesforce’s own internal IT team. In the spirit of “dogfooding,” Salesforce makes extensive use of their own product to run their business. Salesforce’s production org is known as Org62. When Lex first joined Salesforce’s IT team, there was no automated process in place to deploy customizations to Org62. There was an automated build process based on Ant that was used to build the multi-gigabyte JAR file that is Salesforce itself, but there was not a mechanism to manage or track the customizations made to Org62.

Lex and team set about adapting some existing Perl scripts and setting up a Jenkins instance that could be used to deploy code-based customizations. Of particular interest was building and deploying the tool known as GUS, Salesforce’s Grand Unified System for tracking features and bugs for their product team.³ At the time (prior to the rise of StackExchange), there were not many resources available for debugging issues with Jenkins or other aspects of their automation, but Lex and team managed to pioneer an automated workflow that allowed them to deploy GUS and other innovations to Org62.

Nevertheless, the customizations deployed by the IT team were only a fraction of the total customizations being made by users in production. Salesforce grew to market domination on the promise of quick and easy customization. So it is not surprising that a massive number of Salesforce employees using Org62 expected to be able to make their changes directly in production. Salesforce actually had hundreds of people with System Administrator privileges in their org. And they were all making legitimate, valuable customizations to serve the needs of thousands of other Salesforce employees.

Despite the expertise of these Salesforce admins, some small percentage of their changes caused unexpected issues, small and large. Lex and the IT team regularly found themselves having to tackle critical production issues with often mysterious origins. Hours of debugging would frequently yield the conclusion that an ad hoc change made by one of these admins had unintended consequences. Naturally, the IT team began advocating for more control over changes being made in production.

Perhaps more than they would have at any other company, the IT team experienced resistance. “What part of clicks not code don’t you understand?” Why should teams be forced to route quick and simple changes through a regimented release schedule that would cause weeks of delay to a valuable improvement? The most strident resistance came from business users who depended on their admin colleagues to make quick changes.

After years of back and forth debate, the IT team finally began to compile statistics on the cost of responding to incidents caused by unplanned production changes. The incident response itself (scuttling a team of several developers to debug an incident, often outside of work hours) cost many thousands of dollars’ worth of productive time. They showed how changes such as a field being removed from a layout could prevent hundreds of users from doing their jobs; or a validation rule being added could break critical integrations with the payment processing system. It was clear that the cost of each incident was easily tens of thousands of dollars, if not more. It was clearly more costly to allow uncontrolled innovation than it was to restrict it.

And thus Salesforce themselves locked all the sysadmins out of production, at least in the sense of not allowing application changes to be made. Since roughly 2013, the most “admin-friendly” company in the world has forced all admins to go through a DevOps workflow to deploy their updates. With the result that every change is tracked, changes are propagated to every development and testing org in the same way they’re propagated to production, and incidents are far less common and far easier to debug than they were before.

If that’s the way Salesforce manages their own org, why should we treat our orgs with any less care?

So what does it mean to “lock people out of production”?

It is very helpful to have only one or two profiles or permission sets that grant admin-level privileges and to label them clearly as having elevated privileges. As a security precaution, it can also be helpful to add a dynamic step to your build process that enforces this. You can write a script that reviews all profiles and permission sets other than the ones with elevated privileges and either fails the build if restricted permissions are detected or explicitly disables those permissions by setting the XML permission to false. Actually setting permissions to false ensures that elevated permissions are removed from profiles, even if they happen to be added manually in production. Copado’s Compliance Hub is designed to help with concerns such as this.

What’s Safe to Change Directly in Production?

The beauty of Salesforce is the ease of making small changes quickly, without having to involve programmers. It’s thus very important to draw a clear distinction between which changes can safely be made in production and which changes need to go through the delivery pipeline. This distinction depends on your company’s needs and may evolve over time.

If these two reasons don’t apply, it’s safe to make the change directly in production. But the overwhelming majority of configuration changes actually fall into one of those categories.

The exceptions to this are when other metadata depend on these things, such as reports whose values are used as part of Visualforce pages.

Migrating such configuration data through your delivery pipeline requires a different set of tools from metadata migrations. Commercial tools such as Copado, AutoRABIT, and Gearset include hierarchical data migration as a native capability. There are also dedicated tools that specialize in data migration such as Prodly Moover.

Tracking Issues and Feature Requests

Here we come full circle.

The DevOps continuum is often depicted as a loop (either a circle or an infinity loop), because unlike linear production processes, software development is an ongoing iterative process. New features are requested, developed, tested, and released. Users identify defects (often from hard-to-foresee edge cases) and make new requests for the software to further improve their lives. Those defects and feature requests then begin their journey around this loop, as they are developed, tested, and delivered, only to yield to the next round of requests.

Requests often begin their lives as support cases. Admins (as distinct from app builders) are often directly beholden to end users, who might submit support tickets when they encounter issues. Support cases have a different lifecycle from agile user stories, in that support requests are definitely tied to the user(s) who submitted them, and there may be SLAs that dictate a response be sent promptly. It falls to the admin to triage those tickets and respond with instructions on how to do something or actions such as password resets.

It is only when there is not an immediate solution to a support case that it graduates into an issue/defect report or a feature request and then demands a different approach. At Appirio, our IT helpdesk used Salesforce cases to track support requests and a homegrown, Salesforce-based agile project management tool called CMC to manage Issues and User Stories. Our cases had a button that allowed them to be converted into an Issue (for bugs) or a Story (for feature requests).

How to distinguish between these two types of request? The distinction is the same as explained earlier in “What’s Safe to Change Directly in Production?” Things that are safe to change directly in production can be attended to quickly by admins. All other changes become requests for features or issue fixes.

Such a distinction is important for several reasons. First, it sets clear expectations for users that their request can’t be completed immediately. Second, it allows issues and features to be handled and prioritized alongside other work that the app builder admins and developers need to tackle.

Issues and feature requests imply changes that might affect the behavior of the org, or have side effects, and thus need a clear justification and need to be tracked and tested like other configuration changes. Having a continuous delivery pipeline in place opens up the possibility of expediting changes that are low risk and/or high priority.

Tracking a request as a feature or defect should not imply an indefinite lead time though. Being able to release simple, low-risk changes quickly is a key benefit of an automated deployment pipeline.

Summary

Salesforce admins are the frontline workers making the Salesforce org better for users. Salesforce empowers them to do this through its ease of customization, and we shouldn’t remove that ability without replacing it with something better. A smooth running delivery pipeline that provides version control, automated deployments across all environments, and robust automated testing is a superior system. But it is necessarily more complex and will require admins to grow comfortable with the changed approach and slight delay.

Reducing Lead Time (from committing a change to seeing it in production) is a key DevOps metric. And no one is more sensitive to long lead times than admins, who are accustomed to making instantaneous changes in production. Even most Salesforce developers lack familiarity with continuous delivery; so this is definitely an unfamiliar territory for most admins. But there’s joy and efficiency in admins being able to collaborate with developers in a development environment. And there’s relaxation and confidence in being able to roll back changes or investigate their origin. And fundamentally, a situation where orgs are out of sync for no discernible reason is a massive time waster for everyone involved.

Getting your admins participating happily in a delivery pipeline is the ultimate accomplishment in Salesforce release management. Don’t expect to get there immediately, and remember that some people will be slower to change than others. But take note of your successes and build on them. Share internally when version control “saves your butts.” And keep tracking and working to improve the key DevOps success metrics: lead time, deployment frequency, change failure rate, recovery time, and uptime.