It’s important to clarify what we mean by “Salesforce” in this book, since the company has grown by acquisition, and the techniques shared in this book do not apply to all of the products that now fall under the Salesforce brand.
What makes Salesforce different? How is DevOps done in the Salesforce world? What is Salesforce DX? And how does it facilitate DevOps on Salesforce?
“Salesforce” vs. Salesforce
Salesforce is a vast and growing company with a vast and growing suite of products. It’s a challenge even to keep up with the names of the various products that Salesforce releases or attains through acquisition.
The focus of this book is on what’s known internally at Salesforce as “Salesforce Core.” Salesforce Core consists of Sales Cloud (the core CRM application), as well as Service Cloud (for customer support), Community Cloud (to create customer-facing web applications), and the Lightning Platform (previously known as Force.com). The first three of these constitute the SaaS part of Salesforce, while the Lightning Platform constitutes the PaaS component. All of these components of Salesforce Core work together seamlessly.
In the architecture of Salesforce, “Salesforce Core” is a massive, multi-gigabyte JAR file that is deployed across data centers and “Pods” around the world to allow secure multitenant access to customers via the Internet.
The “markitecture” of Salesforce (how Salesforce is presented externally by their marketing department) depicts a vast and cohesive range of other products such as Tableau, Heroku, Marketing Cloud, and Commerce Cloud, but these are all truly separate products, generally the result of acquisitions. Sharing data or functionality between these products and Salesforce Core requires some type of integration.
Although Salesforce is working hard to integrate these products in a way that is transparent to customers, these other products are developed and deployed in fundamentally different ways. For example, the methods of developing and deploying customizations to Marketing Cloud or Commerce Cloud are entirely different from the way customizations are developed and deployed to Salesforce Core.
Therefore, throughout this book, whenever we refer to “Salesforce,” we are referring either to the company itself or to this Core platform, and not to any of these other products.
How Is Salesforce Different?
On-premise systems require you to manage all of the resources yourself. IaaS, PaaS, and SaaS delegate progressively
Managing the Salesforce development lifecycle requires a unique skillset and approach compared to most other platforms. You can say that the DevOps movement was jointly realized by both developers (working in traditional languages such as Java) and system admins or operators (working on traditional infrastructure such as servers and databases). There are thus a vast range of well-established tools and techniques in the DevOps world for developing and deploying code and for managing and updating infrastructure. Unfortunately, almost none of them can directly be used for Salesforce.
To illustrate this, let’s look at how teams would manage infrastructure using AWS or deploy an application onto Heroku (a PaaS product also owned by Salesforce). Every aspect of AWS infrastructure can be represented using JSON configuration. JSON can be used to define which AWS services are used, which data centers they are running in, and how they’re configured. The AWS CLI (command-line interface) can be used in a continuous integration tool such as GitLab CI to automatically deploy updates to AWS infrastructure every time the JSON configuration changes.
Similarly, Heroku provides a platform to deploy custom application code (in Java, PHP, Ruby, etc.) and to specify which services (such as Postgres databases) are needed to support that application. The Heroku CLI can be used to automatically update the application whenever the codebase changes. Heroku also provides a tool called Pipelines that allows you to visualize and manage the entire development lifecycle without requiring a third-party CI tool.
AWS’s “infrastructure as code” approach is a delight for sysadmins, since it allows them to track changes and automate updates. Similarly, Heroku removes the vast majority of the setup and dependencies that developers would need to deploy their applications. Heroku provides a true Platform as a Service that is ready to receive custom-coded applications built in a traditional way: from the ground up.
Although we can say that Salesforce’s Lightning Platform is a PaaS, it actually works extremely different from true PaaS systems such as Heroku. The Lightning Platform allows you to write custom server-side or client-side code, but that code can only run on Salesforce. Although Salesforce allows you to define custom database tables and fields, that schema cannot be loaded into any database other than Salesforce’s. These customizations are effectively just changes to the configuration of one Salesforce instance. It’s therefore more accurate to say that every customization you can make to Salesforce is basically a Salesforce configuration change.
Salesforce is actually just a big application that happens to allow for infinite customization. But this means that the tools used for managing other IaaS and PaaS products cannot be used to customize Salesforce. Fortunately, however, the release of Salesforce DX means that the techniques and principles used with other technologies can now be ported over to Salesforce. That is the focus of this book.
DevOps in the Salesforce World
Moving from on-premise CRM software to Salesforce removes the need for servers and manual software upgrades. Consolidating customer support and online community management onto the CRM platform removes the need to integrate sales, support, and community applications. Migrating legacy applications onto the Salesforce platform allows those applications to share data and processes with the rest of the business.
Because moving to the cloud dramatically simplifies many of the challenges in delivering IT functionality, many Salesforce customers have been able to innovate quickly without having DevOps practices in place. But even without the hassle of managing servers and building software from scratch, the inevitable growth of complexity eventually causes companies to struggle with issues like orgs getting out of sync and delays in deploying functionality.
Salesforce solves so many IT headaches, it’s almost easy to overlook the fact that it’s created some new ones.
Salesforce has unleashed the ability for companies to focus on their core competencies, instead of struggling to provide basic IT services. Salesforce admins and developers are empowered to spend their time directly creating business value. But with multiple Salesforce admins and developers working at companies year after year, more and more Salesforce customers are finding themselves drowning in all of that “business value.”
If you have tens of thousands of components, each providing business value, but no systematic way of tracking, managing, or deploying them, you now have a new form of business pain and chaos. Far too much of a good thing.
What Is Salesforce DX?
Salesforce DX is an initiative begun by Salesforce in 2016 and that launched publicly at Dreamforce 2017. “DX” here stands for Developer eXperience. The goal of the initiative is to reenvision the developer experience on Salesforce with a focus on how to empower the Salesforce development community with the tools and processes needed to develop safely and effectively.
Salesforce DX is a very broad initiative, including several teams at Salesforce focused on environment management, custom coding, developer tooling, APIs, and more. Although the Salesforce DX teams tackle very diverse needs, their main focus (at least initially) is to improve the developer tooling and development lifecycle.
The tools and capabilities included in Salesforce DX are made available for free to all Salesforce customers. In that sense, Salesforce DX is an important complement to Trailhead, Salesforce’s free, self-paced, gamified, interactive learning platform. Trailhead is also a major investment for the corporation. Together these are intended to support the growth of skilled Salesforce professionals and to make it easier for those workers to build and innovate on the platform. These multiyear strategic initiatives help ensure that talent shortages and worker inefficiency will not be the limiting factor on the company’s aggressive 30% annual growth.
What Are the Elements of Salesforce DX?
Version control is the source of truth for each project. Development is done principally in scratch orgs (temporary Salesforce environments created from version control) as opposed to Developer sandboxes. Scratch orgs are discussed more in Chapter 6: Environment Management.
Code and metadata should be subdivided into packages representing discrete functionality. This reduces complexity and allows packages to be developed and deployed independently of one another. unlocked packages are discussed more in Chapter 9: Deploying. In addition, Salesforce DX includes new developer tools: the Salesforce CLI and a set of extensions for the Visual Studio Code IDE. The Salesforce CLI replaces an earlier set of Ant-based tools called the Ant Migration Tool, while the VS Code extensions replace the original Force.com IDE built on Eclipse. Salesforce’s choice to build on Visual Studio Code was prescient, since that free tool has come to dominate the code editor marketplace.
All of these capabilities and tools are built on top of Salesforce’s existing APIs such as the Metadata API, and so continuing to improve those APIs has also been the responsibility of the DX team. A persistent challenge for teams hoping to automate their development lifecycle has been that some configuration settings can’t be deployed automatically. Meanwhile, the platform continues to expand its capabilities. Many Einstein and Community capabilities were not initially supported by the APIs. A major victory for the DX team was enforcing automated processes behind the scenes to ensure that all new features on the platform are API accessible when they are released.
In addition, the DX teams have done a significant amount of developer outreach and evangelism to help customers understand how to take advantage of these capabilities. It’s my hope that this book can also be of benefit in this effort.
The Dev Hub
Some aspects of Salesforce DX are installed on a developer’s local machine or CI runner, while other aspects are capabilities built on top of Salesforce itself. The capabilities built into Salesforce include the ability to create new, ephemeral Salesforce orgs called scratch orgs and also to create and publish versions of unlocked packages. To make use of these, a development team needs to designate a production Salesforce org to be a Dev Hub and enable the “Dev Hub” setting in that org.
Enabling Dev Hub has no risk or side effects to the org. And since scratch orgs created from that org carry no data or metadata, giving developers access to a Dev Hub does not constitute a security risk. There is even a free license type called “Free Limited Access License” that can be enabled in a production org to allow developers or consultants to use a Dev Hub even if they wouldn’t otherwise have a user license in that org. This license type does not expose any data or metadata in the production org; it just gives access to use the Dev Hub functionality.
Each developer also needs appropriate permissions on that Dev Hub (see “Permissions Needed to Use the Dev Hub” in Chapter 6: Environment Management). For training purposes, Developer Edition orgs, including those created in Trailhead, can be used as Dev Hubs, although their limits are too restrictive for production use.
Scratch Orgs
As mentioned, scratch orgs are temporary, disposable environments similar to virtual machines. They are populated with code and metadata stored in version control and can be used to support development, testing, and continuous integration. These orgs live for only 7 days by default, although they can be configured to live up to 30 days. Scratch orgs are used for source-driven development, while sandboxes remain useful as long-running test environments. Packages and metadata are developed in scratch orgs and can be deployed to sandboxes and production using CI/CD.
Second-Generation Packaging
One of the most important characteristics of our bodies is the existence of many internal organs that each perform specialized functions while also working together as a whole. Well-organized software follows a similar pattern, known as modular architecture. The division of software into independent modules is a long-standing best practice that makes software easier to understand, maintain, and update.
Modular architecture in software exists at many levels, such as methods, classes, files, and packages. Each of these represents a layer of abstraction that brings similar benefits. The general idea is that the details of what is inside the module should not concern other parts of the system outside the module. What is of concern is the interface between that module and the rest of the system, for example, the input parameters and return values for a method. This allows each module a degree of independence, to be internally changed and refactored, as long as they don’t change their interface with the rest of the system. Similarly, when viewed from outside the module, all that matters is its interface. By hiding the underlying details, the whole system becomes easier to understand and work with.
Modules that comprise collections of many files that all perform related functions are called packages. Most if not all high-level software languages support different types of packages such as JavaScript modules, Ruby Gems, or .NET NuGet packages. This type of packaging is enormously helpful for developers since it allows them to build on pre-packaged solutions such as any of the 800,000 JavaScript modules on NPM1 rather than attempting to recreate such solutions themselves.
Salesforce has long supported the creation and installation of managed and unmanaged packages on its platform. Managed packages are typically used by ISVs to create commercial applications, since they hide their internal IP and prevent most functionality from being modified. Unmanaged packages are often used as a mechanism to share or install groups of related functionality. The AppExchange has been a market-leading business “app store” since its inception,2 and most of the “apps” available there are in fact managed or unmanaged packages that can be installed in a Salesforce org.
The challenge is that although unmanaged packages can be used to install related functionality, the metadata within that package does not remain part of that package. The package exists like a cardboard shipping container that does not perform any useful function once it has been delivered and unpacked.
The challenge with managed packages is that the method for creating and updating them is very challenging, involving the use of a separate Developer Edition org, packaging orgs, namespaces, and so on. For these reasons, few enterprises build or use managed packages to migrate functionality between their environments.
The consequence is that although most Salesforce orgs make use of commercial managed packages to extend their org’s functionality, homegrown customizations to orgs are almost never organized into packages. It’s common for large enterprises to have tens of thousands of unpackaged pieces of metadata (classes, objects, fields, Flows, etc.) that exist together as an undifferentiated collection.
Some organizations have disciplined themselves to use pseudo-namespace prefixes to distinguish related pieces of functionality, but for the most part it is not easy to see which pieces of metadata are closely related to one another without taking time to inspect their contents.
To address these problems, Salesforce DX introduces two types of second-generation packages: unlocked packages and second-generation managed packages. The former is mostly for use by enterprises, since it does not hide its contents or prevent them from being modified after they’ve been installed. The latter is a more flexible successor to managed packages, designed for ISVs to be able to deploy and update functionality while still retaining control over most of the contents of those packages.
This book is mostly oriented toward helping enterprises to manage their customizations, and so we discuss unlocked packages at great length. Fortunately, the methods for creating and deploying unlocked packages and second-generation managed packages are almost identical so most of the content in this book is relevant to second-generation managed packages as well.
Metadata API vs. SFDX Source Formats
Another important innovation in Salesforce DX was recognizing that the file format made available by the Metadata API was not conducive to team development in version control. It’s common for standard Salesforce objects like the Account object to be used by many applications and extensively customized. Those objects are represented by the Metadata API as XML files that can grow to tens or hundreds of thousands of lines long. Naturally, developers working collaboratively on these objects frequently encounter merge conflicts, issues with sorting tags, and invalid XML.
The traditional “Metadata API” structure of Salesforce files
The new “Source” structure of Salesforce files
You can read in detail about this format in the Salesforce DX Developer’s Guide.3
Salesforce Command-Line Interface (CLI)
The Salesforce CLI is a powerful command-line interface that simplifies development and build automation when working with your Salesforce org. Based on the Heroku CLI, the Salesforce team built a flexible, open source CLI engine called OCLIF, the Open CLI Framework. The Salesforce CLI was the first tool built on OCLIF, although other tools have followed.
The Salesforce CLI allows common Salesforce API commands to be called from the command line and also encapsulates complex processes like synchronizing source with a scratch org in concise commands. Importantly it also allows the output from those commands to be exported in JSON format so that it can easily be parsed and possibly passed as input to other commands.
The Salesforce CLI can be used directly on the command line, included in scripts such as CI jobs , and is also the underlying engine powering the Visual Studio Code extensions.
Summary
Salesforce is an extremely powerful and versatile platform. But it’s unique in many ways and it hasn’t been easy for professional developers to adapt their tools and techniques to working on this platform.
Salesforce DX is a major strategic initiative from Salesforce to ensure that industry best practices such as DevOps and modular architecture are possible on the platform. Salesforce DX includes many components such as scratch orgs, unlocked packages, and the Salesforce CLI. It also encompasses many teams such as the Apex and API teams working behind the scenes to empower these capabilities.
The significance of this shift to DevOps is explained in detail in the next chapter.