Collaboration with Development and Testing

The first part of the book focuses on collaboration with development and testing. I’ll cover some of the tools and techniques to improve how you work and communicate about your work.

You can’t be the lone sysadmin anymore known for saying “no.” The nature of the work may start at understanding operating systems, but it spans across understanding services across different platforms while working in collaboration with other teams within the organization and potentially external to your team. You must adopt tools and practices from across the organization to better perform your job.

You need to be comfortable with using the terminal and graphical interfaces. Just about every tool I’ll cover has some aspect of command line usage. Being able to explore and use the tools helps you understand when problems arise with the automation. When you have to debug the automation, you need to know whether it’s the tool or your use of the tool.

You can’t ignore version control. For years, DORA’s annual State of DevOps report has reported that the use of version control highly correlates to high IT performers.¹ Version control is fundamental to collaboration with other parts of the organization whether you’re writing code to set up local development and test environments or deploying applications in a consistent and repeatable manner. Version control is also critical for managing your documentation whether it’s README’s embedded in a project repository, or as a separate project that spans content for the organization. You administer tests of the code you write, as well as the infrastructure that you build within version control.

You build and maintain virtual images and containers for use locally as well as within the cloud. All of this requires some understanding of how to read, debug, and in some cases write code in a particular language. Depending on the environment, Ruby, Python, or Go may be in use.

Whether you are collaborating on a project with development, or just within your role-specific Operations team, you need to define and build development environments to replicate the work quickly that others have done. You then can make small changes to projects — whether they are infrastructure code, deployment scripts, or database changes — before committing code to version control and having it tested.

Configuration and Deployment

The second part of this book covers deploying and configuring software and services. Systems administration practices that work well when managing isolated systems are generally not transferable to cloud environments. Storage and networking are fundamentally different in the cloud, changing how we architect reliable systems and plan to remediate disasters. For example, network tuning that you might handcraft with ttcp testing between nodes in your data centers is no longer applicable when your cloud provider limits your network capacity. Instead, balance the abilities gained from administering networks in the data center along with in-depth knowledge about the cloud providers limits to build out reliable systems in the cloud.

In addition to version control, you need to build reusable, versioned artifacts from source. This will include building and configuring a continuous integration and continuous delivery pipeline. Automation of your infrastructure reduces the cost of creating and maintaining environments, reduces the risk of single points of critical knowledge, and simplifies the testing and upgrading of environments.

Scaling Production Operations

The third part of the book covers scaling system administration in greater detail. As a company grows, monitoring and observability, capacity planning, log management and analysis, security and compliance, on-call and incident management are critical areas to maintain, monitor and manage risk to the organization.

The landscape of user expectations and reporting has changed with services such as Facebook, Twitter, and Yelp providing areas for individuals to report their dissatisfaction. To maintain the trust of your users (and potential users), in addition to improvements to how you manage and analyze your logs, you need to update security and compliance tools and processes. You also need to establish a robust incident response to issues when we discover them (or worse when our users find them).

Detailed systems monitoring adds application insights, deeper observability, and tracing. In the past, system administration focused more on system metrics, but as you scale to larger and more complex environments, system metrics are less helpful and in some cases not available. Individual systems are less critical as you focus on the quality of the application and the impact on your users.

Capacity planning goes beyond spreadsheets that examine hardware projections and network bandwidth utilization. With cloud computing, you don’t have the long lead times between analysis of need and delivery of infrastructure. You may not spend time performing traditional tasks such as ordering hardware, and “racking and stacking” of hardware in a data center. Instance availability is near instantaneous, and you don’t need to pay for idle systems anymore.

Whether containerized microservices, serverless, or monolithic applications, log management, and analysis needs have become more complex. The matrix of possible events and how to provide additional context to your testing, debugging, and utilization of services is critical to the functioning of the business.

The system administrator role is a critical role that encompasses a wide range of ever-evolving skills. Throughout this book, I share the fundamental skills to support architecting robust highly scalable services. I’ll focus on the tools and technologies to integrate into your work so that you can be a more effective systems administrator.

A Role by any Other Name

To keep current with the tides of change within the industry, organizations have taken to retitling their system administration postings to devops engineer or site reliability engineer (SRE). Sometimes this is a change in name only with the original sysadmin roles and responsibilities remaining the same. Other times these new titles encompass an entirely new role with similar responsibilities. Often it’s an amalgamation of old and new positions within operations, testing, and development. Let’s talk a little about the differences in these role titles and set some common context around them.

Finding Your Next Opportunity

One of the reasons you might have picked up this book, is that you’ve been within your position for awhile, and you’re looking to your next opportunity. How do you identify positions that would be good for your skills and experiences and desired growth? Across organizations, different roles mean different things, so it’s not as straightforward as just substituting a new title and doing a search. Often it seems the person writing a job posting isn’t doing the job being described, as the postings will occasionally include a mishmash of technology and tools.

A danger to avoid is thinking that somehow there is some inherent hierarchy implied by the different roles even as some folks in industry or even within an organization assume this. Names only have as much power as we give them. While responsibilities are changing and we need to add and update our skills, this isn’t a reflection of individuals or the roles that they have now.

There is a wide range of potential titles. Don’t limit yourself by the role title itself, and don’t limit your search to just “sysadmin” or even “sre” and “devops”. From “IT Operations” to “Cloud Engineer” the variety of potential roles are diverse.

Before you even examine jobs, think about the skills you have. As a primer, think about what technical stacks are you familiar with? How familiar are you with the various technologies described in this book? Think about where you want to grow. Write all of this down.

As you review job reqs, as you note skills that you don’t have that you’d like to have write those down. Compare your skill evaluation with the job requirements and work towards improving those areas. Even if you don’t have experience in these areas, during interviews if you are able to clearly talk about where you are compared to where you want to be for those skills it goes a long way to showing your pursuit of continuous learning (which is a desirable skill).

Today, sysadmins can be devops engineers or site reliability engineers or neither. Many SRE skills overlap with sysadmin skills. It can be frustrating with years of experience as a sysadmin to see a lack of opportunities with the role sysadmin. If examined, often the roles advertised as SRE or devops engineer have very similar skills and expectations of individuals. Identify your strengths, and compare them with jobs requirements from positions that sound interesting. Map out your path and work on those skills.

Wrapping Up

Now that you know what to expect in the coming chapters and have some high-level context for how the sysadmin role has changed over the years, let’s dig into it. In the upcoming section, you will start examining tools and technologies to level up core foundational skills and practices that are critical to how you collaborate and work within local environments and communicate about the work that you are doing.