9 A practical interlude
It’s time to put some of your new knowledge to work. In this chapter, we’re not going to teach you anything new. Instead, we’re going to walk you through a detailed example using what you’ve learned. This is an absolutely real-world example: We’re going to set ourselves a task and then let you follow our thought processes as we figure out how to complete it. This chapter is the epitome of what this book is all about, because instead of just handing you the answer on how to do something, we’re helping you realize that you can teach yourself.
9.1 Defining the task
First of all, we’re assuming that you’re working on any operating system running PowerShell 7.1 or higher. The example we’re going to work through may very well work on earlier versions of Windows PowerShell, but we have not tested this.
In the world of DevOps, there is a specific language that almost always pops up—besides PowerShell, of course. It’s a rather controversial language among IT pros and DevOps engineers—people either love it or hate it. Any guesses? If you guessed YAML, you’re right! YAML stands for “YAML ain’t markup language” (it’s what we call a recursive acronym, when the acronym contains the acronym), and although it says that it isn’t, in a lot of ways it is similar to a simple markup language—in other words, it’s just a file with a certain structure to it, just like CSVs and JSON have a certain structure to them. Since we see a lot of YAML in the DevOps world, it’s important that we have the tools to interact with YAML files.
9.2 Finding the commands
The first step in solving any task is to figure out which commands will do it for you. Your results might differ from ours, depending on what you have installed, but it’s the process we’re going through that’s important. Because we know we want to manage some virtual machines, we’ll start with YAML as a keyword:
Hmm. That wasn’t helpful. Nothing showed up. Okay, let’s try another approach—this time, we focus on commands rather than on help files:
Okay, so there are no commands whose names contain YAML. Disappointing! So now we have to see what might be in the online PowerShell Gallery:
PS C:Scripts > find-module *YAML* | format-table -auto Version Name Repository Description ------- ---- ---------- ----------- 0.4.0 powershell-yaml PSGallery Powershell module for serializing... 1.0.3 FXPSYaml PSGallery PowerShell module used to... 0.2.0 Gainz-Yaml PSGallery Gainz: Yaml... 0.1.0 Gz-Yaml PSGallery # Gz-Yaml...
Much more promising! So let’s install that first module:
PS C:Scripts > install-module powershell-yaml You are installing the module(s) from an untrusted repository. If you trust this repository, change its InstallationPolicy value by running the Set-PSRepository cmdlet. Are you sure you want to install software from 'https://go.microsoft.com/fwlink/?LinkID=397631&clcid=0x409'? [Y] Yes [A] Yes to All [N] No [L] No to All [S] Suspend [?] Help(default is "N"): y
Now, at this point, you do have to be careful. Although Microsoft runs PowerShell Gallery, it doesn’t validate any of the code that others publish. So we did take a break in our process, review the code that we’d just installed, and made sure we were comfortable with it before we continued. It also helps that the author of this module is a fellow MVP, Boe Prox, and we trust him. Now let’s see the commands we just gained:
PS C:Scripts > get-command -module powershell-yaml | format-table -auto CommandType Name Version Source ----------- ---- ------- ------ Function ConvertFrom-Yaml 0.4.0 powershell-yaml Function ConvertTo-Yaml 0.4.0 powershell-yaml
Okay, those seem straightforward. ConvertTo and ConvertFrom—sounds like all we need.
9.3 Learning to use the commands
With any luck, the author included help with their module. If you ever write a module someday, always remember that if other folks are going to use it, you should probably—no, you should—include help so that consumers of your module know how to use it. If you didn’t, it’d be like shipping a piece of IKEA furniture without the instruction manual—don’t do it! There is another book called Learn PowerShell Toolmaking in a Month of Lunches (Manning, 2012), where two of its authors, Don Jones and Jeffery Hicks, cover how to write modules and how to put help in them—and adding help is the right thing to do. Let’s see whether the author did the right thing:
PS C:Scripts > help ConvertFrom-Yaml NAME ConvertFrom-Yaml SYNTAX ConvertFrom-Yaml [[-Yaml] <string>] [-AllDocuments] [-Ordered] ➥ [-UseMergingParser] [<CommonParameters>] PARAMETERS -AllDocuments Add-Privilege
Drat! No help. Well, in this instance, it’s not too bad because PowerShell helps even though the author didn’t write any help. PowerShell still gives you the syntax, parameters, output, and aliases—that’s enough info for this straightforward command. So we need a sample YAML file. . . . Well, it’s worth using a live example from the PowerShell GitHub repository:
https://raw.githubusercontent.com/PowerShell/PowerShell/master/.vsts-ci/templates/credscan.yml
This is the Azure Pipelines YAML file that the PowerShell team uses to run CredScan—a tool used for scanning if secrets or credentials have accidentally been added to code. The PowerShell team has it set up to run every time someone sends a pull request (aka a code change) to the PowerShell GitHub repository so that it can be caught immediately. Running tasks during a pull request is a common practice called continuous integration (CI).
Go ahead and download that file, and let’s read the file from PowerShell:
PS C:Scripts > Get-Content -Raw /Users/travis/Downloads/credscan.yml parameters: pool: 'Hosted VS2017' jobName: 'credscan' displayName: Secret Scan jobs: - job: ${{ parameters.jobName }} pool: name: ${{ parameters.pool }} displayName: ${{ parameters.displayName }} steps: - task: securedevelopmentteam.vss-secure-development-tools.build-task ➥ -credscan.CredScan@2 displayName: 'Scan for Secrets' inputs: suppressionsFile: tools/credScan/suppress.json debugMode: false - task: securedevelopmentteam.vss-secure-development-tools.build-task ➥ -publishsecurityanalysislogs.PublishSecurityAnalysisLogs@2 displayName: 'Publish Secret Scan Logs to Build Artifacts' continueOnError: true - task: securedevelopmentteam.vss-secure-development-tools.build-task ➥ -postanalysis.PostAnalysis@1 displayName: 'Check for Failures' inputs: CredScan: true ToolLogsNotFoundAction: Error
Okay great, so we’ve figured out how to read in the YAML file. Next let’s convert it into something a bit easier to work with:
PS C:Scripts > Get-Content -Raw /Users/travis/Downloads/credscan.yml | ➥ ConvertFrom-Yaml Name Value ---- ----- parameters {pool, jobName, displayName} jobs {${{ parameters.displayName }}}
Well, then. That was easy. Let’s see what kind of object we have:
PS C:Scripts > Get-Content -Raw /Users/travis/Downloads/credscan.yml | ➥ ConvertFrom-Yaml | gm TypeName: System.Collections.Hashtable Name MemberType Definition ---- ---------- ---------- Add Method void Add(System.Object key... Clear Method void Clear(), void IDictionary.Clear() Clone Method System.Object Clone(), ... Contains Method bool Contains(System.Object key)... ContainsKey Method bool ContainsKey(System.Object key) ContainsValue Method bool ContainsValue(System.Object... CopyTo Method void CopyTo(array array, int... Equals Method bool Equals(System.Object obj) GetEnumerator Method System.Collections.IDictionary... GetHashCode Method int GetHashCode() GetObjectData Method void GetObjectData(System.Runtim... GetType Method type GetType() OnDeserialization Method void OnDeserialization(System.Object...Remove Method void Remove(System.Object key), voi... ToString Method string ToString() Item ParameterizedProperty System.Object Item(System.Object ➥ key... Count Property int Count {get;} IsFixedSize Property bool IsFixedSize {get;} IsReadOnly Property bool IsReadOnly {get;} IsSynchronized Property bool IsSynchronized {get;} Keys Property System.Collections.ICollection K... SyncRoot Property System.Object SyncRoot {get;} Values Property System.Collections.ICollection Value...
Okay, so it’s a hash table. Hash tables are just a grab bag of stuff. You’ll see them a lot throughout your PowerShell journey. These are great because they can easily be turned into other formats. Let’s try to take what we got in YAML and turn it into another very important data structure in DevOps—JSON. Let’s see what we have to work with:
PS C:Scripts > Get-Help *json* Name Category Module Synopsis ---- -------- ------ -------- ConvertFrom-Json Cmdlet Microsoft.PowerShell.Utility... ConvertTo-Json Cmdlet Microsoft.PowerShell.Utility... Test-Json Cmdlet Microsoft.PowerShell.Utility...
Bingo, a ConvertTo-Json cmdlet. Let’s use the pipeline to convert YAML into JSON. We’ll need to use ConvertTo-Json’s Depth parameter (you can read about this from the help), which allows us to specify how deep the cmdlet should go in trying to make the JSON structure. For what we’re doing, 100 is a safe bet. All right, let’s bring it together:
PS C:Scripts > Get-Content -Raw /Users/travis/Downloads/credscan.yml | ➥ ConvertFrom-Yaml | ConvertTo-Json -Depth 100 { "parameters": { "pool": "Hosted VS2017", "jobName": "credscan", "displayName": "Secret Scan" }, "jobs": [ { "job": "${{ parameters.jobName }}", "pool": { "name": "${{ parameters.pool }}" }, "steps": [ { "task": "securedevelopmentteam.vss-secure-development-tools.build ➥ -task-credscan.CredSca "inputs": { "debugMode": false, "suppressionsFile": "tools/credScan/suppress.json" }, "displayName": "Scan for Secrets" }, { nalysislogs.PublishSecurityAnalysisLogs@2", "continueOnError": true, "displayName": "Publish Secret Scan Logs to Build Artifacts" }, { "task": "securedevelopmentteam.vss-secure-development-tools.build ➥ -task-postanalysis.PostAnalysis@1", "inputs": { "CredScan": true, "ToolLogsNotFoundAction": "Error" }, "displayName": "Check for Failures" } ], "displayName": "${{ parameters.displayName }}" } ] }
It works! We now have some JSON based on a YAML file. This is a useful exercise, as there are many different DevOps tools out in the wild that accept YAML or JSON (AutoRest, Kubernetes, to name a couple). As a result, you might prefer YAML, but your coworker prefers JSON. Now you have an easy way to share with each other by converting it this way.
Now, we freely admit that this isn’t a complicated task. But the task itself isn’t the point of this chapter. The point is how we figured it out. What did we do?
-
We started by searching the local help files for any that contained a specific keyword. When our search term didn’t match a command name, PowerShell did a full search of all the help files’ contents. That’s useful, because if a file had even mentioned YAML, we’d have found it.
-
We moved on to searching for specific command names. This would help find commands for which no help files were installed. Ideally, commands should always have help files, but we don’t live in an ideal world, so we always take this extra step.
-
Finding nothing locally, we searched PowerShell Gallery and found a promising-looking module. We installed the module and reviewed its commands.
-
Even though the module author didn’t provide help, PowerShell helped us out and we were able to figure out how to run the commands to convert to and from YAML. This helps us see how the commands’ data is structured and what sorts of values the commands are expecting.
-
Using the information we had gathered to that point, we were able to implement the change we wanted.
9.4 Tips for teaching yourself
Again, the real point of this book is to teach you how to teach yourself—and that’s what this chapter illustrates. Here are a few tips:
-
Don’t be afraid of the help, and be sure to read the examples. We say that over and over, and it’s like nobody believes us. We still see newcomers, right in front of us, secretly going to Google to find examples. What’s so scary about the help files? If you’re willing to read someone’s blog, why not give the examples in the help files a shot first?
-
Pay attention. Every bit of information on the screen is potentially important—don’t mentally skip over the stuff that you’re not immediately looking for. That’s easy to do, but don’t. Instead, look at each thing, and try to figure out what it’s for and what information you can derive from it.
-
Don’t be afraid to fail. Hopefully you have a virtual machine that you can play in—so use it. Newcomers are constantly asking us questions like, “Hey, if I do such and such, what will happen?” to which we’ve started replying, “No idea—try it.” Experimentation is good. In a virtual machine, the worst that can happen is you have to roll back to a snapshot, right? So give it a whirl, whatever you’re working on.
-
If one thing doesn’t work, don’t bang your head against a wall—try something else.
Everything gets easier with time, patience, and practice—but be sure that you’re thinking along the way.
9.5 Lab
Note For this lab, you can use any OS you prefer (try it on multiple ones if you would like), and make sure you are using PowerShell 7.1 or higher.
Now it’s your turn. We’re assuming that you’re working in a virtual machine or other machine that is okay to mess up a little in the name of learning. Please don’t do this in a production environment on a mission-critical computer!
This exercise will be about secrets management. DevOps engineers should be very familiar with this concept. The idea is simple: We have a bunch of sensitive information (passwords, connection strings, etc.) that we need to use in our commands, but we need to keep those secrets in a secure location. We also may want to share those secrets with other members on our team—and an email is not secure enough, folks!
The PowerShell team has recently been working on a module called Secrets Management to do just this. It’s a generic module for interacting with any secret store that supports it. Some will be local secret stores like the macOS Keychain, and others will be cloud services like Azure Key Vault and HashiCorp Vault. Your goal is to grab this module, store a secret in your secret store of choice, and then retrieve it. If you use a cloud-based secret store, try to retrieve the secret from a different machine as the ultimate test.
9.6 Lab answer
We’re sure you were expecting us to give you a full list of commands to run to complete this lab exercise. However, this entire chapter has been about figuring these things out on your own. The secrets management module is well documented. If you have been following along with us, then you will not have any problems with this lab.