14 Multitasking with background jobs
Everyone’s always telling you to multitask, right? Why shouldn’t PowerShell help you with that by doing more than one thing at a time? It turns out that PowerShell can do exactly that, particularly for longer-running tasks that might involve multiple target computers. Make sure you’ve read chapter 13 before you dive into this chapter, because we’ll be taking those remoting concepts a step further.
Heads Up We will be using a lot of the Az cmdlets in this chapter, which does require an active Azure subscription. These are just the examples we chose to highlight.
14.1 Making PowerShell do multiple things at the same time
You should think of PowerShell as a single-threaded application, meaning that it can do only one thing at a time. You type a command, you press Enter, and the shell waits for that command to execute. You can’t run a second command until the first command finishes.
But with its background jobs functionality, PowerShell has the ability to move a command onto a separate background thread or a separate background PowerShell process. That enables the command to run in the background as you continue to use the shell for another task. You have to make that decision before running the command; after you press Enter, you can’t decide to move a long-running command into the background.
After commands are in the background, PowerShell provides mechanisms to check on their status, retrieve any results, and so forth.
14.2 Synchronous vs. asynchronous
Let’s get some terminology out of the way first. PowerShell runs normal commands synchronously, meaning you press Enter and then wait for the command to complete. Moving a job into the background allows it to run asynchronously, meaning you can continue to use the shell for other tasks as the command completes. Let’s look at some important differences between running commands in these two ways:
-
When you run a command synchronously, you can respond to input requests. When you run commands in the background, there’s no opportunity to see input requests—in fact, they’ll stop the command from running.
-
Synchronous commands produce error messages when something goes wrong. Background commands produce errors, but you won’t see them immediately. You’ll have to make arrangements to capture them, if necessary. (Chapter 24 discusses how you do that.)
-
If you omit a required parameter on a synchronous command, PowerShell can prompt you for the missing information. On a background command, it can’t, so the command will fail.
-
The results of a synchronous command start displaying as soon as they become available. With a background command, you wait until the command finishes running and then retrieve the cached results.
We typically run commands synchronously to test them out and get them working properly, and run them in the background only after we know they’re fully debugged and working as we expect. We follow these measures to ensure that the command will run without problems and that it will have the best chance of completing in the background. PowerShell refers to background commands as jobs. You can create jobs in several ways, and you can use several commands to manage them.
14.3 Creating a process job
The first type of job we cover is perhaps the easiest: a process job. This is a command that runs in another PowerShell process on your machine in the background.
To launch one of these jobs, you use the Start-Job command. A -ScriptBlock parameter lets you specify the command (or commands) to run. PowerShell makes up a default job name (Job1, Job2, etc.), or you can specify a custom job name by using the -Name parameter. Rather than specifying a script block, you can specify the -FilePath parameter to have the job execute an entire script file full of commands. Here’s a simple example:
PS /Users/travisp/> start-job -scriptblock { gci } Id Name PSJobTypeName State HasMoreData Location Command -- ---- ------------- ----- ----------- -------- ------- 1 Job1 BackgroundJob Running True localhost gci
The command creates the job object, and as the previous example shows, the job begins running immediately. The job is also assigned a sequential job ID number, which is shown in the table.
The command also has a -WorkingDirectory parameter that allows you to change where your job starts on the filesystem. By default, it always starts in the home directory. Don’t ever make assumptions about file paths from within a background job: use absolute paths to make sure you can refer to whatever files your job command may require, or use the -WorkingDirectory parameter. Here’s an example:
PS /Users/travisp/> start-job -scriptblock { gci } -WorkingDirectory /tmp Id Name PSJobTypeName State HasMoreData Location Command -- ---- ------------- ----- ----------- -------- ------- 3 Job3 BackgroundJob Running True localhost gci
Sharp-eyed readers will note that the first job we created is named Job1 and given the ID 1, but the second job is Job3 with ID 3. It turns out that every job has at least one child job, and the first child job (a child of Job1) is given the name Job2 and the ID 2. We’ll get to child jobs later in this chapter.
Here’s something to keep in mind: although process jobs run locally, they do require PowerShell remoting to be enabled, which we covered in chapter 13.
14.4 Creating a thread job
There’s a second type of job that ships as part of PowerShell that we’d like to talk about. It’s called a thread job. Rather than running in a totally different PowerShell process, a thread job will spin up another thread in the same process. Here’s an example:
PS /Users/travisp/> start-threadjob -scriptblock { gci } Id Name PSJobTypeName State HasMoreData Location Command -- ---- ------------- ----- ----------- -------- ------- 1 Job1 ThreadJob Running False PowerShell gci
Looks very similar to the previous job output, huh? Only two differences—the PSJobTypeName, which is ThreadJob, and the Location, which is PowerShell. This tells us that this job ran within the process that we’re currently using, but in a different thread.
Since the overhead of spinning up a new thread is drastically faster than spinning up a new process, thread jobs are fantastic for short-term scripts and commands that you want to start fast and run in the background. Inversely, you can use process jobs for long-running scripts on your machine.
Heads Up Although thread jobs start faster, keep in mind that one process can only have so many threads running at the same time before it starts to slow down. PowerShell baked in a “throttle limit” of 10 to help prevent you from bogging down PowerShell too much. This means that only 10 thread jobs can run at the same time. If you want to up the limit, you can. Just specify the -ThrottleLimit parameter and pass in the new limit you want to use. You’ll eventually start seeing diminishing returns if you start 50, 100, 200 thread jobs at a time. Keep that in mind.
14.5 Remoting, as a job
Let’s review the final technique you can use to create a new job: PowerShell’s remoting capabilities, which you learned about in chapter 13. There’s an important difference: whatever command you specify in the -scriptblock (or -command, which is an alias for the same parameter) will be transmitted in parallel to each computer you specify. Up to 32 computers can be contacted at once (unless you modify the -throttleLimit parameter to allow more or fewer), so if you specify more than 32 computer names, only the first 32 will start. The rest will start after the first set begins to finish, and the top-level job will show a completed status after all of the computers finish.
Unlike the other two ways to start a job, this technique requires you to have PowerShell v6 or higher installed on each target computer and remoting over SSH to be enabled in PowerShell on each target computer. Because the command physically executes on each remote computer, you’re distributing the computing workload, which can help improve performance for complex or long-running commands. The results come back to your computer and are stored with the job until you’re ready to review them.
In the following example, you’ll also see the -JobName parameter that lets you specify a job name other than the boring default:
PS C:> invoke-command -command { get-process } -hostname (get-content .allservers.txt ) -asjob -jobname MyRemoteJob WARNING: column "Command" does not fit into the display and was removed. Id Name State HasMoreData Location -- ---- ----- ----------- -------- 8 MyRemoteJob Running True server-r2,lo...
14.6 Jobs in the wild
We wanted to use this section to show an example of a PowerShell module that exposes its own PSJobs so you can look out for this pattern in your PowerShell journey. Let’s take the command New-AzVm, for example:
PS /Users/travisp/> gcm New-AzVM -Syntax New-AzVM -Name <string> -Credential <pscredential> [-ResourceGroupName <string>] [-Location <string>] [-Zone <string[]>] [-VirtualNetworkName <string>] [-AddressPrefix <string>] [-SubnetName <string>] [-SubnetAddressPrefix <string>] [-PublicIpAddressName <string>] [-DomainNameLabel <string>] [-AllocationMethod <string>] [-SecurityGroupName <string>] [-OpenPorts <int[]>] [-Image <string>] [-Size <string>] [-AvailabilitySetName <string>] [-SystemAssignedIdentity] [-UserAssignedIdentity <string>] [-AsJob] [-DataDiskSizeInGb <int[]>] [-EnableUltraSSD] [-ProximityPlacementGroup <string>] [-HostId <string>] [-DefaultProfile <IAzureContextContainer>] [-WhatIf] [-Confirm] [<CommonParameters>] New-AzVM [-ResourceGroupName] <string> [-Location] <string> [-VM] <PSVirtualMachine> [[-Zone] <string[]>] [-DisableBginfoExtension] [-Tag <hashtable>] [-LicenseType <string>] [-AsJob] [-DefaultProfile <IAzureContextContainer>] [-WhatIf] [-Confirm] [<CommonParameters>] New-AzVM -Name <string> -DiskFile <string> [-ResourceGroupName <string>] [-Location <string>] [-VirtualNetworkName <string>] [-AddressPrefix <string>] [-SubnetName <string>] [-SubnetAddressPrefix <string>] [-PublicIpAddressName <string>] [-DomainNameLabel <string>] [-AllocationMethod <string>] [-SecurityGroupName <string>] [-OpenPorts <int[]>] [-Linux] [-Size <string>] [-AvailabilitySetName <string>] [-SystemAssignedIdentity] [-UserAssignedIdentity <string>] [-AsJob] [-DataDiskSizeInGb <int[]>] [-EnableUltraSSD] [-ProximityPlacementGroup <string>] [-HostId <string>] [-DefaultProfile <IAzureContextContainer>] [-WhatIf] [-Confirm] [<CommonParameters>]
Notice a familiar parameter? -AsJob! Let’s see what it does in this command:
PS /Users/travisp/> Get-Help New-AzVM -Parameter AsJob -AsJob <System.Management.Automation.SwitchParameter> Run cmdlet in the background and return a Job to track progress. Required? false Position? named Default value False Accept pipeline input? False Accept wildcard characters? false
This parameter tells New-AzVM to return a Job. If we fire off that cmdlet, after we put in a username and password for the VM, we’ll see that we get a Job back.
PS /Users/travisp/> New-AzVm -Name myawesomevm -Image UbuntuLTS -AsJob cmdlet New-AzVM at command pipeline position 1 Supply values for the following parameters: Credential User: azureuser Password for user azureuser: *********** Id Name PSJobTypeName State HasMoreData Location Command -- ---- ------------- ----- ----------- -------- ------- 8 Long Running O... AzureLongRunni... Running True localhost New-AzVM
What makes this so awesome is that you can manage these jobs just as you would the jobs that were returned from Start-Job or Start-ThreadJob. You’ll see later how we go about managing jobs, but this is an example of how custom jobs might appear. Look for the -AsJob parameter!
14.7 Getting job results
The first thing you’ll probably want to do after starting a job is to check whether your job has finished. The Get-Job cmdlet retrieves every job currently defined by the system and shows you each one’s status:
PS /Users/travisp/> get-job Id Name PSJobTypeName State HasMoreData Location Command -- ---- ------------- ----- ----------- -------- ------- 1 Job1 BackgroundJob Completed True localhost gci 3 Job3 BackgroundJob Completed True localhost gci 5 Job5 ThreadJob Completed True PowerShell gci 8 Job8 BackgroundJob Completed True server-r2, lo... 11 MyRemoteJob BackgroundJob Completed True server-r2, lo... 13 Long Running O... AzureLongRunni... Running True localhost New-AzVM
You can also retrieve a specific job by using its ID or its name. We suggest that you do that and pipe the results to Format-List *, because you’ve gathered some valuable information:
PS /Users/travisp/> get-job -id 1 | format-list * State : Completed HasMoreData : True StatusMessage : Location : localhost Command : gci JobStateInfo : Completed Finished : System.Threading.ManualResetEvent InstanceId : e1ddde9e-81e7-4b18-93c4-4c1d2a5c372c Id : 1 Name : Job1 ChildJobs : {Job2} PSBeginTime : 12/12/2019 7:18:58 PM PSEndTime : 12/12/2019 7:18:58 PM PSJobTypeName : BackgroundJob Output : {} Error : {} Progress : {} Verbose : {} Debug : {} Warning : {} Information : {}
Try it Now If you’re following along, keep in mind that your job IDs and names might be different from ours. Focus on the output of Get-Job to retrieve your job IDs and names, and substitute yours in the examples. Also keep in mind that Microsoft has expanded the job object over the last few PowerShell versions, so your output when looking at all properties might be different.
The ChildJobs property is one of the most important pieces of information, and we’ll cover it in a moment. To retrieve the results from a job, use Receive-Job. But before you run this, you need to know a few things:
-
You have to specify the job from which you want to receive results. You can do this by job ID or job name, or by getting jobs with
Get-Joband piping them toReceive-Job. -
If you receive the results of the parent job, those results will include all output from all child jobs. Alternatively, you can choose to get the results from one or more child jobs.
-
Typically, receiving the results from a job clears them out of the job output cache, so you can’t get them a second time. Specify
-keepto keep a copy of the results in memory. Or you can output the results to a CLIXML file, if you want to retain a copy to work with. -
The job results may be deserialized objects, which you learned about in chapter 13. These are snapshots from the point in time when they were generated, and they may not have any methods that you can execute. But you can pipe the job results directly to cmdlets such as
Sort-Object,-Format-List,Export-CSV,ConvertTo-HTML,Out-File, and so on, if desired.
PS /Users/travisp/> receive-job -id 1 Directory: /Users/travisp Mode LastWriteTime Length Name ---- ------------- ------ ---- d---- 11/24/2019 10:53 PM Code d---- 11/18/2019 11:23 PM Desktop d---- 9/15/2019 9:12 AM Documents d---- 12/8/2019 11:04 AM Downloads d---- 9/15/2019 7:07 PM Movies d---- 9/15/2019 9:12 AM Music d---- 9/15/2019 6:51 PM Pictures d---- 9/15/2019 9:12 AM Public
The preceding output shows an interesting set of results. Here’s a quick reminder of the command that launched this job in the first place:
When we received the results from Job1, we didn’t specify -keep. If we try to get those same results again, we’ll get nothing, because the results are no longer cached with the job:
Here’s how to force the results to stay cached in memory:
PS /Users/travisp/> receive-job -id 3 –keep Directory: /Users/travisp Mode LastWriteTime Length Name ---- ------------- ------ ---- d---- 11/24/2019 10:53 PM Code d---- 11/18/2019 11:23 PM Desktop d---- 9/15/2019 9:12 AM Documents d---- 12/8/2019 11:04 AM Downloads d---- 9/15/2019 7:07 PM Movies d---- 9/15/2019 9:12 AM Music d---- 9/15/2019 6:51 PM Pictures d---- 9/15/2019 9:12 AM Public
You’ll eventually want to free up the memory that’s being used to cache the job results, and we’ll cover that in a bit. But first, let’s look at a quick example of piping the job results directly to another cmdlet:
PS /Users/travisp> receive-job -name myremotejob | sort-object PSComputerName ➥ | Format-Table -groupby PSComputerName PSComputerName: localhost NPM(K) PM(M) WS(M) CPU(s) Id ProcessName PSComputerName ------ ----- ----- ------ -- ----------- -------------- 0 0 56.92 0.70 484 pwsh localhost 0 0 369.20 70.17 1244 Code localhost 0 0 71.92 0.20 3492 pwsh localhost 0 0 288.96 15.31 476 iTerm2 localhost
This was the job we started by using Invoke-Command. The cmdlet has added the PSComputerName property so we can keep track of which object came from which computer. Because we retrieved the results from the top-level job, this includes all of the computers we specified, which allows this command to sort them on the computer name and then create an individual table group for each computer. Get-Job can also keep you informed about which jobs have results remaining:
PS /Users/travisp> get-job Id Name PSJobTypeName State HasMoreData Location Command -- ---- ------------- ----- ----------- -------- ------- 1 Job1 BackgroundJob Completed False localhost gci 3 Job3 BackgroundJob Completed True localhost gci 5 Job5 ThreadJob Completed True PowerShell gci 8 Job8 BackgroundJob Completed True server-r2, lo... 11 MyRemoteJob BackgroundJob Completed False server-r2, lo... 13 Long Running O... AzureLongRunni... Running True localhost New-AzVM
The HasMoreData column will be False when no output is cached with that job. In the case of Job1 and MyRemoteJob, we’ve already received those results and didn’t specify -keep at that time.
14.8 Working with child jobs
We mentioned earlier that most jobs consist of one top-level parent job and at least one child job. Let’s look at a job again:
PS /Users/travisp> get-job -id 1 | format-list * State : Completed HasMoreData : True StatusMessage : Location : localhost Command : dir JobStateInfo : Completed Finished : System.Threading.ManualResetEvent InstanceId : e1ddde9e-81e7-4b18-93c4-4c1d2a5c372c Id : 1 Name : Job1 ChildJobs : {Job2} PSBeginTime : 12/27/2019 2:34:25 PM PSEndTime : 12/27/2019 2:34:29 PM PSJobTypeName : BackgroundJob Output : {} Error : {} Progress : {} Verbose : {} Debug : {} Warning : {} Information : {}
Try it Now Don’t follow along for this part, because if you’ve been following along up to now, you’ve already received the results of Job1. If you’d like to try this, start a new job by running Start-Job -script { dir }, and use that new job’s ID instead of the ID number 1 we used in our example.
You can see that Job1 has a child job, Job2. You can get it directly now that you know its name:
PS /Users/travisp> get-job -name job2 | format-list * State : Completed StatusMessage : HasMoreData : True Location : localhost Runspace : System.Management.Automation.RemoteRunspace Debugger : System.Management.Automation.RemotingJobDebugger IsAsync : True Command : dir JobStateInfo : Completed Finished : System.Threading.ManualResetEvent InstanceId : a21a91e7-549b-4be6-979d-2a896683313c Id : 2 Name : Job2 ChildJobs : {} PSBeginTime : 12/27/2019 2:34:25 PM PSEndTime : 12/27/2019 2:34:29 PM PSJobTypeName : Output : {Applications, Code, Desktop, Documents, Downloads, Movies, ➥ Music...} Error : {} Progress : {} Verbose : {} Debug : {} Warning : {} Information : {}
Sometimes a job has too many child jobs to list in that form, so you may want to list them a bit differently, as follows:
PS /Users/travisp> get-job -id 1 | select-object -expand childjobs Id Name PSJobTypeName State HasMoreData Location Command -- ---- ------------- ----- ----------- -------- ------- 2 Job2 Completed True localhost gci
This technique creates a table of the child jobs for job ID 1, and the table can be whatever length it needs to be to list them all. You can receive the results from any individual child job by specifying its name or ID with Receive-Job.
14.9 Commands for managing jobs
Jobs also use three more commands. For each of these, you can specify a job either by giving its ID, giving its name, or getting the job and piping it to one of these cmdlets:
-
Remove-Job—This deletes a job, and any output still cached with it, from memory. -
Stop-Job—If a job seems to be stuck, this command terminates it. You can still receive whatever results were generated to that point. -
Wait-Job—This is useful if a script is going to start a job or jobs and you want the script to continue only when the job is done. This command forces the shell to stop and wait until the job (or jobs) is completed, and then allows the shell to continue.
For example, to remove the jobs that we’ve already received output from, we’d use the following command:
PS /Users/travisp> get-job | where { -not $_.HasMoreData } | remove-job PS /Users/travisp> get-job Id Name PSJobTypeName State HasMoreData Location Command -- ---- ------------- ----- ----------- -------- ------- 3 Job3 BackgroundJob Completed True localhost gci 5 Job5 ThreadJob Completed True PowerShell gci 8 Job8 BackgroundJob Completed True server-r2, lo... 13 Long Running O... AzureLongRunni... Completed True localhost New-AzVM
Jobs can also fail, meaning that something went wrong with their execution. Consider this example:
PS /Users/travisp> invoke-command -command { nothing } -hostname notonline -asjob -jobname ThisWillFail Id Name PSJobTypeName State HasMoreData Location Command -- ---- ------------- ----- ----------- -------- ------- 11 ThisWillFail BackgroundJob Failed False notonline nothing
Here, we started a job with a bogus command and targeted a nonexistent computer. The job immediately failed, as shown in its status. We don’t need to use Stop-Job at this point; the job isn’t running. But we can get a list of its child jobs:
PS /Users/travisp> get-job -id 11 | format-list * State : Failed HasMoreData : False StatusMessage : Location : notonline Command : nothing JobStateInfo : Failed Finished : System.Threading.ManualResetEvent InstanceId : d5f47bf7-53db-458d-8a08-07969305820e Id : 11 Name : ThisWillFail ChildJobs : {Job12} PSBeginTime : 12/27/2019 2:45:12 PM PSEndTime : 12/27/2019 2:45:14 PM PSJobTypeName : BackgroundJob Output : {} Error : {} Progress : {} Verbose : {} Debug : {} Warning : {} Information : {}
And we can then get that child job:
PS /Users/travisp> get-job -name job12 Id Name PSJobTypeName State HasMoreData Location Command -- ---- ------------- ----- ----------- -------- ------- 12 Job12 Failed False notonline nothing
As you can see, no output was created for this job, so you won’t have any results to retrieve. But the job’s errors are stored in the results, and you can get them by using Receive-Job:
PS /Users/travisp> receive-job -name job12 OpenError: [notonline] The background process reported an error with the ➥ following message: The SSH client session has ended with error message: ➥ ssh: Could not resolve hostname notonline: nodename nor servname provided, ➥ or not known.
The full error is much longer; we truncated it here to save space. You’ll notice that the error includes the hostname that the error came from, [notonline]. What happens if only one of the computers can’t be reached? Let’s try:
PS /Users/travisp> invoke-command -command { nothing } -computer notonline,server-r2 -asjob -jobname ThisWilLFail Id Name PSJobTypeName State HasMoreData Location Command -- ---- ------------- ----- --------- -------- ------- 13 ThisWillFail BackgroundJob Running True notonline,lo... nothing
After waiting for a bit, we run the following:
PS /Users/travisp> get-job 13 Id Name PSJobTypeName State HasMoreData Location Command -- ---- ------------- ----- ----------- -------- ------- 13 ThisWillFail BackgroundJob Failed False notonline,lo... nothing
The job still fails, but let’s look at the individual child jobs:
PS /Users/travisp> get-job -id 13 | select -expand childjobs Id Name PSJobTypeName State HasMoreData Location Command -- ---- ------------- ----- ----------- -------- ------- 14 Job14 Failed False notonline nothing 15 Job15 Failed False localhost nothing
Okay, they both fail. We have a feeling we know why Job14 doesn’t work, but what’s wrong with Job15?
PS /Users/travisp> receive-job -name job15 Receive-Job : The term 'nothing' is not recognized as the name of a cmdlet , function, script file, or operable program. Check the spelling of the na me, or if a path was included, verify that the path is correct and try aga in.
Ah, that’s right, we told it to run a bogus command. As you can see, each child job can fail for different reasons, and PowerShell tracks each one individually.
14.10 Common points of confusion
Jobs are usually straightforward, but we’ve seen folks do one thing that causes confusion. Don’t do this:
Doing so starts up a temporary connection to Server-R2 and starts a local job. Unfortunately, that connection immediately terminates, so you have no way to reconnect and retrieve that job. In general, then, don’t mix and match the three ways of starting jobs. The following is also a bad idea:
That’s completely redundant; keep the Invoke-Command section and use the -AsJob parameter to have it run in the background.
Less confusing, but equally interesting, are the questions new users often ask about jobs. Probably the most important of these is, “Can we see jobs started by someone else?” The answer is no. Jobs and thread jobs are contained entirely within the PowerShell process, and although you can see that another user is running PowerShell, you can’t see inside that process. It’s like any other application: you can see that another user is running Microsoft Word, for example, but you can’t see what documents that user is editing, because those documents exist entirely inside of Word’s process.
Jobs last only as long as your PowerShell session is open. After you close it, any jobs defined within it disappear. Jobs aren’t defined anywhere outside PowerShell, so they depend on its process continuing to run in order to maintain themselves.
14.11 Lab
The following exercises should help you understand how to work with various types of jobs and tasks in PowerShell. As you work through these exercises, don’t feel you have to write a one-line solution. Sometimes it’s easier to break things down into separate steps.
-
Create a one-time thread job to find all the text files (
*.txt) on the filesystem. Any task that might take a long time to complete is a great candidate for a job. -
You realize it would be helpful to identify all text files on some of your servers. How would you run the same command from task 1 on a group of remote computers?
-
What cmdlet would you use to get the results of a job, and how would you save the results in the job queue?
14.12 Lab answers
-
Of course, you would use whatever job ID was applicable or the job name.