3

Working with Objects in PowerShell

Everything we do in PowerShell revolves around working with objects. Objects, in PowerShell, may have properties or methods (or both). It is difficult to describe an object without resorting to this; an object is a representation of a thing or item of data. Let's use an analogy to attempt to give meaning to the term.

A book is an object and has properties that describe its physical characteristics, such as the number of pages, the weight, or size. It has metadata (information about data) properties that describe the author, the publisher, the table of contents, and so on.

The book might also have methods. A method affects the state of an object. For example, there might be methods to open or close the book or methods to jump to different chapters. A method might also convert an object into a different format. For example, there might be a method to copy a page, or even destructive methods such as one to split the book.

PowerShell has a variety of commands that work with arrays (or collections) of objects in a pipeline.

In this chapter, we are going to cover the following topics:

• Pipelines
• Members
• Enumerating and filtering
• Selecting and sorting
• Grouping and measuring
• Comparing
• Importing, exporting, and converting

Pipelines

The pipeline is one of the most prominent features of PowerShell. The pipeline is used to send output from one command to another command as input.

Most of the output from a command is sent to what is known as standard output, often shortened to stdout.

Standard output

The term standard output is used because there are different kinds of output. Each of these different types of output is sent to a different stream, allowing each to be read separately. In PowerShell, the streams are Standard, Error, Warning, Verbose, Debug, and Information.

When assigning the output of a command to a variable, the assigned value is taken from the standard output (the output stream) of a command. For example, the following command assigns the data from the standard output to a variable:

$computerSystem = Get-CimInstance -ClassName Win32_ComputerSystem 

Non-standard output, such as Verbose, will not be assigned to the variable.

Non-standard output

In PowerShell, each of the streams has a command associated with it:

Table 3.1: Streams and their associated commands

In PowerShell 5 and later, the Write-Host command is a wrapper for Write-Information. It sends output to the information stream.

Prior to Windows PowerShell 5, Write-Host did not have a dedicated stream; the output could only be captured via a transcript, that is, by using the Start-Transcript command to log console output to a file.

For example, if you add the Verbose switch to the preceding command, more information is shown. This extra information is not held in the variable; it is sent to a different stream:

PS> $computerSystem = Get-CimInstance Win32_ComputerSystem -Verbose
VERBOSE: Perform operation 'Enumerate CimInstances' with following parameters, ''namespaceName' = rootcimv2,'className' = Win32_ComputerSystem'.
VERBOSE: Operation 'Enumerate CimInstances' complete.
PS> $computerSystem
Name    PrimaryOwnerName    Domain       TotalPhysicalMemory    Model
----    ----------------    ------       -------------------    -----
NAME    Username            WORKGROUP    17076875264            Model

The object pipeline

Languages such as Batch scripting (on Windows) or tools on Linux and Unix often use a pipeline to pass text between commands. When the output from one command is piped to another, it is up to the next command in the pipeline to figure out what the text from the input pipeline means.

PowerShell, on the other hand, sends objects from one command to another when using the pipeline.

The pipe (|) symbol is used to send the standard output between commands.

In the following example, the output of Get-Process is sent to the Where-Object command, which applies a filter. The filter restricts the list of processes to those that are using more than 50MB of memory:

Get-Process | Where-Object WorkingSet64 -gt 50MB

When piping commands, a line break may be added after a pipe:

Get-Process |
    Where-Object WorkingSet64 -gt 50MB |
    Select-Object -Property Name, ID

Or, in PowerShell 7, the pipe may be placed at the start of the following line within a script or script block. To demonstrate in the console, hold Shift and press Return at the end of each line; the prompt will change to >> for each subsequent line:

Get-Process
    | Where-Object WorkingSet -gt 50MB
    | Select-Object Name, ID

When the pipe is placed at the end of a line, the command can be pasted or typed in the console as-is. When using a pipe at the beginning of a new line, Shift + Return must be used in the console to add the new line without ending the code block.

No special action is required in a script or function to use the pipe at the beginning of the line.

Members

At the beginning of this chapter, the idea of properties and methods was introduced. These are part of a set of items collectively known as members. These members are used to interact with an object. A few of the more frequently used members are NoteProperty, ScriptProperty, ScriptMethod, and Event.

This chapter focuses on the property members Property, NoteProperty, and ScriptProperty.

The Get-Member command

The Get-Member command can be used to view the different members of an object. For example, it can be used to list the members of a process object returned by Get-Process. The automatic variable $PID holds the process ID of the current PowerShell process:

Get-Process -Id $PID | Get-Member 

Get-Member offers filters using its parameters (MemberType, Static, and View). For example, to view the properties of the PowerShell process object, the following can be used:

Get-Process -Id $PID | Get-Member -MemberType Property 

The Static parameter is covered in Chapter 7, Working with .NET.

The View parameter is set to All by default. It has three additional values:

• Base: This shows properties that are derived from a .NET object
• Adapted: This shows members handled by PowerShell's Adapted Type System (ATS)
• Extended: This shows members added by PowerShell's Extended Type System (ETS)

Accessing object properties

The properties of an object in PowerShell may be accessed by writing the property name after a period. For example, the Name property of the current PowerShell process may be accessed by using the following code:

$process = Get-Process -Id $PID 
$process.Name 

PowerShell also allows us to access these properties by enclosing a command in parentheses:

(Get-Process -Id $PID).Name 

The properties of an object are objects themselves. For example, the StartTime property of a process is a DateTime object. The DayOfWeek property may be accessed using the following code:

$process = Get-Process -Id $PID 
$process.StartTime.DayOfWeek 

The variable assignment step may be skipped if parentheses are used:

(Get-Process -Id $PID).StartTime.DayOfWeek 

If a property name has a space, it may be accessed using a number of different notation styles. For example, a property named 'Some Name' may be accessed by quoting the name or enclosing the name in curly braces:

$object = [PSCustomObject]@{ 'Some Name' = 'Value' } 
$object."Some Name" 
$object.'Some Name' 
$object.{Some Name} 

A variable may also be used to describe a property name:

PS> $object = [PSCustomObject]@{ 'Some Name' = 'Value' }
PS> $propertyName = 'Some Name'
PS> $object.$propertyName
Value

The ability to assign a property or assign a new value to a property is governed by the access modifiers for a property.

Access modifiers

An access modifier for a property has two possible values: Get, indicating that the property can be read; and Set, indicating that the property can be written to (changed).

Depending on the type of object, properties may be read-only or read/write. These may be identified using Get-Member and by inspecting the access modifiers.

In the following example, the values in curly braces at the end of each line is the access modifier:

PS> $File = New-Item NewFile.txt 
PS> $File | Get-Member -MemberType Property
    TypeName: System.IO.FileInfo
Name              MemberType    Definition 
----              ----------    ---------- 
Attributes        Property      System.IO.FileAttributes Attributes {get;set;}
CreationTime      Property      datetime CreationTime {get;set;} 
CreationTimeUtc   Property      datetime CreationTimeUtc {get;set;} 
Directory         Property      System.IO.DirectoryInfo Directory {get;} 
DirectoryName     Property      string DirectoryName {get;} 
Exists            Property      bool Exists {get;} 

When the modifier is {get;}, the property value is read-only; attempting to change the value results in an error:

PS> $File = New-Item NewFile.txt -Force
PS> $File.Name = 'NewName'
InvalidOperation: 'Name' is a ReadOnly property.

When the modifier is {get;set;}, the property value may be read and changed. In the preceding example, CreationTime has the set access modifier. The value can be changed; in this case, it may be set to any date after January 1, 1601:

$File = New-Item NewFile.txt -Force
$File.CreationTime = Get-Date -Day 1 -Month 2 -Year 1692 

The result of the preceding command can be seen by reviewing the properties for the file in PowerShell:

Get-Item NewFile.txt | Select-Object -ExpandProperty CreationTime

Alternatively, you can use File Explorer to view a file's properties, as shown in Figure 3.1:

Figure 3.1: Changing the Created date

In the preceding example, the change made to CreationTime is passed from the object representing the file to the file itself. The object used here, based on the .NET class System.IO.FileInfo, is written in such a way that it supports the change. A property may indicate that it can be changed (by supporting the set access modifier in Get-Member) and still not pass the change back to whatever the object represents.

Using methods

Methods effect a change in state. That may be a change to the object associated with the method, or it may take the object and convert it into something else.

Methods are called using the following notation in PowerShell:

<Object>.Method()

When a method is called without parentheses, PowerShell will show the overload definitions. Overloading a method is a .NET concept; it comes into play when two or more methods share the same name but have different arguments and implementations:

PS> 'thisString'.Substring
OverloadDefinitions
-------------------
string Substring(int startIndex)
string Substring(int startIndex, int length)

An example of a method that takes an object and converts it into something else is shown here. In this case, a date is converted into a string:

PS> $date = Get-Date -Day 1 -Month 1 -Year 2010
PS> $date.ToLongDateString()
01 January 2010

The string generated in the preceding command is dependent on the current culture in use by PowerShell. In the preceding example, the culture is set to en-GB (Great Britain).

If a method expects to have arguments (or parameters), the notation becomes the following:

<Object>.Method(Argument1, Argument2)

An example of a method that takes arguments might be the ToString method on a DateTime object. Get-Date can be used to create a DateTime object using the current date:

PS> (Get-Date).ToString('u')
2016-12-08 21:18:49Z

In the preceding example, the letter u is one of the standard date and time format strings (https://docs.microsoft.com/dotnet/standard/base-types/standard-date-and-time-format-strings) and represents a universal sortable date/time pattern. The same result may be achieved by using the Format parameter of Get-Date:

PS> Get-Date -Format u
2016-12-08 21:19:31Z

The advantage that this method on a DateTime object has over the parameter for Get-Date is that the date can be adjusted before conversion by using some of the other properties and methods:

(Get-Date).Date.AddDays(-1).ToString('u')

The result of this command is the start of yesterday (midnight, one day before today).

An example of a method that changes a state might be seen on a tcpClient object. TCP connections must be opened before data can be sent over a network. The following example creates a tcpClient object, then opens a connection to the RPC Endpoint Mapper on the current computer:

$tcpClient = New-Object System.Net.Sockets.TcpClient
$tcpClient.Connect("127.0.0.1", 135)

The Connect method does not return anything, although it will throw an error if the connection fails. The method affects the state of the object and is reflected by the Connected property:

PS> $tcpClient.Connected
True

The state of the object may be changed again by calling the Close method to disconnect:

$tcpClient.Close()

The Add-Member command

Add-Member allows new members to be added to existing objects. This can be useful if the object type must be preserved. Select-Object can also add new members, but it changes the object type when doing so.

Starting with an empty object, it is possible to add new properties:

PS> $empty = New-Object Object
PS> $empty |
>>     Add-Member -NotePropertyName New -NotePropertyValue 'Hello world'
PS> $empty
New
---
Hello world

Add-Member may also add a ScriptProperty or a ScriptMethod member. When writing script-based properties and methods, the reserved variable $this is used to refer to itself:

PS> $empty = New-Object Object
PS> $empty |
>>     Add-Member -NotePropertyName New -NotePropertyValue 'Hello world'
PS> $empty |
>>     Add-Member -Name Calculated -MemberType ScriptProperty -Value {
>>         $this.New.Length
>>     }
PS> $empty
New            Calculated
---            ----------
Hello world            11

Methods may be added as well, for example, a method to replace the word world in the new property:

$empty = New-Object Object
$empty | Add-Member -NotePropertyName New -NotePropertyValue 'Hello world'
$params = @{
    Name       = 'Replace'
    MemberType = 'ScriptMethod'
    Value      = { $this.New -replace 'world', 'everyone' }
}
$empty | Add-Member @params

Running the newly added Replace method will show the updated string:

PS> $empty.Replace()
Hello everyone

As well as being used to present information to an end user, the properties and methods of an object are used when enumerating and filtering collections of objects.

Enumerating and filtering

Enumerating, or listing, the objects in a collection in PowerShell does not need a specialized command. For example, if the results of Get-PSDrive were assigned to a variable, enumerating the content of the variable is as simple as writing the variable name and pressing Return, allowing the values to be viewed:

PS> $drives = Get-PSDrive
PS> $drives
Name    Used (GB)    Free (GB)    Provider      Root
----    ---------    ---------    --------      ----
Alias                             Alias 
C          319.37       611.60    FileSystem    C:
Cert                              Certificate    
Env                               Environment 
...

ForEach-Object may be used to work on an existing collection or objects, or used to work on the output from another command in a pipeline.

Where-Object may be used to filter an existing collection or objects, or it may be used to filter the output from another command in a pipeline.

The ForEach-Object command

ForEach-Object is used to work on each object in an input pipeline. For example, the following command works on each of the results from Get-Process in turn by running the specified script block (enclosed in { }):

Get-Process | ForEach-Object { 
    Write-Host $_.Name -ForegroundColor Green 
} 

The script block, an arbitrary block of code, is used as an argument for the Process parameter. The preceding command may explicitly include the Process parameter name shown here:

Get-Process | ForEach-Object -Process { 
    Write-Host $_.Name -ForegroundColor Green 
}

The script block executes once for each object in the input pipeline, that is, once for each object returned by Get-Process. The special variable $_ is used to access the current object.

The variable $PSItem may be used in place of $_ if desired. There is no difference between the two variable names. $_ is the more commonly used name.

The Process parameter is accompanied by the Begin and End parameters. Begin and End are used to run script blocks before the first value is sent to the Process script block, and after the last value has been received.

Begin and End parameters

If ForEach-Object is given a single script block as an argument, it is passed to the Process parameter. The Process script block runs once for each object in the input pipeline.

ForEach-Object also supports Begin and End parameters. Begin runs once before the first value in the pipeline is passed. End runs once after the last value in the input pipeline.

The behavior of these parameters is shown in the following example:

1..5 | ForEach-Object -Begin {
    Write-Host "Starting the pipeline. Creating value."
    $value = 0
} -Process {
    Write-Host "Adding $_ to value."
    $value += $_
} -End {
    Write-Host "Finished the pipeline. Displaying value."
    $value
}

1 | ForEach-Object { "Process: $_" }

1 | ForEach-Object { "Begin" } { "Process: $_" }

1 | ForEach-Object { "Begin" } { "Process: $_" } { "End" }

The Parallel parameter

In PowerShell 7, ForEach-Object gains a Parallel parameter. This, as the name suggests, can be used to run process blocks in parallel rather than one after another.

By default, ForEach-Object runs 5 instances of the process block at a time; this is controlled by the ThrottleLimit parameter. The limit may be increased (or decreased) depending on where the bottleneck is with a given process.

Running a simple ForEach-Object command with a Start-Sleep statement shows how the output is grouped together as each set of jobs completes:

1..10 | ForEach-Object -Parallel {
    Start-Sleep -Seconds 2
    $_
}

When using ForEach-Object without the Parallel parameter, variables created before the command are accessible without any special consideration:

$string = 'Hello world'
1 | ForEach-Object {
    # The string variable can be used as normal
    $string
}

When using Parallel, each parallel script block runs in a separate thread. Variables created outside the ForEach-Object command must be accessed by prefixing the variable name with the using: scope modifier as shown here:

$string = 'Hello world'
1 | ForEach-Object -Parallel {
    # The $string variable is only accessible if using is used.
    $using:string
}

For the most part, the $using scope modifier is one-way. That is, values may be read from the scope, but new values cannot be set. For example, the following attempt to write a value to $using:newString will fail:

PS> 1 | ForEach-Object -Parallel {
>>     $using:newString = $_
>> }
ParserError:
Line |
   2 |      $using:newString = $_
     |      ~~~~~~~~~~~~~~~~
     | The assignment expression is not valid. The input to an assignment
     | operator must be an object that is able to accept assignments,
     | such as a variable or a property.

Advanced array types and hashtables can be changed; however, parentheses are required around the variable name. For example:

$values = @{}
1..5 | ForEach-Object -Parallel {
    ($using:values).Add($_, $_)
}
$values

While it is possible to store values like this, the hashtable used previously is not built to be changed from multiple threads, as it is in the example. Hashtables will be discussed in Chapter 5, Variables, Arrays, and Hashtables. Generally speaking, output from a parallel ForEach-Object should be sent to the output pipeline; for example:

$output = 1..5 | ForEach-Object -Parallel { $_ }

ForEach-Object is most often used to add complexity when processing the output from another command, or when working on a collection. ForEach-Object may also be used to read a single property, or execute a method of every object in a collection.

The MemberName parameter

ForEach-Object may also be used to get a single property or execute a single method on each of the objects. For example, ForEach-Object may be used to return only the Path property when using Get-Process:

Get-Process | ForEach-Object -MemberName Path

Or, ForEach-Object may be used to run the ToString method on a set of dates:

PS> @(
>>     Get-Date '01/01/2019'
>>     Get-Date '01/01/2020'
>> ) | ForEach-Object ToString('yyyyMMdd')
20190101
20200101

ForEach-Object offers a great deal of flexibility.

The Where-Object command

Filtering the output from commands may be performed using Where-Object. For example, processes might be filtered for those started after 9am today. If there are no processes started after 9am, then the statement will not return anything:

Get-Process | Where-Object StartTime -gt (Get-Date 9:00:00) 

The syntax shown in help for Where-Object does not quite match the syntax used here. The help text is as follows:

Where-Object [-Property] <String> [[-Value] <Object>] -GT ...

In the preceding example, we see the following:

• StartTime is the argument for the Property parameter (first argument by position)
• The comparison is greater than, as signified by the gt switch parameter
• The date (using the Get-Date command) is the argument for the Value parameter (second argument by position)

Based on that, the example might be written as follows:

Get-Process |
   Where-Object -Property StartTime -Value (Get-Date 9:00:00) -gt 

However, it is far easier to read StartTime is greater than <some date>, so most examples tend to follow that pattern.

Where-Object also accepts filters using the FilterScript parameter. FilterScript is often used to describe more complex filters, filters where more than one term is used:

Get-Service | Where-Object {
    $_.StartType -eq 'Manual' -and 
    $_.Status -eq 'Running'
}

When a filter like this is used, the conditions are evaluated in the order they are written in. This can be used to avoid conditions that may otherwise cause errors.

In the following example, Test-Path is used before Get-Item, which is used to test the last time a file was written on a remote computer (via the administrative share):

$date = (Get-Date).AddDays(-90)
'Computer1', 'Computer2' | Where-Object {
    (Test-Path "\$_c$	empfile.txt") -and
    (Get-Item "\$_c$	empfile.txt").LastWriteTime -lt $date
}

If Test-Path is removed, the snippet will throw an error if either the computer or the file does not exist.

Selecting and sorting

Select-Object acts on an input pipeline; either an existing collection of objects, or the output from another command. Select-Object can be used to select a subset of properties, to change properties, or to add new properties. Select-Object also allows a user to limit the number of objects returned.

Sort-Object can be used to perform both simple and complex sorting based on an input pipeline.

Using Select-Object is a key part of working with PowerShell, output can be customized to suit circumstances in a number of ways.

The Select-Object command

Select-Object is an extremely versatile command as shown by each of the following short examples, which demonstrate some of the simpler uses of the command.

Select-Object may be used to limit the properties returned by a command by name:

Get-Process | Select-Object -Property Name, Id 

Select-Object can limit the properties returned from a command using wildcards:

Get-Process | Select-Object -Property Name, *Memory 

Select-Object can list everything but a few properties:

Get-Process | Select-Object -Property * -ExcludeProperty *Memory* 

Select-Object can get the first few objects in a pipeline:

Get-ChildItem C: -Recurse | Select-Object -First 2

Or Select-Object can select the last few objects in a pipeline:

Get-ChildItem C: | Select-Object -Last 3 

Select-Object can Skip items at the beginning – in this example, the fifth item:

Get-ChildItem C: | Select-Object -Skip 4 -First 1 

Or it can Skip items at the end. This example returns the third from the end:

Get-ChildItem C: | Select-Object -Skip 2 -Last 1

Select-Object can get an object from a pipeline by index:

Get-ChildItem C: | Select-Object -Index 3, 4, 5

In PowerShell 7 and above, Select-Object can omit certain indexes:

Get-ChildItem C: | Select-Object -SkipIndex 3, 4, 5

Select-Object also offers a number of more advanced uses. The following sections describe calculated properties, the ExpandProperty parameter, the Unique parameter, and property sets.

Calculated properties are perhaps one of the most commonly used features of Select-Object.

Calculated properties

Select-Object can be used to add new properties to or rename existing properties on objects in an input pipeline.

Calculated properties are described using a hashtable with specific key names. The format is described in help for Select-Object. In addition to names, the following hashtable formats are acceptable for the Property parameter:

@{ Name = 'PropertyName'; Expression = { 'PropertyValue' } }
@{ Label = 'PropertyName'; Expression = { 'PropertyValue' } }
@{ n = 'PropertyName'; e = { 'PropertyValue' } }
@{ l = 'PropertyName'; e = { 'PropertyValue' } }

The expression is most often a script block, this allows other commands to be executed, mathematical operations to be performed, substitutions to be made, and so on.

If a property is being renamed, a quoted string can be used instead of the script block. The following two examples have the same result:

Get-Process | Select-Object @{ Name = 'ProcessID'; Expression = 'ID' }
Get-Process | Select-Object @{ Name = 'ProcessID'; Expression = { $_.ID } }

If the list of properties is long, it can be better to enclose the list in @(), the array operator, allowing the properties to be spread across different lines:

Get-Process | Select-Object -Property @(
    'Name'
    @{Name = 'ProcessId'; Expression = 'ID' }
    @{Name = 'FileOwner'; Expression = { (Get-Acl $_.Path).Owner }}
)

Any number of properties might be added in this manner. The preceding example includes the output from Get-Acl; if more than one property were required, ForEach-Object might be added to the command:

Get-Process | Where-Object Path | ForEach-Object {
    $acl = Get-Acl $_.Path
    Select-Object -InputObject $_ -Property @(
        'Name'
        @{Name = 'ProcessId'; Expression = 'ID' }
        @{Name = 'FileOwner'; Expression = { $acl.Owner }}
        @{Name = 'Access';    Expression = { $acl.AccessToString }}
    )
}

When Select-Object is used with the Property parameter, a new custom object is always created. If the existing object type must be preserved, Add-Member should be used instead. For example, if the object includes methods that must be accessible later in a script, the Property parameter of Select-Object should not be used.

The following example shows that the object type is preserved if the Property parameter is not used. The following command shows that the type is Process, which allows any of the methods, such as WaitForExit, to be used later on in a script:

(Get-Process | Select-Object -First 1).GetType()

If the Property parameter is used, the output is the PSCustomObject type. The resulting object will not have any of the methods specific to the Process type:

(Get-Process | Select-Object -Property Path, Company -First 1).GetType()

Calculated properties are extremely flexible, allowing an object to be modified, or a more complex object to be created with a relatively small amount of code.

The ExpandProperty parameter

The ExpandProperty parameter of Select-Object may be used to expand a single property of an object. This might be used to expand a property containing a string, leaving the output as an array of strings:

Get-Process | Select-Object -First 5 -ExpandProperty Name

If ExpandProperty is omitted, the returned object will be a PSCustomObject object with a Name property rather than the simpler array of strings.

Expanding a single property containing a string, or a numeric value, or an array of either, is the most common use of the ExpandProperty parameter.

Occasionally, it may be desirable or necessary to expand a property containing a more complex object. The members of the expanded property are added to the custom object:

Get-ChildItem $env:SYSTEMROOT*.dll |
    Select-Object -Property FullName, Length -ExpandProperty VersionInfo |
    Format-List *

Conflicting member names will cause an error to be raised; the conflicting name is otherwise ignored.

It is possible, if unusual, to use this technique to build up a single custom object based on the output from multiple commands:

Get-CimInstance Win32_ComputerSystem |
    Select-Object -Property @(
        @{n='ComputerName';e={ $_.Name }}
        'DnsHostName'
        @{n='OSInfo';e={ Get-CimInstance Win32_OperatingSystem }}
    ) |
    Select-Object * -ExpandProperty OSInfo |
    Select-Object -Property @(
        'ComputerName'
        'DnsHostName'
        @{n='OperatingSystem';e='Caption'}
        'SystemDirectory'
    )

The resulting object will contain two properties from Win32_ComputerSystem and two properties from Win32_OperatingSystem.

The Unique parameter

Select-Object returns unique values from arrays of simple values with the Unique parameter:

1, 1, 1, 3, 5, 2, 2, 4 | Select-Object -Unique

1, 1, 1, 3, 5, 2, 2, 4 | Sort-Object | Get-Unique

In the following example, an object is created with one property called Number. The value for the property is 1, 2, or 3. The result is two objects with a value of 1, two with a value of 2, and so on:

PS> 1, 2, 3, 1, 2, 3 | ForEach-Object {
>>     [PSCustomObject]@{
>>         Number = $_
>>     }
>>  }
Number
------
     1
     2
     3
     1
     2
     3

Select-Object can remove the duplicates from the set in this example using the -Unique parameter if a list of properties (or a wildcard for the properties) is set:

PS> 1, 2, 3, 1, 2, 3 |
>>    ForEach-Object {
>>        [PSCustomObject]@{
>>            Number = $_
>>        }
>>    } | 
>>    Select-Object -Property * -Unique
Number
------
     1
     2
     3 

Select-Object builds up a collection of unique objects by comparing each property of each object to every unique object that came before it in a pipeline. This allows Select-Object to work without relying on an ordered collection (as Get-Unique requires) but is consequently slower; the work of comparing increases every time a unique object is encountered.

Property sets

A property set is a pre-defined list of properties that might be used when exploring an object. The property set is stored with the object itself. Select-Object can be used to select the properties within a specified property set.

In the following example, Get-Member is used to view property sets available on the objects returned by Get-Process:

PS> Get-Process -Id $PID | Get-Member -MemberType PropertySet
    TypeName: System.Diagnostics.Process
Name               MemberType     Definition 
----               ----------     ---------- 
PSConfiguration    PropertySet    PSConfiguration {Name, Id, ... 
PSResources        PropertySet    PSResources {Name, Id, Hand...

Select-Object may then be used to display one of the property sets, PSConfiguration:

PS> Get-Process -Id $PID | Select-Object -Property PSConfiguration
Name   Id PriorityClass FileVersion
----   -- ------------- -----------
pwsh 2220        Normal 7.0.0.0

The Sort-Object command

The Sort-Object command allows objects to be sorted. By default, Sort-Object will sort objects in ascending order:

PS> 5, 4, 3, 2, 1 | Sort-Object
1
2
3
4
5

Strings are sorted in ascending order, irrespective of uppercase or lowercase:

PS> 'ccc', 'BBB', 'aaa' | Sort-Object
aaa
BBB
ccc

When dealing with more complex objects, Sort-Object may be used to sort based on a named property. For example, processes may be sorted based on the Id property:

Get-Process | Sort-Object -Property Id 

Objects may be sorted on multiple properties; for example, a list of files may be sorted on LastWriteTime and then on Name:

Get-ChildItem C:WindowsSystem32 | 
    Sort-Object LastWriteTime, Name 

In the preceding example, items are first sorted on LastWriteTime. Items that have the same value for LastWriteTime are then sorted based on Name.

Sort-Object is not limited to sorting on existing properties. A script block (a fragment of script, enclosed in curly braces) can be used to create a calculated value for sorting. For example, it is possible to order items based on a word, as shown in this example:

$examResults = @(
    [PSCustomObject]@{ Exam = 'Music';   Result = 'N/A';  Mark = 0 }
    [PSCustomObject]@{ Exam = 'History'; Result = 'Fail'; Mark = 23 }
    [PSCustomObject]@{ Exam = 'Biology'; Result = 'Pass'; Mark = 78 }
    [PSCustomObject]@{ Exam = 'Physics'; Result = 'Pass'; Mark = 86 }
    [PSCustomObject]@{ Exam = 'Maths';   Result = 'Pass'; Mark = 92 }
)
$examResults | Sort-Object {
    switch ($_.Result) {
        'Pass' { 1 }
        'Fail' { 2 }
        'N/A'  { 3 }
    }
}

The result of sorting the objects is shown here:

Exam       Result    Mark
----       ------    ----
Maths      Pass        92
Physics    Pass        86
Biology    Pass        78
History    Fail        23
Music       N/A         0 

In the preceding example, when Sort-Object encounters a Pass result it is given the lowest numeric value (1) to sort on. As Sort-Object defaults to ascending order, this means exams with a result of Pass appear first in the list. This process is repeated to give a numeric value to each of the other possible results.

Sorting within each Result set varies depending on the version of PowerShell. Windows PowerShell changes the order of the elements within each set, listing Maths, Physics, then Biology. PowerShell 6 and above, on the other hand, maintains the original order, listing Biology, then Physics, then Maths within the pass set.

As Sort-Object is capable of sorting on more than one property, the preceding example can be taken further to sort on mark next. This makes the output order entirely predictable, regardless of the version of PowerShell:

PS> $examResults | Sort-Object { 
>>     switch ($_.Result) { 
>>         'Pass' { 1 } 
>>         'Fail' { 2 } 
>>         'N/A'  { 3 } 
>>     } 
>> }, Mark
Exam     Result    Mark
----     ------    ----
Biology    Pass      78
Physics    Pass      86
Maths      Pass      92
History    Fail      23
Music       N/A       0

Adding the Descending parameter to Sort-Object will reverse the order of both fields:

PS> $examResults | Sort-Object { 
>>     switch ($_.Result) { 
>>         'Pass' { 1 } 
>>         'Fail' { 2 } 
>>         'N/A'  { 3 } 
>>     } 
>> }, Mark -Descending
Exam     Result    Mark
----     ------    ----
Music       N/A       0
History    Fail      23
Maths      Pass      92
Physics    Pass      86
Biology    Pass      78

The ordering behavior can be made property-specific using the notation that is shown in the following example:

PS> $examResults | Sort-Object { 
>>     switch ($_.Result) { 
>>         'Pass' { 1 } 
>>         'Fail' { 2 } 
>>         'N/A'  { 3 } 
>>     } 
>> }, @{ Expression = { $_.Mark }; Descending = $true } 
Exam     Result    Mark
----     ------    ----
Maths      Pass      92
Physics    Pass      86
Biology    Pass      78
History    Fail      23
Music       N/A       0

The hashtable, @{}, is used to describe an expression (a calculated property; in this case, the value for Mark) and the sorting order, which is either ascending or descending.

In the preceding example, the first sorting property, based on the Result property, is sorted in ascending order as this is the default. The second property, Mark, is sorted in descending order.

Once a set of data is has been prepared by selecting and sorting, grouping and measuring can be used to work on the collection.

Grouping and measuring

Group-Object is a powerful command that allows you to group objects together based on a property or expression.

Measure-Object supports several simple mathematical operations, such as counting the number of objects, calculating an average, calculating a sum, and so on. Measure-Object also allows characters, words, or lines to be counted in text fields.

The Group-Object command

The Group-Object command shows a group and count for each occurrence of a value in a collection of objects.

Given the sequence of numbers shown, Group-Object creates a Name that holds the value it is grouping, a Count as the number of occurrences of that value, and a Group property as the set of similar values:

PS> 6, 7, 7, 8, 8, 8 | Group-Object
Count    Name                   Group
-----    ----                   -----
    1    6                      {6}
    2    7                      {7, 7}
    3    8                      {8, 8, 8}

The Group property may be removed using the NoElement parameter, which simplifies the output of the command:

PS> 6, 7, 7, 8, 8, 8 | Group-Object -NoElement
Count    Name 
-----    ---- 
    1    6 
    2    7 
    3    8 

Group-Object can group based on a specific property. For example, it might be desirable to list the number of occurrences of a file in an extensive folder structure. In the following example, the C:WindowsAssembly folder contains different versions of DLLs for different versions of packages, including the .NET Framework:

Get-ChildItem C:WindowsAssembly -Filter *.dll -Recurse | 
    Group-Object Name 

Combining Group-Object with commands such as Where-Object and Sort-Object allows reports about the content of a set of data to be generated extremely quickly, for example, a report on the names of the top five files that appear more than once in a file tree:

Get-ChildItem C:WindowsAssembly -Filter *.dll -File -Recurse |
    Group-Object Name -NoElement |
    Where-Object Count -gt 1 |
    Sort-Object Count, Name -Descending |
    Select-Object Name, Count -First 5

The output from the preceding command will vary from one computer to another; it depends on the installed software, development kits, .NET Framework version, and so on. The output from the preceding command might be similar to the following example:

Name                                                   Count
----                                                   -----
Microsoft.Web.Diagnostics.resources.dll                14
Microsoft.Web.Deployment.resources.dll                 14
Microsoft.Web.Deployment.PowerShell.resources.dll      14
Microsoft.Web.Delegation.resources.dll                 14
Microsoft.Web.PlatformInstaller.resources.dll          13

As was seen with Sort-Object, Group-Object can group on more than one property. For example, we might group on both a filename and the size of a file (the Length property of a file):

Get-ChildItem C:WindowsAssembly -Filter *.dll -Recurse |
    Group-Object Name, Length -NoElement |
    Where-Object Count -gt 1 |
    Sort-Object Name -Descending |
    Select-Object Name, Count -First 5

As with the previous example, the output from the command will vary from one computer to another:

Name                                                 Count
----                                                 -----
WindowsFormsIntegration.Package.ni.dll, 100352           2
Templates.Editorconfig.Wizard.resources.ni.dll, 9216    13
Templates.Editorconfig.Wizard.resources.ni.dll, 8192    13
System.Web.ni.dll, 16939008                              2
System.Web.ni.dll, 14463488                              2

In the preceding example, System.Web.ni.dll is listed twice (with a count of two in each case). Each pair of files has the same file size.

Like Sort-Object, Group-Object is not limited to properties that already exist. Calculated properties can be used to create a new value to group. For example, grouping on an email domain in a list of email addresses might be useful. The domain is obtained by splitting on the @ character:

PS> '[email protected]', '[email protected]', '[email protected]' |
>>     Group-Object { ($_ -split '@')[1] }
Count    Name                      Group 
-----    ----                      ----- 
    2    one.example               {[email protected], [email protected]}
    1    two.example               {[email protected]} 

In this example, the split operator is used to split on the @ character; everything to the left is stored in index 0, while everything to the right is stored in index 1.

By default, Group-Object returns the collection of objects shown in each of the preceding examples. Group-Object can also return a hashtable using the AsHashtable parameter.

When using the AsHashTable parameter, you can add the AsString parameter. For more complex values, this ensures the groups can be accessed. The AsString parameter forces the key for each entry in the hashtable to be a string; for example:

$hashtable = @(
    [IPAddress]'10.0.0.1'
    [IPAddress]'10.0.0.2'
    [IPAddress]'10.0.0.1'
) | Group-Object -AsHashtable -AsString
$hashtable['10.0.0.1']

If the AsString parameter was excluded from the preceding example, the value used to access the key would have to be an IPAddress type. For example:

$hashtable = @(
    [IPAddress]'10.0.0.1'
    [IPAddress]'10.0.0.2'
    [IPAddress]'10.0.0.1'
) | Group-Object -AsHashtable 
$hashtable[[IPAddress]'10.0.0.1']

In the preceding example, attempting to access the key without the [IPAddress] type will fail; no value will be returned, and no error will be shown.

By default, Group-Object is not case sensitive. The strings one, ONE, and One are all considered equal when grouping. The CaseSensitive parameter forces Group-Object to differentiate between items where cases differ:

PS> 'one', 'ONE', 'One' | Group-Object -CaseSensitive
Count Name                    Group
----- ----                    -----
   1  one                     {one}
   1  ONE                     {ONE}
   1  One                     {One}

Group-Object can be used to count occurrences of a value within a collection of objects. Measure-Object is useful when it is necessary to analyze the values, such as when determining the average of a specific property.

The Measure-Object command

When used without any parameters, Measure-Object returns a value for Count, which is the number of items passed in using the pipeline; for example:

PS> 1, 5, 9, 79 | Measure-Object
Count     : 4
Average   :
Sum       :
Maximum   :
Minimum   :
Property  :

Each of the remaining properties is empty, unless requested using their respective parameters. For example, Sum may be requested:

PS> 1, 5, 9, 79 | Measure-Object -Sum
Count      : 4
Average    :
Sum        : 94
Maximum    :
Minimum    :
Property   :

Adding the remaining parameters adds values to the rest of the fields (except Property):

PS> 1, 5, 9, 79 | Measure-Object -Average -Maximum -Minimum -Sum
Count       : 4
Average     : 23.5
Sum         : 94
Maximum     : 79
Minimum     : 1
Property    :

The value for Property is added when Measure-Object is asked to work against a particular property (instead of a set of numbers). For example:

PS> Get-Process | Measure-Object WorkingSet -Average
Count       : 135
Average     : 39449395.2
Sum         :
Maximum     :
Minimum     :
Property    : WorkingSet

When working with text, Measure-Object can count characters, words, or lines. For example, it can be used to count the number of lines, words, and characters in a text file:

PS> Get-Content C:WindowsWindowsUpdate.log |
>>    Measure-Object -Line -Word -Character
Lines    Words    Characters    Property
-----    -----    ----------    --------
    3       32           268

Group-Object and Measure-Object are essential parts of a PowerShell toolkit. They can significantly simplify analyzing collections of data for repetition or performing simple mathematical operations. Each of these commands is used against a single collection of objects; when working with more than one collection of objects, it may be necessary to compare.

Comparing

You can use the Compare-Object command to compare collections of objects with one another.

Compare-Object must be supplied with values for the ReferenceObject and DifferenceObject parameters, which are normally collections or arrays of objects. If either value is null, then an error will be displayed. If both values are equal, Compare-Object does not return anything by default. For example, the reference and difference objects in the following example are identical:

Compare-Object -ReferenceObject 1, 2 -DifferenceObject 1, 2 

If there are differences, Compare-Object displays the results, as shown here:

PS> Compare-Object -ReferenceObject 1, 2, 3, 4 -DifferenceObject 1, 2
InputObject SideIndicator
----------- -------------
          3 <=
          4 <=

This shows that ReferenceObject (the collection on the left, denoted by the direction of the <= arrow) has the values, but DifferenceObject (the collection on the right) does not.

Compare-Object has several other parameters that may be used to change the output. The IncludeEqual parameter adds values that are present in both collections to the output:

PS> $params = @{
>>     ReferenceObject  = 1, 2, 3, 4
>>     DifferenceObject = 1, 2
>>     IncludeEqual     = $true
>> }
PS> Compare-Object @params
 
InputObject SideIndicator
----------- -------------
          1 ==
          2 ==
          3 <=
          4 <=

ExcludeDifferent will omit the results that differ. This parameter makes sense if IncludeEqual is also set; without this, the command will always return nothing.

The PassThru parameter is used to return the original object instead of the representation showing the differences. In the following example, it is used to select values that are common to both the reference and difference objects:

PS> $params = @{
>>     ReferenceObject  = 1, 2, 3, 4
>>     DifferenceObject = 1, 2
>>     ExcludeDifferent = $true
>>     IncludeEqual     = $true
>>     PassThru         = $true
>> }
PS> Compare-Object @params
1
2

Compare-Object can compare based on properties of objects, as well as the simpler values in the preceding examples. This can be a single property, or a list of properties. For example, the following command compares the contents of C:WindowsSystem32 with C:WindowsSysWOW64, returning files that have the same name and are the same size in both:

$params = @{
    ReferenceObject  = Get-ChildItem C:WindowsSystem32 -File 
    DifferenceObject = Get-ChildItem C:WindowsSysWOW64 -File
    Property         = 'Name', 'Length'
    IncludeEqual     = $true
    ExcludeDifferent = $true
}
Compare-Object @params

By default, Compare-Object writes an error if either the reference or difference objects are null. If Compare-Object is used when there is a chance of either being empty, the following technique can be used to avoid an error being generated provided neither contains an explicit null value:

$reference = Get-ChildItem C:WindowsSystem32	cpmon*.ini
$difference = Get-ChildItem C:WindowsSysWOW64	cpmon*.ini
Compare-Object @($reference) @($difference) -Property Name

The array operator (@()) around each parameter value will be discarded by PowerShell. If $difference is empty, it will be treated as an empty array instead of it being a null value.

Collections of objects generated by any of the preceding commands might be exported to move data outside of PowerShell. The result of any of these operations might be exported to a file for another user, system, or program to use.

Importing, exporting, and converting

Getting data in and out of PowerShell is a critical part of using the language. There are several commands dedicated to this task, including:

• Export-Csv
• Import-Csv
• Export-CliXml
• Import-CliXml
• Tee-Object

Other PowerShell modules, such as the ImportExcel module, available in the PowerShell Gallery can be used to extend the output formats available. Commands such as ConvertTo-Html, ConvertTo-Json, and ConvertFrom-Json are explored in later chapters.

The Export-Csv command

The Export-Csv command writes data from objects to a text file; for example:

Get-Process | Export-Csv processes.csv

By default, Export-Csv writes a comma-delimited file using UTF8 encoding and completely overwrites any file using the same name.

Export-Csv may be used to add lines to an existing file using the Append parameter. When the Append parameter is used, the input object must have each of the fields listed in the CSV header or an error will be thrown unless the Force parameter is used:

PS> Get-Process -ID $PID |
>>     Select-Object Name, Id |
>>     Export-Csv .Processes.csv
PS> Get-Process explorer |
>>     Select-Object Name |
>>     Export-Csv .Processes.csv -Append
Export-Csv: Cannot append CSV content to the following file: .Processes.csv. The appended object does not have a property that corresponds to the following column: Id. To continue with mismatched properties, add the -Force parameter, and then retry the command. 

If you use the Append parameter and the input object has more fields than the CSV, the extra fields are silently dropped when writing the CSV file. For example, the value held in Id is ignored when writing the results to the existing CSV file:

Get-Process pwsh |
    Select-Object Name | Export-Csv .Processes.csv 
Get-Process explorer |
    Select-Object Name, Id |
    Export-Csv .Processes.csv -Append

Export-Csv in Windows PowerShell writes a header line to each file, which details the .NET type it has just exported. If the preceding example was used in Windows PowerShell, the header line would be as follows:

#TYPE Selected.System.Diagnostics.Process

This header is only included in PowerShell 7 (and PowerShell Core) when explicitly using the IncludeTypeInformation parameter.

Export-Csv in Windows PowerShell can be instructed to exclude this header using the NoTypeInformation parameter:

Get-Process | Export-Csv processes.csv -NoTypeInformation 

ConvertTo-Csv in Windows PowerShell is like Export-Csv, except that instead of writing content to a file, content is written as command output:

PS> Get-Process powershell | Select-Object Name, Id | ConvertTo-Csv
#TYPE Selected.System.Diagnostics.Process
"Name","Id"
"pwsh","404"

Both Export-Csv and ConvertTo-Csv are limited in what they can do with arrays of objects in properties. For example, ConvertTo-Csv is unable to display the values that are in an array:

PS> [PSCustomObject]@{
>>     Name = "Numbers"
>>     Value = 1, 2, 3, 4, 5
>> } | ConvertTo-Csv -NoTypeInformation
"Name","Value"
"Numbers","System.Object[]"

The value of the Value property in the CSV content is taken from the ToString method, which is called on the property named Value; for example:

PS> $object = [PSCustomObject]@{
>>     Name = "Numbers"
>>     Value = 1, 2, 3, 4, 5
>> }
PS> $object.Value.ToString()
System.Object[]

If a CSV file is expected to hold the content of an array, code must be written to convert it into a suitable format. For example, the content of the array can be written after converting it to a string:

PS> [PSCustomObject]@{
>>     Name  = "Numbers"
>>     Value = 1, 2, 3, 4, 5
>> } | ForEach-Object {
>>     $_.Value = $_.Value -join ', '
>>     $_
>> } | ConvertTo-Csv -NoTypeInformation
"Name","Value"
"Numbers","1, 2, 3, 4, 5"

In the preceding example, the value of the property is joined using a comma followed by a space. The modified object (held in $_) is passed on to the ConvertTo-Csv command.

The Import-Csv command

Comma-Separated Value (CSV) files are structured text. Applications such as Microsoft Excel can work with CSV files without changing the file format, although Excel's advanced features cannot be saved to a CSV file.

By default, Import-Csv expects the input to have a header row, be comma-delimited, and use ASCII file encoding. If any of these items are different, the command parameters may be used. For example, a tab may be set as the delimiter:

Import-Csv TabDelimitedFile.tsv -Delimiter `t 

A tick (grave accent) followed by t (`t) is used to represent the tab character in PowerShell.

Data imported using Import-Csv will always be formatted as a string. If Import-Csv is used to read a file containing the following text, each of the numbers will be treated as a string:

Name,Position 
Jim,35 
Matt,3 
Dave,5 

Attempting to use Sort-Object on the imported CSV file results in values being sorted as if they were strings, not numbers:

PS> Import-Csv .positions.csv | Sort-Object Position
Name    Position
----    --------
Matt    3
Jim     35
Dave    5

You can use Sort-Object to consider the value for Position as an integer by using a script block expression:

PS> Import-Csv .positions.csv | Sort-Object { [Int]$_.Position }
Name Position
---- --------
Matt  3
Dave  5
Jim   35

This conversion problem exists regardless of whether the data in a CSV file is numeric, a date, or any type other than a string.

The ConvertFrom-Csv command is similar to Import-Csv in that it reads CSV content and creates custom objects from that content. The difference is that ConvertFrom-Csv reads strings from standard input instead of a file. In the following example, a string is converted into a custom object using the ConvertFrom-Csv command with the Header parameter:

PS> "powershell,404" | ConvertFrom-Csv -Header Name, Id
Name          Id
----          --
powershell    404

ConvertFrom-Csv expects either an array of strings or a single string with line-breaks. The following example includes a header in the string:

"Name,Id
Powershell,404" | ConvertFrom-Csv

CSV is a simple and accessible format. However, CSV is a pure-text format; it cannot express different value types (such as numbers and dates) – all data is a string. The CliXml format is at the other end of the spectrum: it can be used to store complex data types.

Export-Clixml and Import-Clixml

Export-Clixml creates representations of objects in XML files. The CLI acronym stands for Common Language Infrastructure, a technical standard developed by Microsoft. Export-Clixml is extremely useful where type information about each property must be preserved.

For example, the following object may be exported using Export-Clixml:

[PSCustomObject]@{ 
    Integer = 1 
    Decimal = 2.3 
    String  = 'Hello world' 
} | Export-Clixml .object.xml 

The resulting XML file shows the type for each of the properties it has just exported:

PS> Get-Content object.xml
<Objs Version="1.1.0.1" xmlns="http://schemas.microsoft.com/powershell/2004/04">
  <Obj RefId="0">
    <TN RefId="0">
      <T>System.Management.Automation.PSCustomObject</T>
      <T>System.Object</T>
    </TN>
    <MS>
      <I32 N="Number">1</I32>
      <Db N="Decimal">2.3</Db>
      <S N="String">Hello world</S>
    </MS>
  </Obj>
</Objs>

In the preceding example, I32 is a 32-bit integer (Int32). Db is a double-precision floating-point number (double). S is a string.

With this extra information in the file, PowerShell can rebuild the object, including the different types, using Import-Clixml, as follows:

$object = Import-Clixml .object.xml

Once imported, the value types can be inspected using the GetType method:

PS> $object.Decimal.GetType()
IsPublic    IsSerial    Name            BaseType
--------    --------    ----            --------
   True         True    Double          System.ValueType

The ability to rebuild the original object allows Export-CliXml to convert credential objects to text. When it does so, password values are encrypted using Data Protection API (DPAPI) on Windows. For example, providing a username and password when prompted will create an XML file holding the encrypted password:

Get-Credential | Export-CliXml -Path secret.xml

The file can be opened in a text editor to view the encrypted password. The credential can be imported again using Import-CliXml:

$credential = Import-CliXml -Path secret.xml

The password for the credential is encrypted using the current user account (protected by the login password). The key used is held in the user profile; the resulting file can only be decrypted on the computer it was created on (without a roaming profile).

The Tee-Object command

The Tee-Object command is used to send output to two places at the same time. Tee-Object is used to write output to a console and a file or variable.

For example, the following command both displays the output of Get-Process on the screen and writes the content to a $processes variable.

Get-Process | Tee-Object -Variable processes

Tee-Object writes file output as the console sees it (table or list) rather than writing in CSV or another format.

Summary

The pipeline is a key component of PowerShell. It allows data, as objects, to be sent from one command to another. Each command can act on the data it has received and, in many cases, return more data.

PowerShell includes a variety of commands for working with objects in a pipeline. These commands are used again and again no matter how experienced a PowerShell developer might be.

The Get-Member command allows the members (properties, methods, and so on) to be explored, which can be used to understand what an object is capable of.

ForEach-Object is a common command used to run arbitrary code against objects in a pipeline. Where-Object may be used to filter a pipeline, returning only relevant objects.

The Select-Object command is used to define what properties should be returned. You can also use Select-Object to include or remove objects, for example, by selecting the first few objects from a pipeline. The Sort-Object command takes pipeline input and allows it to be sorted based on property names, or more complex criteria described by a script block.

You can perform comparisons of collections of objects using the Compare-Object command.

Finally, content in PowerShell can be exported to and imported from files in a variety of different ways. This chapter explored exporting and importing content using the Export-Csv and Import-Csv commands as well as the more complex output created by Export-CliXml.

The next chapter, Chapter 4, Operators, explores the wide variety of operators available in PowerShell, ranging from arithmetic and comparison operators to binary and redirection operators.