In addition to the AST, PowerShell can also convert a script into a series of tokens, each representing an element of a script.

In PowerShell 2, the Tokenize static method of System.Management.Automation.PSParser may be used; an example is as follows:

$script = @'
# A short script
if ($true) {
 Write-Host 'Hello world'
}
'@
$errors = @()
$tokens = [System.Management.Automation.PSParser]::Tokenize($script, [Ref]$errors)

The tokens array contains objects describing each part of the script. The first of these is shown as follows; it describes the comment at the start of the script:

Content     : # A short script
Type        : Comment
Start       : 0
Length      : 16
StartLine   : 1
StartColumn : 1
EndLine     : 1
EndColumn   : 17

With PowerShell 3, two static methods on the System.Management.Automation.Language.Parser Parser type may be used: ParseInput and ParseFile. The two methods return an AST; tokens are returned via a reference to an array. An example is as follows:

$script = @'
# A short script
if ($true) {
    Write-Host 'Hello world'
}
'@
$tokens = $errors = @()
$ast = [System.Management.Automation.Language.Parser]::ParseInput(
    $script,
    [Ref]$tokens,
    [Ref]$errors
)

The token that's returned is structured differently from the token returned by the Tokenize method in PowerShell 2. The comment token is shown as follows:

Text       : # A short script
TokenFlags : ParseModeInvariant
Kind       : Comment
HasError   : False
Extent     : # A short script

Both AST nodes and the PowerShell 3 token objects are used by PSScriptAnalyzer.