13
Web Requests and Web Services
Representational State Transfer (REST) and Simple Object Access Protocol (SOAP) are often used as labels to refer to two different approaches to implementing a web-based Application Programming Interface (API).
The growth of cloud-based services in recent years has pushed the chances of working with such interfaces from rare to almost certain.
This chapter covers the following topics:
- Web requests
- Working with REST
- Working with SOAP
Before beginning with the main topics of the chapter, there are some technical requirements to consider.
Technical requirements
In addition to PowerShell and PowerShell Core, you'll need Visual Studio 2019 Community Edition or better to follow the SOAP service example.
SOAP interfaces typically use the New-WebServiceProxy command in Windows PowerShell. This command is not available in PowerShell 7 as the assembly it depends on is not available. The command is unlikely to be available in PowerShell Core unless it is rewritten.
Web requests, such as getting content from a website or downloading files, are a common activity in PowerShell, and this is the first topic we'll cover.
Web requests
A background in web requests is valuable before delving into interfaces that run over the top of the Hypertext Transfer Protocol (HTTP).
PowerShell can use Invoke-WebRequest to send HTTP requests. For example, the following command will return the response to a GET request for the Hey, Scripting Guy blog:
Invoke-WebRequest https://blogs.technet.microsoft.com/heyscriptingguy/
Parsing requires Internet Explorer
In Windows PowerShell, UseBasicParsing was an important parameter. Its use was mandatory when working on Core installations of Windows Server as Internet Explorer was not installed. It was also often used to improve the performance of a command where parsing was not actually required.
In PowerShell Core, all requests use basic parsing. The parameter is deprecated and present to support backward compatibility only. The parameter does not affect the output of the command.
A web request has an HTTP method. A request might use SSL, and sometimes may need to work with invalid or self-signed certificates. The web request is the foundation for working with web-based APIs.
HTTP methods
HTTP supports several different methods, including the following:
GETHEADPOSTPUTDELETECONNECTOPTIONSTRACEPATCH
These methods are defined in the HTTP 1.1 specification: https://www.w3.org/Protocols/rfc2616/rfc2616-sec9.html.
It is common to find that a web server only supports a subset of these. In many cases, supporting too many methods is deemed to be a security risk. The Invoke-WebRequest command can be used to verify the list of HTTP methods supported by a site, for example:
PS> Invoke-WebRequest www.indented.co.uk -Method OPTIONS |
>> Select-Object -ExpandProperty Headers
Key Value
--- -----
Allow GET, HEAD
HTTPS
If a connection to a web service uses HTTPS (HTTP over Secure Sockets Layer (SSL)), the certificate must be validated before a connection can complete and a request can be completed. If a web service has an invalid certificate, an error will be returned.
The badssl site can be used to test how PowerShell might react to different SSL scenarios: https://badssl.com/.
For example, when attempting to connect to a site with an expired certificate (using Invoke-WebRequest), the following message will be displayed in Windows PowerShell:
PS> Invoke-WebRequest https://expired.badssl.com/
Invoke-WebRequest : The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel.
At line:1 char:1
+ Invoke-WebRequest https://expired.badssl.com/
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-WebRequest], WebException
+ FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeWebRequestCommand
In PowerShell 7, this message changes to Invoke-WebRequest: The remote certificate is invalid according to the validation procedure.
In Windows PowerShell, Invoke-WebRequest cannot bypass or ignore an invalid certificate on its own (using a parameter). Certificate validation behavior may be changed by adjusting the CertificatePolicy on the ServicePointManager: https://docs.microsoft.com/dotnet/api/system.net.servicepointmanager?view=netframework-4.8.
The CertificatePolicy is a process-specific policy control affecting web requests in .NET made by that process. The policy is applied using the ServicePointManager, which manages HTTP connections.
Removing certificate handling changes
PowerShell should be restarted to reset the certificate policies to system defaults. Changes made by ServicePointManager apply to the current PowerShell process only and do not persist once PowerShell is closed.
In PowerShell 7, Invoke-WebRequest has a parameter that allows certificate errors to be ignored, as the following shows:
Invoke-WebRequest https://expired.badssl.com/ -SkipCertificateCheck
Chain of trust
Certificates are based on a chain of trust. Authorities are trusted to carry out sufficient checks to prove the identity of the certificate holder. Skipping certificate validation is insecure and should only be used against known hosts that can be trusted.
It may be necessary to ignore an SSL error when making a web request for any number of reasons; for example, the certificate might be self-signed. SSL errors in PowerShell 7 can be bypassed using the preceding parameter. SSL errors in Windows PowerShell are explored in the next section.
Bypassing SSL errors in Windows PowerShell
If a service has an invalid certificate, the best response is to fix the problem. When it is not possible or practical to address the real problem, a workaround can be created.
The approach described here applies to Windows PowerShell only. PowerShell Core does not include the ICertificatePolicy type.
This modification applies to the current PowerShell session and will reset to default behavior every time a new PowerShell session is opened.
The certificate policy used by the ServicePointManager can be replaced with a customized handler by writing a class (PowerShell, version 5) that replaces the CheckValidationResult method:
Class AcceptAllPolicy: System.Net.ICertificatePolicy {
[Boolean] CheckValidationResult(
[System.Net.ServicePoint] $servicePoint,
[System.Security.Cryptography.X509Certificates.X509Certificate] $certificate,
[System.Net.WebRequest] $webRequest,
[Int32] $problem
) {
return $true
}
}
[System.Net.ServicePointManager]::CertificatePolicy = [AcceptAllPolicy]::new()
Once the policy is in place, certificate errors will be ignored as the previous method returns true no matter its state:
PS> Invoke-WebRequest "https://expired.badssl.com/"
StatusCode : 200
StatusDescription : OK
...
CertificatePolicy is obsolete
The CertificatePolicy property on the ServicePointManager is marked as obsolete in Microsoft Docs. This approach appears to be required when using Invoke-WebRequest in PowerShell 5.
Requests made by System.Net.WebClient in Windows PowerShell are satisfied by this simpler approach, which trusts all certificates:
[System.Net.ServicePointManager]::ServerCertificateValidationCallback = { $true }
This approach is not feasible with PowerShell 7. Requests made using WebClient may be replaced by either Invoke-WebRequest or the HttpClient.
Capturing SSL errors
The ServerCertificateValidationCallback property of ServicePointManager does not work as expected in PowerShell 7. Attempts to assign and use a script block may result in an error being displayed, as shown here, when making a web request:
PS> [System.Net.ServicePointManager]::ServerCertificateValidationCallback = { $true }
PS> $webClient = [System.Net.WebClient]::new()
PS> $webClient.DownloadString('https://expired.badssl.com/')
MethodInvocationException: Exception calling "DownloadString" with "1" argument(s): "The SSL connection could not be established, see inner exception. There is no Runspace available to run scripts in this thread. You can provide one in the DefaultRunspace property of the System.Management.Automation.Runspaces.Runspace type. The script block you attempted to invoke was: $true "
The .NET SslStream type (System.Net.Security.SslStream) offers a potential alternative for capturing detailed certificate validation information. The method used in the following example works in both Windows PowerShell and PowerShell Core.
This example converts certificate validation information using Export-CliXml. Assigning the parameters to a global variable is possible, but certain information is discarded when the callback ends, including the elements of the certificate chain. Note that the following example uses three different namespaces. If the example is pasted into the console, then these using statements must appear on a single line, each separated by a semi-colon:
using namespace System.Security.Cryptography.X509Certificates
using namespace System.Net.Security
using namespace System.Net.Sockets
$remoteCertificateValidationCallback = {
param (
[Object]$sender,
[X509Certificate2]$certificate,
[X509Chain]$chain,
[SslPolicyErrors]$sslPolicyErrors
)
$psboundparameters | Export-CliXml C: empCertValidation.xml
# Always indicate SSL negotiation was successful
$true
}
try {
[Uri]$uri = 'https://expired.badssl.com/'
$tcpClient = [TcpClient]::new()
$tcpClient.Connect($Uri.Host, $Uri.Port)
$sslStream = [SslStream]::new(
$tcpClient.GetStream(),
# leaveInnerStreamOpen: Close the inner stream when complete
$false,
$remoteCertificateValidationCallback
)
$sslStream.AuthenticateAsClient($Uri.Host)
} catch {
throw
} finally {
if ($tcpClient.Connected) {
$tcpClient.Close()
}
}
$certValidation = Import-CliXml C: empCertValidation.xml
Once the content of the XML file has been loaded, the content may be investigated. For example, the certificate that was exchanged can be viewed:
$certValidation.Certificate
Or, the response can be used to inspect all the certificates in the key chain:
$certValidation.Chain.ChainElements | ForEach-Object Certificate
The ChainStatus property exposes details of any errors during chain validation:
$certValidation.Chain.ChainStatus
ChainStatus is summarized by the SslPolicyErrors property.
The HTTP methods demonstrated can be put to use with Representational State Transfer (REST).
Working with REST
A REST-compliant web service allows a client to interact with the service using a set of predefined stateless operations. REST is not a protocol; it is an architectural style.
Whether or not an interface is truly REST-compliant is not particularly relevant when the goal is to use one in PowerShell. Interfaces must be used according to any documentation that has been published.
Invoke-RestMethod
The Invoke-RestMethod command can execute methods exposed by web services. The name of a method is part of the Uniform Resource Identifier (URI); it is important not to confuse this with the Method parameter. The Method parameter is used to describe the HTTP method. By default, Invoke-RestMethod uses HTTP GET.
Simple requests
The REST API provided by GitHub may be used to list repositories made available by the PowerShell team.
The API entry point, the common URL all REST methods share, is https://api.github.com as documented in the GitHub reference: https://docs.github.com/rest.
When working with REST, documentation is important. The way an interface is used is common, but the way it may respond is not (as this is an architectural style, not a strict protocol).
The specific method being called is documented on a different page of the following reference: https://docs.github.com/rest/reference/repos#list-user-repositories.
The name of the user forms part of the URI; there are no arguments for this method. Therefore, the following command will execute the method and return detailed information about the repositories owned by the PowerShell user (or organization):
Invoke-RestMethod -Uri https://api.github.com/users/powershell/repos
Windows PowerShell is likely to throw an error relating to SSL/TLS when running this command. This is because the site uses TLS 1.2 whereas, by default, Invoke-RestMethod reaches as far as TLS 1.0. PowerShell 7 users should not experience this problem.
This Windows PowerShell problem can be fixed by tweaking the SecurityProtocol property of ServicePointManager as follows:
using namespace System.Net
[ServicePointManager]::SecurityProtocol = [ServicePointManager]::SecurityProtocol -bor 'Tls12'
The bitwise -bor operator is used to add TLS 1.2 to the default list defined by a combination of the .NET version and the Windows version. That typically includes Ssl3 and Tls. TLS 1.1 (Tls11) may be added in a similar manner if required. The change applies to the current PowerShell process and does not persist.
All examples use TLS 1.2
This setting is required for the examples that follow when running Windows PowerShell.
Older versions of Windows may require a patch from Windows Update to gain support for TLS 1.2.
Requests with arguments
The search code method of the GitHub REST API is used to demonstrate how arguments can be passed to a REST method.
The documentation for the method is found in the following API reference: https://docs.github.com/rest/reference/search#search-code.
The following example uses the search code method by building a query string and appending that to the end of the URL. The search looks for occurrences of the Get-Content term in PowerShell language files in the PowerShell repository. The search term is therefore as follows; the search string here should not be confused with a PowerShell command:
Get-Content language:powershell repo:powershell/powershell
Get-Content is not the PowerShell command Get-Content
PowerShell has a Get-Content command. The Get-Content term used in the previous search string should not be confused with the PowerShell command.
Converting the example from the documentation for the search method, the URL required is as follows. Spaces may be replaced by + when encoding the URL: https://api.github.com/search/code?q=Get-Content+language:powershell+repo:powershell/powershell.
In Windows PowerShell, which can use the HttpUtility type within the System.Web assembly, the task of encoding the URL can be simplified:
using assembly System.Web
$queryString = [System.Web.HttpUtility]::ParseQueryString('')
$queryString.Add(
'q',
'Get-Content language:powershell repo:powershell/powershell'
)
$uri = 'https://api.github.com/search/code?{0}' -f $queryString
Invoke-RestMethod $uri
The result is a custom object that includes the search results:
PS> Invoke-RestMethod $uri
total_count incomplete_results items
----------- ------------------ -----
80 False {@{name=Get-Content.Tests....
Running $queryString.ToString() shows that the colon character has been replaced by %3, and the forward-slash in the repository name has been replaced by %2. The %3 and %2 are HTTP encodings of the colon and forward-slash characters.
The arguments for the search do not necessarily have to be passed as a query string. Instead, a body for the request may be set, as shown here:
Invoke-RestMethod -Uri https://api.github.com/search/code -Body @{
q = 'Get-Content language:powershell repo:powershell/powershell'
}
Invoke-RestMethod converts the body (a Hashtable) to JSON and handles any encoding required. The result of the search is the same whether the body or a query string is used.
In both cases, details of the files found are held within the items property of the response. The following example shows the file name and path:
$params = @{
Uri = 'https://api.github.com/search/code'
Body = @{
q = 'Get-Content language:powershell repo:powershell/powershell'
}
}
Invoke-RestMethod @params |
Select-Object -ExpandProperty items |
Select-Object name, path
This pattern, where the actual results are nested under a property in the response, is frequently seen with REST interfaces. Exploration is often required.
It is critical to note that REST interfaces are case-sensitive; using a parameter named Q would result in an error message, as shown here:
PS> Invoke-RestMethod -Uri https://api.github.com/search/code -Body @{
>> Q = 'Get-Content language:powershell repo:powershell/powershell'
>> }
Invoke-RestMethod: {"message":"Validation Failed","errors":[{"resource":"Search","field":"q","code":"missing"}],"documentation_url":"https://developer.github.com/v3/search"}
The GitHub API returns an easily understood error message in this case. This will not be true of all REST APIs; it is common to see a generic error returned by an API. An API may return a simple HTTP 400 error and leave it to the user or developer to figure out what went wrong.
Working with paging
Many REST interfaces will return large result sets from searches in pages, a subset of the results. The techniques used to retrieve each subsequent page can vary from one API to another. This section explores how those pages can be retrieved from the web service.
The GitHub API exposes the link to the next page in the HTTP header. This is consistent with RFC 5988 (https://tools.ietf.org/html/rfc5988#page-6).
In PowerShell 7, it is easy to retrieve and view the header when using Invoke-RestMethod:
$params = @{
Uri = 'https://api.github.com/search/issues'
Body = @{
q = 'documentation state:closed repo:powershell/powershell'
}
ResponseHeadersVariable = 'httpHeader'
}
Invoke-RestMethod @params | Select-Object -ExpandProperty items
Once run, the link field of the header may be inspected via the httpHeader variable:
PS> $httpHeader['link']
<https://api.github.com/search/issues?q=documentation+state%3Aclosed+repo%3Apowershell%2Fpowershell&page=2>; rel="next",
<https://api.github.com/search/issues?q=documentation+state%3Aclosed+repo%3Apowershell%2Fpowershell&page=34>; rel="last"
PowerShell 7 can also automatically follow this link by using the FollowRelLink parameter. This might be used in conjunction with the MaximumFollowRelLink parameter to ensure a request stays within any rate limiting imposed by the web service. See https://docs.github.com/en/rest/overview/resources-in-the-rest-api#rate-limiting for the GitHub API. The following request searches for closed documentation issues, following the paging link twice:
$params = @{
Uri = 'https://api.github.com/search/issues'
Body = @{
q = 'documentation state:closed repo:powershell/powershell'
}
FollowRelLink = $true
MaximumFollowRelLink = 2
}
Invoke-RestMethod @params | Select-Object -ExpandProperty items
Windows PowerShell, unfortunately, cannot automatically follow this link. Nor does the Invoke-RestMethod command expose the header from the response. When working with complex REST interfaces in Windows PowerShell, it is often necessary to fall back to the Invoke-WebRequest or even HttpWebRequest classes.
The example that follows uses Invoke-WebRequest in Windows PowerShell to follow the next link in a similar manner to Invoke-RestMethod in PowerShell 7:
# Used to limit the number of times "next" is followed
$followLimit = 2
# The initial set of parameters, describes the search
$params = @{
Uri = 'https://api.github.com/search/issues'
# PowerShell will convert this to JSON
Body = @{
q = 'documentation state:closed repo:powershell/powershell'
}
ContentType = 'application/json'
}
# Just a counter, works in conjunction with followLimit.
$followed = 0
do {
# Get the next response
$response = Invoke-WebRequest @params
# Convert and leave the results as output
$response.Content |
ConvertFrom-Json |
Select-Object -ExpandProperty items
# Retrieve the links from the header and find the next URL
if ($response.Headers['link'] -match '<([^>]+?)>;s*rel="next"') {
$next = $matches[1]
} else {
$next = $null
}
# Parameters which will be used to get the next page
$params = @{
Uri = $next
}
# Increment the followed counter
$followed++
} until (-not $next -or $followed -ge $followLimit)
Because of the flexible nature of REST, implementations of page linking may vary. For example, links may appear in the body of a response instead of the header. Exploration is a requirement when working around a web API.
Working with authentication
There are many authentication systems that can be used when working with web services.
For services that expect to use the current user account to authenticate, the UseDefaultCredential parameter may be used to pass authentication tokens without explicitly passing a username and password. A service integrated into an Active Directory domain, expecting to use Kerberos authentication, is an example of such a service.
REST interfaces written to provide automation access tend to offer reasonably simple approaches to automation, often including basic authentication.
GitHub offers several different authentication methods, including basic and OAuth. These are shown here when attempting to request the email addresses for a user that requires authentication.
GitHub user basic auth is deprecated
GitHub removed basic authentication entirely in November 2020. The following basic authentication example will fail unless a personal access token is used as the password.
https://developer.github.com/changes/2020-02-14-deprecating-password-auth/
Using basic authentication
Basic authentication with a username and password is the simplest method available. For the GitHub API, the password must be a personal access token, basic authentication with a username and password was discontinued in November 2020.
You can generate personal access tokens by visiting account settings, then developer settings. Once generated, the personal access token cannot be viewed. The personal access token is used in place of a password:
$params = @{
Uri = 'https://api.github.com/user/emails'
Credential = Get-Credential
}
Invoke-RestMethod @params
In PowerShell 7, the Authentication parameter should be added:
$params = @{
Uri = 'https://api.github.com/user/emails'
Credential = Get-Credential
Authentication = 'Basic'
}
Invoke-RestMethod @params
GitHub provides documentation showing how to add the second authentication factor, although it is not clear how SMS tokens can be requested: https://docs.github.com/rest/overview/other-authentication-methods.
OAuth is an alternative to using a personal access token.
OAuth
OAuth is offered by a wide variety of web services. The details of this process will vary slightly between different APIs. The GitHub documentation describes the process that must be followed: https://docs.github.com/developers/apps/authorizing-oauth-apps#web-application-flow.
Implementing OAuth requires a web browser, or a web browser and a web server. As the browser will likely need to run JavaScript, this cannot be done using Invoke-WebRequest alone.
Creating an application
Before starting with code, an application must be registered with GitHub. This is done by visiting Settings, and then Developer settings, and finally OAuth Apps.
A new OAuth app must be created to acquire a clientId and clientSecret. Creating the application requires a homepage URL and an authorization callback URL. Both should be set to http://localhost:40000 for this example. This URL is used to acquire the authorization code.
The values from the web page will fill the following variables, the client secret must be generated by clicking Generate a new client secret. The secret must be regenerated if lost:
Generate a new client secret in screen text style.
$clientId = 'FromGitHub'
$clientSecret = 'FromGitHub'
Getting an authorization code
Once an application is registered, an authorization code is required. Obtaining the authorization code gives the end user the opportunity to grant the application access to a GitHub account. If the user is not currently logged in to GitHub, it will also prompt them to log on.
A URL must be created that will prompt for authorization:
$authorize = 'https://github.com/login/oauth/authorize?client_id={0}&scope={1}' -f @(
$clientId
'user:email'
)
The 'user:email' scope describes the rights the application would like to have. The web API guide contains a list of possible scopes: https://docs.github.com/developers/apps/scopes-for-oauth-apps.
OAuth browser issues
WPF and Windows Forms both include browser controls that can be used. However, both are based on Internet Explorer, which is not supported by GitHub. At this point, a choice must be made. Either a browser must be controlled, which is challenging, or an HTTP listener must be created to receive the response after OAuth completes. An HTTP listener is the easier path as this can leverage any browser installed on the current computer.
Implementing an HTTP listener
Implementing the web server has two advantages:
- Implementing a web server does not need additional libraries
- The web server can potentially be used on platforms other than Windows
The HttpListener is configured with the callback URL as a prefix. The prefix must end with a forward slash. The operating system gets to choose which browser should be used to complete the request:
$httpListener = [System.Net.HttpListener]::new()
$httpListener.Prefixes.Add('http://localhost:40000/')
$httpListener.Start()
$clientId = Read-Host 'Enter the client-id'
$authorizeUrl = 'https://github.com/login/oauth/authorize?client_id={0}&scope={1}' -f @(
$clientId
'user:email'
)
# Let the operating system choose the browser to use for this request
Start-Process -FilePath $authorizeUrl
$context = $httpListener.GetContext()
$buffer = [Byte[]][Char[]]"<html><body>OAuth complete! Please return to PowerShell!</body></html>"
$context.Response.OutputStream.Write(
$buffer,
0,
$buffer.Count
)
$context.Response.OutputStream.Close()
$httpListener.Stop()
$authorizationCode = $context.Request.QueryString['code']
In either case, the result of the process is code held in the $authorizationCode variable. This code can be used to request an access token.
Requesting an access token
The next step is to create an access token. The access token is valid for a limited time.
The clientSecret is sent with this request; if this were an application that was given to others, keeping the secret would be a challenge to overcome:
$params = @{
Uri = 'https://github.com/login/oauth/access_token'
Method = 'POST'
Body = @{
client_id = $clientId
client_secret = $clientSecret
code = $authorizationCode
}
}
$response = Invoke-RestMethod @params
$token = [System.Web.HttpUtility]::ParseQueryString($response)['access_token']
The previous request used the HTTP method POST. The HTTP method, which should be used with a REST method, is documented for an interface in the Developer Guides.
Each of the requests that follow will use the access token from the previous request. The access token is placed in an HTTP header field named Authorization.
Using a token
We can call methods that require authentication by adding a token to the HTTP header. The format of the Authorization header field is as follows:
Authorization: token OAUTH-TOKEN
OAUTH-TOKEN is replaced, and the Authorization head is constructed as follows:
$headers = @{
Authorization = 'token {0}' -f $token
}
The token can be used in subsequent requests for the extent of its lifetime:
$headers = @{
Authorization = 'token {0}' -f $token
}
Invoke-RestMethod 'https://api.github.com/user/emails' -Headers $headers
Each REST API for each different system or service tends to take a slightly different approach to authentication, authorization, calling methods, and even details like paging. However, despite the differences there, the lessons learned using one API are still useful when attempting to write code for another.
REST is an extremely popular style these days. Before REST became prominent, services built using SOAP were common.
Working with SOAP
Unlike REST, which is an architectural style, SOAP is a protocol. It is perhaps reasonable to compare working with SOAP to importing a .NET assembly (DLL) to work with the types inside. As a result, a SOAP client is much more strongly tied to a server than is the case with a REST interface.
SOAP uses XML to exchange information between the client and server.
Finding a SOAP service
SOAP-based web APIs have become quite rare, less popular by far than REST. The examples in this section are based on a simple SOAP service written for this book.
The service is available on GitHub as a Visual Studio solution. The solution is also available in the GitHub repository containing the code examples used in this chapter: https://github.com/indented-automation/SimpleSOAP.
The solution should be downloaded, opened in Visual Studio (2015 or 2017, Community Edition or better), and debugging should be started by pressing F5. A browser page will be opened, which will show the port number the service is operating on. A 403 error may be displayed; this can be ignored.
Localhost and a port
Throughout this section, localhost and a port are used to connect to the web service. The port is set by Visual Studio when debugging the simple SOAP web service and must be updated to use these examples.
This service is not well-designed; it has been contrived to expose similar patterns in its method calls to those seen in real SOAP services.
A ReadMe file accompanies the project. Common problems running the project will be noted there.
The discovery-based approaches explored in this section should be applicable to any SOAP-based service.
SOAP in Windows PowerShell
The New-WebServiceProxy examples that follow apply to Windows PowerShell only. The New-WebServiceProxy command is not available in PowerShell 7.
New-WebServiceProxy
The New-WebServiceProxy command is used to connect to a SOAP web service. This can be a service endpoint, such as a .NET service.asmx URL, or a WSDL document.
The web service will include methods and may also include other object types and enumerations.
The command accesses a service anonymously by default. If the current user should be passed on, the UseDefaultCredential parameter should be used. If explicit credentials are required, the Credential parameter can be used.
By default, New-WebServiceProxy creates a dynamic namespace. This is as follows:
PS> $params = @{
>> Uri = 'http://localhost:62369/Service.asmx'
>> }
PS> $service = New-WebServiceProxy @params
PS> $service.GetType().Namespace
Microsoft.PowerShell.Commands.NewWebserviceProxy.AutogeneratedTypes.WebServiceProxy4__localhost_62369_Service_asmx
The dynamic namespace is useful as it avoids problems when multiple connections are made to the same service in the same session.
To simplify exploring the web service, a fixed namespace might be defined:
$params = @{
Uri = 'http://localhost:62369/Service.asmx'
Namespace = 'SOAP'
}
$service = New-WebServiceProxy @params
The $service object returned by New-WebServiceProxy describes the URL used to connect, the timeout, the HTTP user agent, and so on. The object is also the starting point for exploring the interface; it is used to expose web services' methods.
Methods
The methods available may be viewed in several ways. The URL used can be visited in a browser or Get-Member may be used. A subset of the output from Get-Member follows:
PS> $service | Get-Member
Name MemberType Definition
---- ---------- ----------
GetElement Method SOAP.Element GetElement(string Name)
GetAtomicMass Method string GetAtomicMass(string Name)
GetAtomicNumber Method int GetAtomicNumber(string Name)
GetElements Method SOAP.Element[] GetElements()
GetElementsByGroup Method SOAP.Element[] GetElementsByGroup(SOAP.Group group)
GetElementSymbol Method string GetElementSymbol(string Name)
SearchElements Method SOAP.Element[] SearchElements(SOAP.SearchCondition[] searchConditions)
The preceding GetElements method requires no arguments and may be called immediately, as shown here:
PS> $service.GetElements() | Select-Object -First 5 | Format-Table
AtomicNumber Symbol Name AtomicMass Group
------------ ------ ---- ---------- -----
1 H Hydrogen 1.00794(4) Nonmetal
2 He Helium 4.002602(2) NobleGas
3 Li Lithium 6.941(2) AlkaliMetal
4 Be Beryllium 9.012182(3) AlkalineEarthMetal
5 B Boron 10.811(7) Metalloid
Methods requiring string or numeric arguments may be similarly easy to call, although the value the method requires is often open to interpretation. In this case, the Name argument may be either an element name or an element symbol:
PS> $service.GetAtomicNumber('oxygen')
8
PS> $service.GetAtomicMass('H')
1.00794(4)
Whether the web service is SOAP or REST, using the service effectively is dependent on being able to locate the service documentation.
Methods and enumerations
The GetElementsByGroup method shown by Get-Member requires an argument of type SOAP.Group as the following shows:
PS> $service | Get-Member -Name GetElementsByGroup
Name MemberType Definition
---- ---------- ----------
GetElementsByGroup Method SOAP.Element[] GetElementsByGroup(SOAP.Group…
SOAP.Group is an enumeration, as indicated by the BaseType:
PS> [SOAP.Group]
IsPublic IsSerial Name BaseType
-------- -------- ---- --------
True True Group System.Enum
The values of the enumeration may be shown by running the GetEnumValues method:
PS> [SOAP.Group].GetEnumValues()
Actinoid
AlkaliMetal
AlkalineEarthMetal
Halogen
Lanthanoid
Metal
Metalloid
NobleGas
Nonmetal
PostTransitionMetal
TransitionMetal
PowerShell will help cast to enumeration values; a string value is sufficient to satisfy the method:
PS> $service.GetElementsByGroup('Nonmetal') | Format-Table
AtomicNumber Symbol Name AtomicMass Group
------------ ------ ---- ---------- -----
1 H Hydrogen 1.00794(4) Nonmetal
6 C Carbon 12.0107(8) Nonmetal
7 N Nitrogen 14.0067(2) Nonmetal
8 O Oxygen 15.9994(3) Nonmetal
15 P Phosphorus 30.973762(2) Nonmetal
16 S Sulfur 32.065(5) Nonmetal
34 Se Selenium 78.96(3) Nonmetal
If the real value of the enumeration must be used, it may be referenced as a static property of the enumeration:
$service.GetElementsByGroup([SOAP.Group]::Nonmetal) | Format-Table
It is relatively common for a method to require an instance of an object provided by the SOAP interface.
Methods and SOAP objects
When working with SOAP interfaces, it is common to encounter methods that need instances of objects presented by the SOAP service. The SearchElements method is an example of this type.
The SearchElements method requires an array of SOAP.SearchCondition as an argument. This is shown in the following by accessing the definition of the method:
PS> $service.SearchElements
OverloadDefinitions
-------------------
SOAP.Element[] SearchElements(SOAP.SearchCondition[] searchConditions)
An instance of SearchCondition may be created as follows:
$searchCondition = [SOAP.SearchCondition]::new()
Exploring the object with Get-Member shows that the operator property is another type from the SOAP service. This is an enumeration, as shown here:
PS> [SOAP.ComparisonOperator]
IsPublic IsSerial Name BaseType
-------- -------- ---- --------
True True ComparisonOperator System.Enum
A set of search conditions may be constructed and passed to the method:
$searchConditions = @(
[SOAP.SearchCondition]@{
PropertyName = 'AtomicNumber'
Operator = 'ge'
Value = 1
}
[SOAP.SearchCondition]@{
PropertyName = 'AtomicNumber'
Operator = 'lt'
Value = 6
}
)
$service.SearchElements($searchConditions)
Overlapping services
When testing a SOAP interface, it is easy to get into a situation where New-WebServiceProxy has been called several times against the same web service. This can be problematic if using the Namespace parameter.
Consider the following example, which uses two instances of the web service:
$params = @{
Uri = 'http://localhost:62369/Service.asmx'
Namespace = 'SOAP'
}
# Original version
$service = New-WebServiceProxy @params
# New version
$service = New-WebServiceProxy @params
$searchConditions = @(
[SOAP.SearchCondition]@{
PropertyName = 'Symbol'
Operator = 'eq'
Value = 'H'
}
)
In theory, there is nothing wrong with this example. In practice, the SOAP.SearchCondition object is created based on the original version of the service created using New-WebServiceProxy. The method is, on the other hand, executing against the newer version.
As the method being called and the type being used are in different assemblies, an error is shown; this is repeated in the following:
PS> $service.SearchElements($searchConditions)
Cannot convert argument "searchConditions", with value: "System.Object[]", for "SearchElements" to type
"SOAP.SearchCondition[]": "Cannot convert the "SOAP.SearchCondition" value of type "SOAP.SearchCondition" to type
"SOAP.SearchCondition"."
At line:1 char:1
+ $service.SearchElements($searchConditions)
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [], MethodException
+ FullyQualifiedErrorId : MethodArgumentConversionInvalidCastArgument
It is still possible to access the second version of SearchCondition by searching for the type, then creating an instance of that:
$searchCondition = ($service.GetType().Module.GetTypes() |
Where-Object Name -eq 'SearchCondition')::new()
$searchCondition.PropertyName = 'Symbol'
$searchCondition.Operator = 'eq'
$searchCondition.Value = 'H'
$searchConditions = @($searchCondition)
$service.SearchElements($searchConditions)
However, it is generally better to avoid the problem by allowing New-WebServiceProxy to use a dynamic namespace. At which point, an instance of the SearchCondition may be created, as the following shows:
('{0}.SearchCondition' -f $service.GetType().Namespace -as [Type])::new()
SOAP in PowerShell 7
Windows PowerShell can use the New-WebServiceProxy command. In PowerShell 7, requests can be manually created and sent using Invoke-WebRequest.
Getting the WSDL document
The Web Services Description Language (WSDL) document for the web service contains details of the methods and enumerations it contains. The document can be requested as follows:
$params = @{
Uri = 'http://localhost:62369/Service.asmx?wsdl'
}
[Xml]$wsdl = Invoke-WebRequest @params | Select-Object -ExpandProperty Content
The document shows the methods that can be executed and the arguments for those methods. Obtaining a list requires a bit of correlation.
Discovering methods and enumerations
You can use the WSDL document to discover the methods available from the SOAP service. The default view of the document presents a list of methods. Clicking on a method will show an example of the expected header and body values.
It is possible to retrieve the methods in PowerShell dynamically as well. Two SOAP versions are presented by the service. SOAP 1.2 is used in the following example, although both will show the same information in this case:
$xmlNamespaceManager = [System.Xml.XmlNamespaceManager]::new($wsdl.NameTable)
# Load everything that looks like a namespace
$wsdl.definitions.PSObject.Properties |
Where-Object Value -match '^http' |
ForEach-Object {
$xmlNamespaceManager.AddNamespace(
$_.Name,
$_.Value
)
}
$wsdl.SelectNodes(
'/*/wsdl:binding[@name="ServiceSoap12"]/wsdl:operation',
$xmlNamespaceManager
).ForEach{
[PSCustomObject]@{
Name = $_.name
Inputs = $wsdl.SelectNodes(
('//s:element[@name="{0}"]/*/s:sequence/*' -f $_.name),
$xmlNamespaceManager
).ForEach{
[PSCustomObject]@{
ParameterName = $_.name
ParameterType = $_.type -replace '.+:'
}
}
Outputs = $wsdl.SelectNodes(
('//s:element[@name="{0}Response"]/*/*/s:element/@type' -f
$_.name),
$xmlNamespaceManager
).'#text' -replace '.+:'
SoapAction = $_.operation.soapAction
}
}
The preceding script shows a rough list of the parameters required and value types returned for each of the methods. For example, the GetElement method expects a string argument and will return an Element object:
Name Inputs Outputs
---- ------ -------
GetElement {@{ParameterName=Name; ParameterType=string}} Element
Enumeration values are also exposed in the WSDL. Continuing from the previous example, they may be retrieved using the following:
$wsdl.SelectNodes(
'/*/*/*/s:simpleType[s:restriction/s:enumeration]',
$xmlNamespaceManager
).ForEach{
[PSCustomObject]@{
Name = $_.name
Values = $_.restriction.enumeration.value
}
}
The Group and ComparisonOperator enumerations will be displayed.
Running methods
You can use Invoke-WebRequest to execute methods by providing a SOAP envelope in the body of the request. The response is an XML document that includes the results of the method. The envelope includes the web service URI, http://tempuri.org:
$params = @{
Uri = 'http://localhost:62369/Service.asmx'
ContentType = 'text/xml'
Method = 'POST'
Header = @{
SOAPAction = 'http://tempuri.org/GetElements'
}
Body = '
<soapenv:Envelope
xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
<soapenv:Body>
<GetElements />
</soapenv:Body>
</soapenv:Envelope>
'
}
$webResponse = Invoke-WebRequest @params
$xmlResponse = [Xml]$webResponse.Content
$body = $xmlResponse.Envelope.Body
$body.GetElementsResponse.GetElementsResult.Element
As the preceding code shows, the response content is specific to the method that was executed.
If a method requires arguments, these must be passed in the body of the request. In the following example, the argument is a single string:
$params = @{
Uri = 'http://localhost:62369/Service.asmx'
ContentType = 'text/xml'
Method = 'POST'
Header = @{
SOAPAction = 'http://tempuri.org/GetElement'
}
Body = '
<soapenv:Envelope
xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"
xmlns="http://tempuri.org/">
<soapenv:Body>
<GetElement>
<Name>Oxygen</Name>
</GetElement>
</soapenv:Body>
</soapenv:Envelope>
'
}
$webResponse = Invoke-WebRequest @params
$xmlResponse = [Xml]$webResponse.Content
$body = $xmlResponse.Envelope.Body
The body shows the object returned by the method:
PS> $body.GetElementResponse.GetElementResult
AtomicNumber : 8
Symbol : O
Name : Oxygen
AtomicMass : 15.9994(3)
Group : Nonmetal
Requests requiring an enumeration value, such as Get
The name of the argument used above correlates with the name and value shown in the WSDL:
<s:element minOccurs="0" maxOccurs="1" name="Name" type="s:string"/>
The preceding method expects a string. If an enumeration value is required, it can be described as a string in the XML envelope.
More complex types can be built based on following the expected structure of the arguments, or following the examples provided when browsing the Service.asmx file. The following example includes two SearchCondition objects:
$params = @{
Uri = 'http://localhost:62369/Service.asmx'
ContentType = 'text/xml'
Method = 'POST'
Body = '
<soapenv:Envelope
xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"
xmlns="http://tempuri.org/">
<soapenv:Body>
<SearchElements>
<searchConditions>
<SearchCondition>
<PropertyName>AtomicNumber</PropertyName>
<Value>1</Value>
<Operator>ge</Operator>
</SearchCondition>
<SearchCondition>
<PropertyName>AtomicNumber</PropertyName>
<Value>6</Value>
<Operator>lt</Operator>
</SearchCondition>
</searchConditions>
</SearchElements>
</soapenv:Body>
</soapenv:Envelope>
'
}
$webResponse = Invoke-WebRequest @params
$xmlResponse = [Xml]$webResponse.Content
$body = $xmlResponse.Envelope.Body
$body.SearchElementsResponse.SearchElementsResult.Element
New-WebServiceProxy in Windows PowerShell took away some of the difficulty of defining the SOAP envelope, but in most cases, it is not difficult to create.
Summary
This chapter explored the use of Invoke-WebRequest and how to work with and debug SSL negotiation problems.
Working with REST explored simple method calls, authentication, and OAuth negotiation, before exploring REST methods that require authenticated sessions.
SOAP is hard to find these days; a sample project was used to show how the capabilities of a SOAP service might be discovered and used.
Chapter 14, Remoting and Remote Management, explores remoting and remote management.