NTLM (NT LAN Manager) is a Microsoft protocol which is still very frequently used in web server authentication scenarios, especially within an enterprise. It is enabled by the use of LDAP (invariably Active Directory). When used for authentication in front of servers that use NTLM, enabling SSO on the NetScaler makes very good sense.
When NTLM SSO fails via the NetScaler, the usual User experience will be that they see two 401 dialog boxes followed by a 403 error.
NTLM is a challenge-based protocol. The exchange involves the server challenging the client to prove its identity in order to be able to see the resource it is requesting.
The following screenshot is an exchange between the NetScaler SNIP and a web server that has NTLM Authentication enabled. You can filter this communication using http || ntlmssp:
Now let's examine each of the steps in more detail:
- First, NetScaler forwards the client's request for an object, and instead of the usual 200OK receives a 401 Unauthorized from the server. Thus, it knows that authentication is in place. The 401 Unauthorized response also contains the WWW-Authenticate: NTLM header, which indicates what authentication mechanism is in place:
- NetScaler then responds to this challenge for credentials by sending information about itself and a number of negotiate flags that clarify to the server what versions of NTLM it can speak. In NTLM lingo, this is called a Type1 message:
- Upon receiving the Type1 message from NetScaler, the server responds with a Type2 message. This response will contain two important details – that of the target domain and a challenge:
- Here, the NetScaler uses the challenge it received in the previous message to construct its next request, which will contain everything needed for the authentication to succeed:
- The server, happy with the authentication details, responds with 200OK and the requested resource.
Traces and counters combined with the aaad.debug prove sufficient in most cases to troubleshoot NTLM issues. Consider the following four step approach to troubleshoot them:
- Obtain a trace while you have the User login. Since NTLM is web-based, you can also use a HTTP tracing tool such as Fiddler to see what response codes the server is sending. Also check that the WWW-Authenticate: NTLM header is indeed sent by the server to ensure the right protocol is being used.
- Look for the SSO-related counters in the following screenshot at the time of the issue. You can use the shell command
nsconmsg –g sso –d currentto see if success or failure ones are going up. The following screenshot shows a successful exchange:
- Look at the
aaad.debuginfo to identify reasons for the failure. - Load balancing can sometimes be an issue if you are pointing to a VIP for authentication and if the handshake is incorrectly being broken up and sent to different servers. If you are seeing this, consider configuring persistence such as Source IP or Cookie.