Real World: Tips for Troubleshooting the Sign-In Process

The following list presents some tips for troubleshooting the sign-in process, ranging from basic tips to strategies for simplifying the discovery of problems by eliminating complexity.

The basics

Simplify

To isolate the component that is failing during the sign-in process, it is a best practice to make the sign-in process as simple as possible. Start with an internal client on the local area network (LAN) before testing with a remote user.

How to change to a simple configuration

—Jason Epperly

Escalation Engineer, Product Support Services

Extracting Errors from Communicator Logs

If you are using Communicator logging to troubleshoot issues, you can extract "error" entries quickly by using FINDSTR. The syntax is as follows:

findstr /I error <path to UCCP log file> <output file>

For example, you might use the following command:

findstr /I error %userprofile%	racingCommunicator-uccp-0.uccplog errors.txt

The errors.txt file contains all log entries tagged as ERROR. You can then find the entry in the source log file and place the error in its proper context. You should then be able to discern what happened before and after the error was thrown.

—Joel Schaeffer

Escalation Engineer, Product Support Services

Direct from the Source: Diagnosing Connection Failures (Load Balancers and Firewalls)

When diagnosing connection issues, my first guess is that a load balancer is involved, especially if the connections close at regular intervals (5 minutes is a popular default). When a network device like a load balancer causes connection problems, both servers will see that the other end has closed the connection. The Winsock error code for the remote host closing the connection is WSAECONNRESET (10054 in decimal, 2746 in hexadecimal, or 0x80002746 as a HRESULT in the logs). In a network trace, both sides will see a TCP segment with the RST (TCP 'reset') flag set. In defense of load balancers, most are optimized for Web traffic that has short-lived connections (unlike connections between Office Communications Server servers); therefore, the default length of time before a load balancer assumes that a connection is "dead" tends to be lower than the value needed by the Office Communications Server.

For firewall issues, the Winsock WSAETIMEDOUT error code (10060 in decimal, 274C in hexadecimal, or 0x8000274C as a HRESULT in the logs) is usually a good indicator. As firewalls tend to drop the network packets they are blocking, connection attempts that fail will simply time out. If you know that the IP address being tried is correct and that the server at the other end is running, you can be more confident that a firewall is blocking the traffic.

—James Undery

Software Design Engineer II

Direct from the Source: Diagnosing Problems with Group Expansion Failures

For help diagnosing Group Expansion failures when using Communicator, use Internet Explorer (IE) to navigate to https://<webfqdn>/GroupExpansion/Int/service.asmx (replace <webfqdn> with the server FQDN that you are using, such as: srv.contoso.com). Using IE, select the Tools, then Internet Options, then Advanced menu item, and then uncheck Show Friendly HTTP Error Messages so that any internal failures also show up in IE on the bottom left of the window.

Communicator obtains the Group Expansion URLs (which are a pool-level setting) through in-band provisioning:

Whenever a user wants to expand a directory list (DL), the Group Expansion URLs are connected. Debugging DL expansion failures involves directly accessing these URLs from within IE so that errors come up front (Internet Information Services [IIS] errors, access errors, proxy errors, or service failures or crashes).

Fully qualified domain names

When a pool is created, the administrator must mention the Web farm FQDN that he will use in the deployment. Two FQDNs must be mentioned here: InternalWebFqdn and ExternalWebFqdn.

InternalWebFqdn is used by intranet clients. This FQDN is either the same as the SIP server (Standard Edition case) or the pool load balancer (Enterprise Edition case #1), or it can be different (Enterprise Edition case #2 in which IIS servers are deployed behind a separate load balancer for scalability purposes).

ExternalWebFqdn is used by remote clients. This is the FQDN of the reverse proxy (or farm) in the DMZ. The reverse proxy, in turn, proxies requests to the internal Web components server.

—Kiran Kulkarni

Software Design Engineer II, OCS Product Group

Direct from the Source: Certificate Issues

What do you do if you see a 401 Unauthorized error in a Validation Wizard or a The Client Did Not Supply A Certificate error in the event log that causes conferencing to fail?

—Sankaran Narayanan

Senior Software Design Engineer, OCS Server

Real World: Useful Database Query for Contact Lists

We used various diagnostic stored procedures internally while developing and deploying the back-end database used by Office Communications Server. One of these procedures returns aggregate statistical information about the nature of the contact lists maintained by all users that are homed in the database. The following command can be executed with osql.exe or with SQL Management Studio against the Real-Time Communications (RTC) database of an Office Communications Server back-end SQL instance:

exec DiagShowContactDistribution

Executing this command returns information such as the following:

MinNumContacts MaxNumContacts AvgNumContacts StdNumContacts
-------------- -------------- -------------- ---------------
1              250            45             37.59

ModeNumContacts CountWithMode
--------------- -------------
28              985

BucketCriteria NumSamples  Percent  Histogram           CumulativePercent
-------------- ----------- -------- ------------------- -------------------
<=          5  3237        5.4      xx                  5.4
<=         25  17840       29.6     xxxxxxxxxxxx        35.0
<=         50  18971       31.5     xxxxxxxxxxxxx       66.5
<=         75  10196       16.9     xxxxxxx             83.5
<=        100  4953        8.2      xxx                 91.7
<=        125  2377        3.9      xx                  95.6
<=        150  1206        2.0      x                   97.7
<=        175  682         1.1                          98.8
<=        200  350         0.6                          99.4
<=        300  381         0.6                          100.0
all others     0           0.0                          100.0

This result displays min/max/average/std deviation/mode as well as a graphical histogram of the distribution of the number of contacts maintained by each user. Of course, it is worth noting that the use of stored procedures is not officially supported by Microsoft because these procedures might change or be withdrawn entirely as the product is updated.

—Shaun Cox

Principal Development Manager, OCS Server

Direct from the Source: Multi-Party IM Establishment Failures

When attempting to bring multiple users into an instant messaging session, several causes for failures are possible. Some tips and potential causes are called out here:

—Sankaran Narayanan

Senior Software Design Engineer, OCS Server

Real World: Investigating Content Download Issues

One of the most common problems we've seen when the console is connected to a conference hosted on a data multipoint control unit (MCU) is the failure to download the content of a PowerPoint/Word document or the content of an uploaded file. In almost all of these cases, the problem is not in the data MCU but in how IIS and the client IE are configured.

The problem

How can you recognize the problem? There are two ways to recognize the download problem, and in both cases, you start by connecting the console to a meeting.

In both cases, the failure is induced by the same configuration problem that you might encounter in an IIS server and/or IE on the client/console machine.

Summary of steps to follow

What's next? In summary, these are the steps to follow:

Let's look at these steps in detail.

Step 1: Grab console tracing information

Let's reproduce the problem by uploading a document (the first bullet under the heading "The problem" in this sidebar).

You must first take a look into the console trace file, which can be found in C:Documents and Settings<user>Local SettingsTemp. You can easily access this folder by running cd /d %temp% in the command console.

In this folder, you'll find few console trace files (run in command console: dir pwconsole-debug*.txt). In Notepad, open the most recent trace file. At the end of the file, you should find a trace line generated by the Downloaded::addRequest() method that looks like the following:

[MC] 21:06:10:064 GMT [THREAD 4888]  [I] Downloader::addRequest()
    - Found previously failed request, will not download https://srv.contoso.com/
etc/place/null/FileTree/IE6HRFPCBJ3K1CCUC1
    HH75UPN8Q/6a24bb55f381433285cc878baa11ed3a/slidefiles/
    xc75dbd0baa6a.epng

This trace line indicates that the console failed to download the content from a specific URL. The URL is printed at the end of the trace line, starting with HTTPS. When debugging the problem, please copy the URL from the actual log for use later in the investigation during Step 2.

Step 2: Was the file uploaded?

The next step is to check whether the file was uploaded by the data MCU. The file is uploaded in 99% of the cases, but let's double-check before going further with our investigation. To do this check, you must go on the machine on which the IIS runs, where you will open the IIS Management Console by clicking Start, then Administrative Tools, and then Internet Information Services (IIS) Manager.

Open the nodes in the left panel tree, Internet Information Services, then <server machine>, then Web Sites, then Default Web Services and there should be a node with a name that matches the URL obtained from the console traces at the end of Step 1 (the "etc" portion of the URL). From that point in the URL on, you can continue to open the nodes in the IIS Management console: Etc, then Null, then FileTree, then <organizerguid>, then <consoleguid> and then sidefiles. Please note that you get the <organizerguid> and <consoleguid> from the URL that you copied at the end of Step 1 (in our example, the <organizeguid> is IE6HRFPCBJ3K1CCUC1HH75UPN8Q and the <conferenceguid> is 6a24bb55f381433285cc878baa11ed3a). Click the sidefiles node, and you should see a list of files in the right-panel list. One of these files should be the .epng file from the end of the URL (in our example, the file is xc75dbd0baa6a.epng).

If the .epng file is not there, then there is likely a bug. You should contact PSS to report the problem observed with the data MCU. That's all you can do, and you don't need to read further!

If the file is there, the data MCU was able to upload the file, and your problem exists in the IIS and/or client IE. Keep reading this chapter!

Step 3: Is the Web proxy the problem?

In a test topology, your IIS server is usually not published in the DNS. If your IE client is configured to resolve the Web HTTP URL by using a Web proxy, IE reports a failure when you pass in a URL like the one you obtained from the console trace file because the server FQDN cannot be resolved (in our case, the server FQDN is srv.contoso.com).

Open IE and go to Tools, then Internet Options, then Connections, and then LAN Settings. You now have two options:

To verify that all is well, let's see whether IE can browse for our URL. Open IE, and paste in the URL that you copied from the trace file. Change it from https to http, remove everything after the server FQDN (the URL in IE will now look like http://srv.contoso.com), and then browse for it. If you see a page that says "Under construction," then you have properly configured IE. If you don't see this page (you will probably see a page with the message, "Internet Explorer cannot display the webpage"), you must look at the IIS management console (opened in Step 2) as well because the server is probably not running.

Let's assume that you saw the "Under construction" page. Change the URL from http to https (the URL in IE will look like https://srv.contoso.com). If you see a page that says "Under construction," then the IIS server is already configured for HTTPS. The console problem should be fixed as well, and you don't need to go further with this document. If you don't see this page (you will probably see a page with the message "Internet Explorer cannot display the webpage"), then the IIS server is running but is not configured for HTTPS. You must take a look at the IIS server configuration.

To double-check that all is well, open the IE, paste in the entire URL you obtained from the console trace file, and browse for it. You should now see a dialog box asking you whether you want to download the file, which means that everything is fine (click Cancel in the dialog box). The console is expected to run properly, and you should see the conference content in its main window.

You should read the following paragraphs only if you determined that the IIS server is not running or is not configured for HTTPS.

Step 4: Is IIS running?

Let's first make sure that IIS is running. In the same IIS Management console that you opened in Step 2, take a look at the <machine>, <default web site> icons in the left-panel tree. These icons should not have any small red circles drawn on top of them, for these circles indicate that you have a big problem within IIS. First, try to restart IIS by right-clicking the machine icon. From the context menu, select All Tasks, then Restart IIS, which should restart IIS. Hopefully, everything will then be clean. If you still see the red marks, I advise you to reinstall IIS and then reinstall the servers. If the problem persists, you should contact PSS.

From this point onward, we'll assume that the IIS is running just fine.

Step 5: Is IIS configured for TLS?

Let's make sure that IIS is configured to listen to TLS/HTTPS requests. Note that the URL you copied from the console trace file is an HTTPS address (it starts with https://). When the console downloads the conference content, it does so by using HTTPS, and therefore the server must be configured to accept such requests. IIS does not accept such requests by default, and there is a chance that you forgot to configure your IIS for TLS/HTTPS.

To check whether IIS is allowed to accept HTTPS requests (and listens to port 443), you must access the same IIS management console (opened in Step 2) and right-click Default Web Site. Select Properties, then select Directory Security and then Server Certificate. Follow the steps in the wizard. In the second wizard step, select Assign An Existing Certificate. (You must first have installed a Web certificate on the server machine. Check the deployment guide to see how you can obtain one.) Select the certificate in the third step, and keep the port set to 443 in the fourth step. Save the settings by clicking Finish in the wizard and then clicking OK on the Properties page. IIS should now be configured for HTTPS (you don't need to restart the service). To confirm that everything is working, repeat the procedures listed in the "Step 3: Is the Web Proxy the Problem?" section in which you tried the HTTPS URL. IE should be able to display the "Under construction" page even when the URL starts with https://.

You should now have a console that displays the uploaded content.

—Vlad Eminovici

Senior Development Lead

Real World: Simple Way to Set Maximum Attendees

There is a simple way to set the maximum number of attendees that your end users can invite to Office Communications Server Audio/Video/Web conferences in Office Communicator and Live Meeting client. Launch the Office Communications Server MMC. Right-click the Forest, click Properties, and select Global Properties. Select the Meetings tab, select the meeting policy you'd like to modify, and then click Edit. Change the Maximum Meeting Size field to the desired value.

The maximum meeting size number applies to both Audio/Video/Web conferences through the Live Meeting client as well as Audio/Video conferences conducted through Office Communicator. Though 250 attendees per session has been the design goal for Office Communications Server conferences, a number of other consideration points must be taken into account during the planning/architectural phase of your deployment. Refer to the Office Communications Server Planning Guide documentation for more details. You certainly do not want to assign a meeting policy that allows a meeting size of 250 attendees to all of your users, given the 5 percent concurrency model on Office Communications Server's MCUs. Be sure to create various meeting policies and assign the appropriate meeting policy to each user and group based on their needs and job functions.

—Ali Rohani

Technical Specialist, Unified Communications, Canada

Direct from the Source: Diagnosing Service Start Failures

This is probably the single most common failure that people encounter. Some useful hints are the following:

It is also worth mentioning that when things go wrong, Office Communications Server services usually throw more than one event log. Generally speaking, the first event log contains the most information, but it is beneficial to go through all of the error event logs. I often see people look at the top-most event log and give up because it is a very general event log that might not point out enough details on its own to completely identify the problem.

—Nirav Kamdar

Senior Development Lead, OCS Server

Direct from the Source: Diagnosing AD/SQL/WMI Failures

Active Directory, SQL database, and Windows Management Instrumentation (WMI) failures are typically reported during service startups. Use wbemtest.exe to access the class about which the service is complaining. It is beneficial to access one class from each of the three Office Communications Server repositories (Active Directory, SQL, and local WMI repositories). The following is one set of samples:

Note that the SQL query can be executed only on roles that belong to a pool (therefore, it won't work on Archiving Server, Access Edge Server, and so forth). Similarly, the Active Directory query won't work on an Access Edge Server. Also, the class referenced in the local WMI query must be changed for Archiving Server and Access Edge Server because this class is installed on all roles except these two.

—Nirav Kamdar

Senior Development Lead, OCS Server

Direct from the Source: Diagnosing Active Directory Errors

If you receive an Active Directory error (whose symbolic name starts with ERROR_DS_*, for example, ERROR_DS_NO_SUCH_OBJECT) during deployment, the first thing to check is which account credentials are being used. Run whoami.exe /groups, and make sure that the RTCUniversalServerAdmins group shows up in your access token. If it does, remember to use the Windows Server 2003 Resource Kit Tools Nltest.exe and NetDiag.exe to rule out problems with Domain Controllers (DCs) and the local machine network.

Nltest. exe /DsGetDC:[Root Domain FQDN] /GC
Nltest. exe /DsGetDC:[Root Domain FQDN] /PDC
Nltest. exe /DsGetDC:[Local Domain FQDN] /PDC
NetDiag.exe

The root and local domain FQDNs (shown in the following example as [Root Domain FQDN] and [Local Domain FQDN]) must be replaced with the root DC FQDN for the top-level domain and DC FQDN for the domain from which you are running the commands. Examples of domain names might be contoso.com for the root domain and eng.contoso.com for the local domain, with the command sequence being:

Nltest. exe /DsGetDC:contoso.com /GC
Nltest. exe /DsGetDC:contoso.com /PDC
Nltest. exe /DsGetDC:eng.contoso.com /PDC
NetDiag.exe

—Yong Zhao

Software Developer on the Office Communications Server team

Direct from the Source: Diagnosing Setup Failures

Setup logs are located under %temp% and have the .log extension (for Microsoft Installer Package [MSI] logs) or.html extension (for activation/prep failures). In MSI logs, look for the phrase "value 3" (MSI errors are cryptic, and this is a quick method to locate them). The .html logs are self explanatory and can be expanded to the point of the failure. LcsCmd's CheckXXState is also usually very helpful. For example, to check whether the AV (Audio/Video) MCU is activated, you can run LcsCmd /mcu /action:CheckLcServerState /role:AVMCU.

—Nirav Kamdar

Senior Development Lead, OCS Server

Real World: Installation and Tool Tips

The following tips can help with install/uninstall issues as well as provide useful information you should know when working with some of the Office Communications Server tools and with the Office Communications Server Resource Kit Tools.

Installation and uninstallation tips

Always uninstall the Office Communications Server product by using Control Panel->Add/Remove Programs. If the uninstall fails, then resolve the error that is thrown and try to uninstall again. If the uninstall still fails, then use the Microsoft Discussion Groups as the next-best solution (go to http://www.microsoft.com/office/community/en-us/default.mspx, and select the appropriate product from the list on the left). Ultimately, you might need to use the LcsVerifyClean.vbs script in the Office Communications Server 2007 Resource Kit Tools to clean up the installation. Never use any other cleanup utilities or manually clean up the state (registry, file system, WMI, SQL, permissions, and so forth).

MSI install/uninstall failures throw errors in the Application Event Log. You can use that information along with the log of installation/uninstallation to diagnose the problem yourself.

Tool tips

On 64-bit machines, launching the 32-bit MMC snap-in requires a switch, such as compmgmt.msc -32 or wrtcsnap2.msc -32 or dsa.msc -32.

Many Office Communications Server 2007 Resource Kit Tools now have a dependency on the Service Pack 1 C Run-Time Libraries (CRTs). If you don't install the SP1 CRTs, then running the application might result in a Failed To Initialize Properly message. If this happens, install SP1 CRTs from http://www.microsoft.com/downloads by searching for Microsoft Visual C++ 2005 SP1 Redistributable Package. Both x86 and x64 versions should be available. The direct link to the x86 download is http://www.microsoft.com/downloads/details.aspx?FamilyID=200B2FD9-AE1A-4A14-984D-389C36F85647&displaylang=en.

If the component that you want to trace does not appear in the OCSLogger components list, click Advanced Options->Additional Components. Click the empty space under executable files. Type the name of the file that you want to trace (such as wsipsetp.dll for LcsSetupCustomActions), and click OK. LcsSetupCustomActions should now appear in the components list.

—Nirav Kamdar

Senior Development Lead, OCS Server