When I wrapped up content creation for my session on Troubleshooting Exchange 2013 for IT/Dev Connections this September in Las Vegas, I soon realized I had a problem. I was tasked with delivering a 60min session but I had created enough content for an entire day of delivery!

I managed to get the session down to around 60min but I still kept all the content in the deck. There’s a ton of information in the slide notes as well as many hidden slides, so be sure to download the deck afterwards if you’re an attendee. If you’re not an attendee, then how’s a last minute trip to Vegas sound?

However, I decided to use this blog to expand on a topic I just couldn’t do justice within a 60min time frame. Hopefully this can give people a look at the type of content being presented at Exchange Connections as well as a starting point if they’d like to grow their troubleshooting skills. So in this post I’d like to cover common break-fix issues seen with Client Access Servers; even though technically some of these components live in the Mailbox Server role now.

Overview

The first step to troubleshooting any technology, I feel, is to understand the functionality of its core components during normal operation. Often time people are given a set of tools to be used in troubleshooting but never truly understand how to interpret the data they’re looking at. Similar to how using NetMon will be of little use to someone who doesn’t have a solid understanding of TCP/IP, looking at Exchange Client Access or IIS data will not prove useful if you do not understand how each of the components interact with each other. Let’s begin by looking at IIS.

What we see above is the IIS Manager on my Server 2012 R2 Exchange 2013 CU6 multi-role server. We find the various web sites as well as the Application Pools that correspond to each application like ActiveSync, PowerShell, or OWA. Because this server is multi-role (has both CAS & Mailbox Roles installed) you will see two separate Exchange web sites:

Default Web Site = Client Access Server Role

Exchange Back End=Mailbox Server Role

The two main services associated with IIS are the IIS Admin Service (inetinfo.exe) & the World Wide Web Publishing Service (w3wp.exe). To oversimplify it, inetinfo.exe corresponds to IIS configuration information whereas w3wp.exe corresponds to each of the various Application Pools. After changing IIS configuration information (like Auth Settings, etc.), the IIS Admin Service will typically be what you’ll want to restart. Whereas, if a particular application still isn’t updating after you’ve made a change (like OWA or ActiveSync) then you may need to Recycle that Application Pool & at worst, restart the WWW Publishing Service.

However, in many cases it’s recommended to simply stop/start the website or recycle the application pool rather than restarting the services or using iisreset (Reference-A Reference-B Reference-C). This is because it’s possible IIS has not saved the necessary changes in time & those changes could be lost by a forcible service restart. Starting/Stopping the websites, recycling the application pools, or using the “/noforce” switch for iisreset is preferred. However, sometimes killing a service is all you can do in a troubleshooting scenario.

Web Sites & Application Pools in Exchange 2013

When troubleshooting IIS, I commonly find myself looking at the Web Site Bindings. These are what “bind” an IP Address, Port Number, Host Name, & (potentially) a Certificate to the web site. Let’s look at the bindings using both PowerShell as well as the GUI.

Using the above series of commands (reference) I was able to import the IIS PowerShell Module & query the bindings of my two web Sites in IIS. I’ve found that using PowerShell is a very handy way to query this data fairly quickly. It’s also useful for when you need to send a customer a set of commands they can run & send the data back to you. Here’s a few of my preferred information gathering commands:

The above series of commands has me navigating to the “Default Web Site” & viewing the various Applications & Virtual Directories underneath it. Notice how the commands work similar to navigating a folder structure. If I need to go back a level I can simply use “cd ..”. Alternatively, if I wanted to export this to a text file I could repeat the last command but with a Format-List at the end “dir | fl > C:\IISOutput.txt”. This can be useful when comparing a known working server to a problematic one. Of course there’s also any number of ways this data can be scripted/manipulated/etc. to fit your needs.

Note: The Default Web Site has bindings of 80 & 443 for HTTP & HTTPS while Exchange Back End has 81 & 444. When you make a connection to Exchange using HTTPS you’re connecting to the Default Web Site & it’s proxying it back to the Exchange Back End web site. Do not change the bindings on the Exchange Back End website.

Now if I go back to the root I can see a list of all the Application Pools in IIS.

Alternatively you could just use the Exchange Management Shell for some of these commands but you might find the IIS Module gives you a bit more flexibility.

Now to look at these settings in the GUI may seem easier but it does require a bit more mouse clicks to get all the same data:

 

 

How it can break

Bindings & Firewalls

So we know how things are laid out but now let’s look at what I most commonly see broken from customers. I’ve always said one of the best ways to learn something is to break it in a lab 🙂

Excluding Certificates (which I’ll discuss later), the most common IIS-related issue I see is related to IIS Bindings. I’ll commonly encounter customers trying to install a 3^rd party application or a Microsoft application that is not explicitly supported on an Exchange 2013 Server (SharePoint, RDS, Lync, etc.) & in the process their bindings will get messed up. Allow me to demonstrate.

Say I’m logged into OWA on my multi-role 2013 server.

Now within IIS I right-click Exchange Back End>Edit Bindings & change the HTTPS binding from 444 to 445

If I now refresh my browser I’ll be greeted with a blank page.

This is because, by design, the Default Web Site has the traditional web server bindings for port 80 & 443, while the Exchange Back End website uses ports 81 & 444 for HTTP/HTTPS connectivity. When the Client Access Server role is communicating with the Mailbox Server role for IIS –related functions, it proxies these connections via HTTPS using port 444. So the expected flow for UserA logging into OWA on ServerA (single server environment for this 1^st example) would be:

UserA using browser client
|
ServerA Default Web Site (over port 443)
|
ServerA Exchange Back End website (over port 444)
|
RPC Communications to the local MSExchange RPC Client Access Service
|
MAPI to the MSExchange Information Store Service

Now how would the traffic flow look if we were connecting to https://ServerA/owa with our browser but our mailbox (UserB) was on a database that was mounted on ServerB? Let’s have a look:

UserB using browser client
|
ServerA Default Web Site (over port 443)
|
ServerB Exchange Back End website (over port 444)
|
RPC Communications to the local MSExchange RPC Client Access Service (on ServerB)
|
MAPI to the MSExchange Information Store Service (on ServerB)

As you can see, in this scenario while the client connects to OWA using 443, CAS proxies that connection to the relevant Mailbox Server over 444 (over the network). If you really want to see this in action then you can use a tool like NETSTAT to view connections between your servers:

In the below example I see a local connection to 443 & an associated Process ID (PID). I can use Task Manager to see that PID correlates to an instance of Internet Explorer (iexplore.exe), which I have open & connected to OWA (https://127.0.0.1/owa).

The below command was run from the same server but for port 444; this output is quite a bit busier. There’s the connection to the local server for the OWA session that I’m logged into (the mailbox I’m logged in with is on a database that’s mounted locally). However, you’ll also find there’s a connection to 10.180.62.191, which is one of my other Exchange servers in the environment. This is for another instance of OWA I have open for a mailbox that’s currently mounted on that server. In that case the PID corresponds to an instance of w3wp.exe (World Wide Web Publishing Service). The other PIDs correspond to background processes like Microsoft.Exchange.ServiceHost.exe (MSEXchange Service Host Service), MSExchangeHMWorker.exe (MSExchange Health Manager Service), & MSExchangeMailboxAssistants.exe (MSExchange Mailbox Assistants Service). These are all background processes that are constantly running behind the curtains to keep Exchange up & running (synthetic transactions, maintenance tasks, etc.).

So it’s fairly common to see customers accidentally change the bindings or delete them. Unfortunately, their attempts to repair the web sites typically result in them using the incorrect port numbers (like putting 443 on the Exchange Back End site). Alternatively, customers (or their network security admins) may block port 444 traffic between servers & suddenly find their servers in a state of sad uselessness.

Recreating Exchange Virtual Directories

Since the time of Exchange 2003, Microsoft has given you the ability to recreate Exchange Virtual Directories for troubleshooting purposes. In Exchange 2013 you have the option to reset/recreate the Virtual Directories either from within EAC or Exchange Management Shell.

Recreating the various Virtual Directories has been a useful troubleshooting step in the past but I’ll be honest when I say that it’s usually done as a last ditch step whenever every other avenue of troubleshooting hasn’t helped. In fact, if recreating the vDir doesn’t resolve the issue I’m usually looking at a /RecoverServer install as the next step. But it has been useful when OWA/ECP/ActiveSync/EWS/OAB/PowerShell/AutoDiscover don’t work as expected & you’d like to reset the relevant Virtual Directory to defaults.

Note: Recreating the Virtual Directories will reset any settings or customizations you have done to it so I recommend running a “Get-OWAVirtualDirectory | FL” or similar command beforehand to grab the existing settings. In fact, if you use EAC to reset the VDirs then you’ll be prompted to save the configuration to a network path.

There are two ways to perform this action, EAC (GUI) or EMS (Shell). Let’s look at the EAC method first:

You can go to EAC>Servers>Virtual Directories, select the Virtual Directory you wish to reset & then click the Reset button.

Here we see the prompt you’ll receive to backup the current Virtual Directory settings before resetting it.

After clicking “Reset” the Virtual Directory will be removed & then recreated. Afterwards you’ll need to restart IIS.

Now how would we do this with shell? It’s fairly simple:

Now this works when we have an issue with the Default Web Site but I’ve actually run across a case where I had to recreate the OWA Virtual Directory on the Exchange Back End site as well. To do this I would run the below commands:

PowerShell

Now what if you’re having issues with the PowerShell vDir? You likely can’t connect to that server to manage it via EMS or EAC so you’re going to have to load the local PowerShell snap-in using the below commands:

Since we’re on the topic of PowerShell, on occasion I’ve found myself having to verify all the proper Modules are added for the PowerShell vDIR.

The best advice I can give you is to compare the loaded modules here to a known working server (or lab machine). On several occasions I’ve found the kerbauth module to be missing & I’ve had to re-add it. I saw it on several occasions in Exchange 2010 but not yet in 2013; but regardless, the proper modules will be needed in order for things to work properly on any version of Exchange.

Note: Also make sure that any & all file directory paths have the proper permissions set on them. Again, it’s helpful to have a known working server to use as a comparison. Also, be sure that all proper Anti-Virus Exclusions have been configured (extremely common scenario). (Reference)

 

Certificates & Naming

By far, Certificates are the most common CAS/IIS-related support issue I see; which is odd considering the core concepts are not that difficult. Much like understanding core TCP/IP functionality, I feel core PKI & SSL knowledge should be something every IT professional should learn early on in their careers.

You don’t have to be an expert but you should understand the 3 golden rules of trust: Do I trust the issuer of this certificate? Is the certificate expired? Is the name I’m using to connect to this service listed on the certificate?

Knowing these things will help us to understand which names we need to put onto our Exchange certificate when requesting it. You can technically get away with only having 1 name on your certificate in a simplistic environment with limited requirements (which also seem to be the environments where less experienced customers are unsure of their options). For instance:

Example-A

Name on certificate:
Mail.Contoso.com

Split DNS Enabled=Yes (Mail.Contoso.com resolves to CAS both internally & externally )

Outlook Anywhere Internal & External Namespace=Mail.Contoso.com

OWA/EWS/OAB/ActiveSync Internal and/or External URL’s=Mail.Contoso.com

AutodiscoverServiceInternalUri=Mail.Contoso.com/AutoDiscover/AutoDiscover.xml

In this example, everything would work except for non-domain joined Outlook clients & ActiveSync automatic profile creation. This is because you won’t have AutoDiscover.Contoso.com on your certificate so the process will not be seamless. You’ll either be greeted with certificate warnings or the connection just won’t work. Now technically you can get non-domain joined Outlook clients to work if you create an SRV record for AutoDiscover but there’s no workaround for ActiveSync. Your users will have to manually enter in the server name when creating ActiveSync devices. Also, depending on how your device handles certificates, you may or may not be able to connect.

Example-B (never seen it in the wild but it would technically work fine)

Name on certificate:
AutoDiscover.Contoso.com

Split DNS Enabled=Yes (AutoDiscover.Contoso.com resolves to CAS both internally & externally)

Outlook Anywhere Internal & External Namespace=AutoDiscover.Contoso.com

OWA/EWS/OAB/ActiveSync Internal and/or External URL’s=AutoDiscover.Contoso.com

AutodiscoverServiceInternalUri=AutoDiscover.Contoso.com/AutoDiscover/AutoDiscover.xml

Of course the downside of this configuration is your users would have to use https://autodiscover.contoso.com/owa to access OWA & I haven’t found a customer yet who was willing to do that. However, all services would work, including Outlook/ActiveSync profile autoconfiguration.

 

I brought these examples up not to tell you how to deploy Exchange (by all means, get a multi-name or wildcard cert) but instead to explain that in the end, all that matters is that the names you configure in Exchange are resolvable to CAS & listed on the cert. You could literally make your Outlook Anywhere namespace “randomseriesofcharacters.contoso.com” & as long as it was on your cert & as long as the name resolved to CAS then it would work.

If you remember nothing else about certificates, just remember Do I trust the issuer of this certificate? Is the certificate expired? Is the name I’m using to connect to this service listed on the certificate?

Certificates-Miscellaneous

Certificates are bound to both the Default Web Site as well as the Exchange Back End site in IIS. If you right-click on Default Web Site>Edit Bindings>Select HTTPS & click Edit you can see the current certificate bound to the site. When you run “Enable-ExchangeCertificate –Thumbprint <Thumbprint> -Services IIS”, this is what it configures. The image below shows my certificate generated by my internal Certificate Authority:

I often see the incorrect certificate listed here or I may see certificates missing. Many customers mistakenly think that the Exchange tools are the only way to Import/Export certificates, but the Certificates MMC Snap-In is a very handy troubleshooting tool.

Start>Run>Type MMC & hit Enter>File>Add/Remove Snap-in>Certificates>Add>Select Computer Account>Next>Finish>Ok

Below is the Local Computer account’s Personal Certificates store; where manually installed certificates are likely to be stored. In short, when you run “Import-ExchangeCertificate” the certificate ends up here. So similarly you can use this console to Import/Export certificates as well.

Note: Your Personal store will likely look different than mine as my lab server is also a DC/CA.

Certificate issues have historically revolved around generating the request, but the Certificate Request GUI’s found in Exchange 2010 & 2013 have made those customer calls much easier.

However, a problem I still see with customers is that don’t understand that when you generate the certificate request on the Exchange server, you need to leave that request intact until you receive the new certificate from your issuing Certificate Authority. If you don’t then your certificate will be missing the private key & be effectively useless. I see this frequently when customers are requesting a certificate multiple times or if they try to use a different server to import the cert on than the one they issued the request from. Once a request has been generated, you’ll see the pending request in the EAC Certificates console; along with an option to Complete the request when you’ve received the certificate from your CA (this process generates the Private Key).

Additional Logging

Lastly, I can’t leave out the plethora of logging that’s now present in the install directory (typically C:\Program Files\Microsoft\Exchange Server\V15\Logging) of every Exchange Server. In fact, the logging is so vigorous that you’ll often find it taking up quite a bit of your disk space. Luckily there are methods to truncate unneeded logs. These logs have come in handy when I’ve had to troubleshoot odd issues in the past related to CAS proxy behavior. I’d suggest taking time to look through these logs using notepad or even better, Log Parser Studio. It’s a tool frequently used by Microsoft Support & great for when you have to parse through many log files trying to find a needle in a haystack.

Conclusion

As this post has already grown quite long, I suppose we’ll end it there. I’m sure I could find something to continue rambling on about but hopefully I’ve done this topic enough justice. If you make it to Dev Connections then I’d be happy to chat with you sometime during the conference about any other oddities that surround the world of troubleshooting Exchange.

Additional References

New-OWAVirtualDirectory
http://technet.microsoft.com/en-us/library/bb123752(v=exchg.150).aspx

Remove-OWAVirtualDirectory
http://technet.microsoft.com/en-us/library/aa998237(v=exchg.150).aspx

Default Settings for Exchange Virtual Directories
http://technet.microsoft.com/en-us/library/gg247612(v=exchg.150).aspx

Re-create OWA virtual Directories in Exchange 2013
http://exchangeonline.in/re-create-owa-virtual-directory-exchange-2013/

Virtual Directories: Exchange 2013
http://blogs.msdn.com/b/mvpawardprogram/archive/2013/03/18/virtual-directories-exchange-2013.aspx

How to Re-Create PowerShell Virtual Directory in a Single Exchange Server Environment
http://careexchange.in/how-to-re-create-powershell-virtual-directory-in-a-single-exchange-server-environment/

Managing Exchange 2013 IIS Virtual Directories & Web Applications
http://blogs.technet.com/b/get-exchangehelp/archive/2013/02/07/managing-exchange-2013-iis-virtual-directories-amp-web-applications.aspx

Troubleshooting Exchange 2010 Management Tools startup issues
http://blogs.technet.com/b/exchange/archive/2010/02/04/3409289.aspx

Useful CAS Reporting Script (Michael B Smith)
http://theessentialexchange.com/blogs/michael/archive/2014/09/09/reporting-on-client-access-server-configurations.aspx