Centralized Logging Service not working in Skype for Business

I haven’t written a new post in a while as I have not been doing as many deployments lately. I have been focused more on evangelism and speaking with different organizations that are thinking about deploying Skype for Business.

That said, recently I had the opportunity to get back in to the game and start a deployment for a large organization. We were experiencing some issues that necessitated us to do some logging on the Front-end servers. I turned to my favorite tool for this, the CLSLogger which takes advantage of the Centralized Logging Service (CLS).

I would start up CLSLogger and when I attempted to start the scenario, I would get an error like this:

WARNING: Failed on 1 agents
Agent - mediation1.domain.com, Reason - Error code - 20000, Message - Unknown error - 
Error calling agent mediation1.domain.com; Could not connect to 
net.tcp://mediation1.domain.com:50001/. The connection attempt lasted for 
a time span of 00:00:02.0228175. TCP error code 10061: 
No connection could be made because the target machine actively
 refused it 10.0.0.40:50001. . Please refer CLS logs for details.

When I went to the server and did a netstat, I saw that CLS was not actively listening. I should have seen the system listening on ports 50001-50003.

PS C:\Windows\system32> netstat -an | findstr 5000*
  UDP    0.0.0.0:500            *:*
  UDP    0.0.0.0:4500           *:*
  UDP    [::]:500               *:*
  UDP    [::]:4500              *:*

I went round and round with this and ultimately had to open a case with Microsoft on it. After doing some traces and pulling the ETL logs, the Microsoft Engineer got back with and asked how we had generated the certificate. He was seeing the following message in the ETL log (which us mere mortals don’t have the ability to read and the real reason I’m writing this blog article):

29 TL_ERROR(TF_COMPONENT) [3]14CC.1728::06/08/2016-16:53:00.193.00000010 
(CLSAgent,CommandProcessor.Initialize:commandprocessor.cs(247))Exception - 
[System.ArgumentException: It is likely that certificate 
'CN=mediation.domain.com, OU=IT, O=Domain, C=US' 
may not have a private key that is capable of key exchange or the process 
may not have access rights for the private key. 
Please see inner exception for detail. ---> System.Security.Cryptography.CryptographicException: 
Invalid provider type specified.

We then looked at the certificates installed on the machine by running “certutil -store my”. (I’ve purposfully deleted identifying information and highlighted in red the Provider which is the key piece of information.)

PS C:\Windows\system32> certutil -store my
my "Personal"
================ Certificate 0 ================
Serial Number: 5xxxxxx
Issuer: 
NotBefore: 4/15/2016 11:29 AM
NotAfter: 4/15/2019 11:59 AM
Subject: CN=, OU=IT, O=Domain, C=US
Non-root Certificate
Cert Hash(sha1): 39 2d..... 
  Key Container = le-360ab342
  Unique container name: 5d26bec417ed43b7840b7bf82c2fb363
  Provider = Microsoft Software Key Storage Provider
Encryption test passed
CertUtil: -store command completed successfully.

When we talked to the group that generated the certificate, we found that they used their own template instead of the Wizard in Skype for Business. While you certainly don’t have to use the Wizard, Skype for Business definitely has some requirements on the certs that will work with it. As an example, going to a key length longer that 256 usually doesn’t work out too well. In this case, the Provider was what was wrong.

I then turned to the Digicert Utility, one of our other favorite tools to generate the Certificate Request (CSR). This then utilized the correct Provider which is the Microsoft RSA SChannel Cryptographic Provider. After we issued the new cert and assigned it, we restarted the servers and checked the certs.

PS C:\Windows\system32> certutil -store my
my "Personal"
================ Certificate 0 ================
Serial Number: 5xxxxxx
Issuer: 
 NotBefore: 6/9/2016 12:02 PM
 NotAfter: 6/9/2019 12:32 PM
Subject: CN=, OU=IT, O=Domain, C=US
Non-root Certificate
Cert Hash(sha1): 3c e2....
  Key Container = 70C232....
  Unique container name: 9ed77....
  Provider = Microsoft RSA SChannel Cryptographic Provider
Encryption test passed

We also started seeing CLS listening as expected:

PS C:\Windows\system32> netstat -an | findstr 5000*
  TCP    0.0.0.0:50001          0.0.0.0:0              LISTENING
  TCP    0.0.0.0:50002          0.0.0.0:0              LISTENING
  TCP    0.0.0.0:50003          0.0.0.0:0              LISTENING
  TCP    [::]:50001             [::]:0                 LISTENING
  TCP    [::]:50002             [::]:0                 LISTENING
  TCP    [::]:50003             [::]:0                 LISTENING
  UDP    0.0.0.0:500            *:*
  UDP    0.0.0.0:4500           *:*
  UDP    [::]:500               *:*
  UDP    [::]:4500              *:*

I hope this helps someone else who might end up in this situation.