Lync/SfB: Quickly access the Certificate Store

In a previous post we wrote about the Checks to do in the Lync/Skype for Business Server Certificate Store, however, sometimes we might also want to manually check it using the Certificate Store MMC.

Since Windows Server 2012 and Windows 8 we can quick access the Certificate Store MMC, for Local Computer and Current User, using Command Prompt/PowerShell or the Windows Search:

Local Computer

certlm

Note: Using the Windows Search we need to add the .msc – certlm.msc

Current User

certmgr

Note: Using the Windows Search we need to add the .msc – certmgr.msc 

Please also check the original post:

PKI Tip: Certificate Store Shortcuts
https://blogs.technet.microsoft.com/xdot509/2013/06/10/pki-tip-certificate-store-shortcuts/

Lync/SfB Server: Event 41026, LS Data MCU after May 2017 .NET Framework update

Update 2017/06/28 – In Workaround #1 we also need to request new Front End certificates with Client and Server authentication in the EKU.

Recently we notice that Lync Server 2010/2013 and Skype for Business Server 2015 Front Ends were generating the Events 41025 and immediately after the Event 41026:

Log Name: Lync Server
Source: LS Data MCU
Date: 5/23/2017 5:31:45 PM
Event ID: 41025
Task Category: (1018)
Level: Information
Keywords: Classic
User: N/A
Computer: sfbfe.uclobby.com
Description:
Connection to the Web Conferencing Edge Server has succeeded

Edge Server Machine FQDN: sfbedge.uclobby.com, Port:8057

Log Name: Lync Server
Source: LS Data MCU
Date: 5/23/2017 5:31:45 PM
Event ID: 41026
Task Category: (1018)
Level: Error
Keywords: Classic
User: N/A
Computer: sfbfe.uclobby.com
Description:
No connectivity with any of Web Conferencing Edge Servers. External Skype for Business clients cannot use Web Conferencing modality.

Cause: Service may be unavailable or Network connectivity may have been compromised.
Resolution:
Verify all Web Conferencing Edge Services in the topology are running, and network connectivity is available.

External Users also reported that couldn’t use WhiteBoard, Polls, Q&A or present PowerPoint with the following errors messages:

We can’t connect to the server for sharing right now.

Network issues are keeping you from sharing notes and presenting whiteboards, polls and uploaded PowerPoint files.

While this is still being investigated a KB article was release with the current workarounds:

LS Data MCU events 41025 and 41026 are constantly generated after you install the May 2017 .NET Framework
https://support.microsoft.com/kb/4023993

The issue is OS independent and affects Lync Server 2010, Lync Server 2013 and Skype for Business Server 2015 and here is a list of the .Net Framework KBs:

  • Windows Server 2008 R2

Description of the Security and Quality Rollup for the .NET Framework 3.5.1 for Windows 7 and Windows Server 2008 R2: May 9, 2017 (KB4014504)
Note: Lync Server 2010 only

Description of the Security Only update for the .NET Framework 3.5.1 for Windows 7 Service Pack 1 and Windows Server 2008 R2 Service Pack 1: May 9, 2017 (KB4014579)
Note: Lync Server 2010 only

Description of the Security and Quality Rollup for the .NET Framework 4.5.2 for Windows 7 Service Pack 1, Windows Server 2008 R2 Service Pack 1, and Windows Server 2008 Service Pack 2: May 9, 2017 (KB4014514)

Description of the Security Only update for the .NET Framework 4.5.2 for Windows 7 SP1, Windows Server 2008 R2 SP1, and Windows Server 2008 SP2: May 9, 2017 (KB4014599)

  • Windows Server 2012

Description of the Security and Quality Rollup for the .NET Framework 4.5.2 for Windows Server 2012: May 9, 2017 (KB4014513)

Description of the Security Only update for the .NET Framework 4.5.2 for Windows Server 2012: May 9, 2017 (KB4014597)

  • Windows Server 2012 R2

Description of the Security and Quality Rollup for the .NET Framework 4.5.2 for Windows 8.1 and Windows Server 2012 R2: May 9, 2017 (KB4014512)

Description of the Security Only update for the .NET Framework 4.5.2 for Windows 8.1 and Windows Server 2012 R2: May 9, 2017 (KB4014595)

  • Windows Server 2016

Windows 10 Version 1607 and Windows Server 2016: May 9, 2017—KB4019472 (OS Build 14393.1198)

This .NET Framework update adds an additional check to the certificate on Enhanced Key Usage (EKU), since all Lync/SfB Server by default use Web Server template, they will only have the Server Authentication in the EKU.

As mentioned in the KB4023993 we can use two workarounds:

Workaround #1

Request new Edge Internal and Front End Pool Certificate with Client and Server Authentication

This workaround requires that we request a new certificate on the Edge Server Internal Interface and in all Front End Servers.

Open the Certification Authority snap-in, right click on Certificate Templates, and then select Manage:

Now in the Certificate Templates Console window, locate the Web Server template, right-click it, and then select Duplicate Template:

In the New Template window select General and add a name:

Note: Please take note of Template Name – WebServerClientandServer. We need to use it to request the new certificate.

In the Extensions Tab , select Application Policies and Edit it:

Add the Client Authentication:

Both Authentication should be present:

Back in Certification Authority snap-in, right click on Certificate Templates > New > Certificate Template to Issue:

Select the new template:

Now that we have the template with Client and Server Authentication, we need to request a new Edge Server Internal Certificate with the recently created template.

Request-CsCertificate -New -Type Internal -Template WebServerClientandServer -FriendlyName “Edge Internal with Client and Server Auth” -Output C:\UCLobby\EdgeIntCliSrv.req

Note: We can also use the -PrivateKeyExportable $true switch to allow the private key to be exported.

In the Active Directory Certificate Services select Request a certificate:

Example: http://ca.gears.lab/certsrv/

Advanced certificate request:

Submit a certificate request by using a base-64-encoded CMC or PKCS #10 file, or submit a renewal request by using a base-64-encoded PKCS #7 file.

We need to select the new certificate template and submit:

We download the new certificate and copy it to the Edge Server and import it:

On the Edge Server import and assign the new certificate:

Import-CsCertificate -Path C:\UCLobby\EdgeIntCliSrv.cer
https://technet.microsoft.com/en-us/library/gg398688.aspx

Note: If we specify the -PrivateKeyExportable $true in the Request-CsCertificate we also need to add it to the Import-csCertificate.

Set-CsCertificate -Type Internal -Thumbprint 335d17df1520a5e30beee96406ffa53e20805342
https://technet.microsoft.com/en-us/library/gg398518.aspx

Please also request new certificates for the Front End Servers with Client and Server Authentication.

After restarting the Lync/SfB Edge and Front End Services the issues should be fixed and external users should be able to use WhiteBoard, Polls, Q&A or present PowerPoint without issues.

Workaround #2

Add a registry key to temporary disable the EKU check

On the all Lync/SfB Front Ends disable the check for the Web Conferencing Service.

Please note that these registry keys are for the default install locations. We can use the following script to assist adding the registry key in the correct location:

Lync/SfB Server: Disable EKU check for Web Conferencing Service
https://gallery.technet.microsoft.com/LyncSfB-Server-Disable-EKU-dab6cb88

Lync Server 2010

reg add HKLM\SOFTWARE\Microsoft\.NETFramework\v2.0.50727\System.Net.ServicePointManager.RequireCertificateEKUs /v “C:\Program Files\Microsoft Lync Server 2010\Web Conferencing\DataMCUSvc.exe” /t REG_DWORD /d 0 /f

Note: Lync Server 2010 still uses the .NET 3.5 this is why we use v2.0.50727.

Lync Server 2013

reg add HKLM\SOFTWARE\Microsoft\.NETFramework\v4.0.30319\System.Net.ServicePointManager.RequireCertificateEKUs /v “C:\Program Files\Microsoft Lync Server 2013\Web Conferencing\DataMCUSvc.exe” /t REG_DWORD /d 0 /f

Skype for Business Server 2015

reg add HKLM\SOFTWARE\Microsoft\.NETFramework\v4.0.30319\System.Net.ServicePointManager.RequireCertificateEKUs /v “C:\Program Files\Skype for Business Server 2015\Web Conferencing\DataMCUSvc.exe” /t REG_DWORD /d 0 /f

After adding the registry key simply restart the Web Conferencing Service

PowerShell

Stop-CsWindowsService -InputObject RTCDATAMCU
Start-CsWindowsService -InputObject RTCDATAMCU

services.msc

Now the external users will be able to use WhiteBoard, Polls, Q&A or present PowerPoint without issues.

Lync/SfB Server: OAuthTokenIssuer, Assigned certificate not found or untrusted.

In a recent support case the OAuth certificate was missing in one of the Front Ends:

We also notice the Missing message in the Deployment Wizard Step 3, for the OAuth certificate:

And in PowerShell we had the following error when we tried to check the certificates:

Get-CsCertificate
https://technet.microsoft.com/en-us/library/gg398227.aspx

Get-CsCertificate : OAuthTokenIssuer: Assigned certificate not found or untrusted. Check that the certificate exists
in the certificate store, that it is not expired and that the certificate chain is valid.

Since the OAuth certificate is a Global setting and it’s replicated, we don’t need to request a new one.

To restore the OAuth certificate, we simply need to restart the Lync/SfB Server Replica Replicator Agent:

During start-up the Replica Replicator Agent will add the OAuth certificate again to the Computer Certificate Store:

We can also check the Deployment Wizard Step 3, to confirm that the correct certificate will be displayed:

For reference, here is the PowerShell output:

Get-CsCertificate -Type OAuthTokenIssuer

PSScript: Lync/SfB Server Certification Store Validation

In a previous post, we published the checks/validations that we should do in the Certification Store in the Lync/SfB servers.

Checks to do in the Lync/SfB Certificate Store

We decided to write a PowerShell with all these checks to make it simple to use. The script will be kept in sync with the post, meaning that when a new check is added, it will also be included in the script.

The PowerShell script is available in the TechNet Gallery:

Lync/Skype4B Certification Store Validation
https://gallery.technet.microsoft.com/LyncSkype4B-Certification-c80a7143

Both script usage and change log are included in the TechNet Gallery description.

Checks to do in the Lync/SfB Server Certificate Store

The checks described in this article are the result of what we normally check during troubleshooting. Some of these already have specific error events, but the objective here is to try to avoid that these events occur.

We plan to keep the post updated and add more checks that we identify as useful. Also, we kindly ask you to add a comment with a test you think that could make a good addition to the list.

Check #1 – Misplaced certificates in Trusted Root CA

Some of us already experienced issues related to having misplaced certificates in Trusted Root CA. In Windows Server 2012, it started to check misplaced certificates and it affected Lync Server 2013:

Lync Server 2013 Front-End service cannot start in Windows Server 2012
https://support.microsoft.com/en-us/kb/2795828

In order to check this, we can use the PowerShell cmdlet mentioned in the above article or this one instead:

Get-Childitem cert:\LocalMachine\root -Recurse | Where-Object {$_.Issuer -ne $_.Subject} | Select Issuer, Subject, Thumbprint | fl

To solve this we need to move the certificate to the proper Store. In this case, we should move it to the Intermediate Certification Authority.

Check #2 – Duplicates in Trusted Root CA

Although this should affect Lync/Skype4B, it is better to check and delete the duplicates:

Get-Childitem cert:\LocalMachine\root | Group-Object -Property Thumbprint | Where-Object {$_.Count -gt 1} | Select-Object -ExpandProperty Group | Select FriendlyName, Issuer, Subject, Thumbprint | fl

Check #3 – More than 100 certificates in Trusted Root CA

This is really important, as it may cause sign-in issues for users. Most of the time, we have less than 50 certificates.

Get-Childitem cert:\LocalMachine\root | Measure

To solve this we have to keep just the certificates that we need. In a Front End, this is actually an easy task, but in a Edge Server we need to be more careful, since the federation with other Lync/Sfb Server environments might get broken if we delete the wrong certificate.

Check #4 – Root CA certificates in Personal Store

Just to have things nice and tidy, we should move these certificates to the Trusted Root CA. But before that, it’s recommended to check whether they are already there, otherwise we might end up with duplicates.

Get-Childitem cert:\LocalMachine\my -Recurse | Where-Object {$_.Issuer -eq $_.Subject} | Select FriendlyName, Issuer, Subject, Thumbprint | fl

Check #5 – Duplicated Friendly Name

Usually, we add different Friendly Names so it gets easier to assign the certificate. In this case, however, it actually gets to be a requirement:

Note: Each certificate Friendly Name must be unique in the computer store.

Certificate requirements for internal servers in Lync Server 2013
https://technet.microsoft.com/en-us/library/gg398094(v=ocs.15).aspx

Again, a simple PowerShell cmdlet:

Get-Childitem cert:\LocalMachine\my | Group-Object -Property FriendlyName | Where-Object {$_.Count -gt 1} | Select-Object -ExpandProperty Group | Select FriendlyName, Issuer, Subject, Thumbprint | fl

Check #6 – Misplaced Root CA certificates in Intermediate CA store (Suggested in the comments)

Get-ChildItem Cert:\LocalMachine\CA | Where-Object {$_.Issuer -eq $_.Subject} | Select Issuer, Subject, Thumbprint | fl

Reassign the private key after deleting a certificate from the snap-in

In case we delete, by mistake, a certificate using the Certificate snap-in, we can still restore it. In order to do this, we need to import the certificate again and reassign the stored private key to it, without having to create a new request.

The following article was published for IIS, but it also applies to Lync/SfB Server and to Exchange Server:

How to assign a private key to a new certificate after you use the Certificates snap-in to delete the original certificate in Internet Information Services
https://support.microsoft.com/en-us/kb/889651

The first step is to import the certificate again to the Computer Personal Store:

privatekey01

After importing the certificate, if it doesn’t have the private key assigned, we cannot use the certificate in any Lync/SfB Server service.

Now we need to take note of the certificate Serial Number. We can get it either in the certificate Details tab:

privatekey02

Note: This is a lab CA and that is why the serial number is just 02, because normally you will have something like ‎04 3b df f8 25 84 cd. This serial number is given by the CA and it is the Serial Number that will be included in the CRL if the certificate is revoked.

Or use the Certutil store switch – https://technet.microsoft.com/en-gb/library/cc732443.aspx:

certutil -store my

privatekey03a

In a Command Prompt/PowerShell, we run the certutil with the repairstore switch:

certutil -repairstore my “Serial Number

 privatekey03

Alternatively, we can use the certificate index in the store. The index number is returned by the certutil -store my command:

privatekey03c

certutil -repairstore my 1

privatekey03b

If the private key is successfully assigned, we will see it in the Computer Personal Store after refreshing it:

privatekey04

Please note that sometimes the private key is lost and we will have to create a new request.

This method will also work if we change Certificate Common Name or add/remove FQDN to the SAN. In this case, we need to follow these steps:

  1. Request the changes in the Certificate Authority online website (note that if we re-key the certificate, we need to import the certificate as usual);
  2. Download the certificate;
  3. Import the certificate to the server;
  4. Take note of the certificate Serial Number;
  5. Run certutil -repairstore my “Serial number”

Merge certificate public and private key with OpenSSL

This post isn’t about Lync Server/Skype for Business Server, but we think it will be a good reference for people that work with Lync/Skype.

When we do an offline certificate request, we will get an .REQ file that looks like this:

—–BEGIN NEW CERTIFICATE REQUEST—–
###################################
—–END NEW CERTIFICATE REQUEST—–

Then we use public or private CA to complete the request, and in return we get a .CER/.CRT file:

—–BEGIN CERTIFICATE—–
###################################
—–END CERTIFICATE—–

The private key, however, is usually stored in the device that generates the request. We can have it in cleartext and it will look like this:

—–BEGIN PRIVATE KEY—–
###################################
—–END PRIVATE KEY—–

We had this customer who sent us the .CER and .KEY. Even though we sent the normal request file created by the Lync Deployment Wizard, still the customer decided to create a new certificate and send us the private key in cleartext.
It’s really important never to store or send the private key of a certificate in cleartext.
We could send a new request, but we really needed to deploy the Edge Server with federation enabled. After some research, we found an easy way to do it using OpenSSL:

OpenSSL.org
https://www.openssl.org/

In this case, we used the OpenSSL for Windows pre-compiled version:

OpenSSL.org – Binary Distributions
https://wiki.openssl.org/index.php/Binaries

Note: Download the 32- or 64-bit to match the Windows version.

Inside the compressed file, we have this:

opensslmerge03

Extract all files to a folder (in this case, we did it to C:OpenSSL) and copy the .CER and .KEY files to this same folder.
If we get a .P7B file with the certificate and the chain, we need to export the certificate first. In order to do this, simply open the file, right-click on the certificate and select All Tasks > Export:

opensslmerge07

When asked for Export File Format, we need to choose Base-64 encoded.509 (.CER):

opensslmerge08

Now in the Command Prompt, go to the folder, run the following command and insert a password (this will be used to import the certificate):

openssl pkcs12 -export -in lync_edge.cer -inkey lync_edge.key -out lync_edge_merged.pfx

opensslmerge04

Note: We can ignore the warning message, since we only need to merge the certificate.

Take notice that the new merged certificate was created in the folder:

opensslmerge05

We can import the certificate and finally have a certificate ready to be used by Lync Server/Skype for Business Server:

opensslmerge06