Cannot remove the Director Pool – Users or Contacts are associated to it.

While trying to decommissioning a Lync Server 2013 Director Pool we got the following error message when we publish the new topology:

This wasn’t expected since a Director Pool shouldn’t have users associated with it.

After troubleshooting the issue, we notice that some users had the attribute msRTCSIP-PrimaryHomeServer associated to the Lync Server 2013 Director Pool.
These users were previously moved to Skype for Business Online, during the move the attribute was updated to the Director Pool that was configured as federation route.
Please note that this behaviour can also happen if we have a Front End Server Pool in the federation route.

Because the msRTCSIP-PrimaryHomeServer attribute isn’t used by Skype for Business Online we can clear it.

The first step is to get the Pool Distinguished Name and the quickest way is using View Logs in the Publishing Wizard:

Then, we Expand all Actions and scroll down to Check Orphaned Users:

In this example the Pool Distinguished Name is:

CN=Lc Services,CN=Microsoft,CN=1:8,CN=Pools,CN=RTC Service,CN=Services,CN=Configuration,DC=uclobby,DC=com

And we assign it to a variable ($PoolDN):

$PoolDN=”CN=Lc Services,CN=Microsoft,CN=1:8,CN=Pools,CN=RTC Service,CN=Services,CN=Configuration,DC=uclobby,DC=com”
$PoolDN

Alternatively, we can use PowerShell to get the Pool Distinguished Name:

Import-module ActiveDirectory
$RTCDN = “AD:\CN=*,CN=Pools,CN=RTC Service,CN=Services,CN=Configuration,”+(Get-ADDomain).distinguishedname
$PoolDN=”CN=Lc Services,CN=Microsoft,”+(Get-ItemProperty -Path $RTCDN -Name dNSHostName,distinguishedname | ?{$_.dNSHostName -eq “<POOL FQDN>“}).distinguishedname

Now we can list the all users that have msRTCSIP-PrimaryHomeServer attribute associated to the pool:

Get-CsUser -LDAPFilter “(msRTCSIP-PrimaryHomeServer=$PoolDN)” | Select SamAccountName,DisplayName,SipAddress,HostingProvider | ft -AutoSize

Note: We can only use this workaround if the HostingProvider is sipfed.online.lync.com.

If we have few users we can simply clear the msRTCSIP-PrimaryHomeServer attribute manually:

Get-ADuser <USERACCOUNT> | Set-ADObject -Identity $_.distinguishedname -Clear “msRTCSIP-PrimaryHomeServer”

However, if we want to clear the attribute for all users associated to the Lync Server 2013 Director Pool we should use the following:

Get-ADObject -LDAPFilter “(& (msRTCSIP-PrimaryHomeServer=$PoolDN)(msRTCSIP-DeploymentLocator=sipfed.online.lync.com))” | Set-ADObject -Clear “msRTCSIP-PrimaryHomeServer”

Note: We added the msRTCSIP-DeploymentLocator since we can only clear the msRTCSIP-PrimaryHomeServer if the users were moved to Skype for Business Online.

After clearing the msRTCSIP-PrimaryHomeServer attribute we successfully remove the Lync Server 2013 Director Pool:

Lync/SfB: Quickly access the Certificate Store

In a previous post we wrote about the Checks to do in the Lync/Skype for Business Server Certificate Store, however, sometimes we might also want to manually check it using the Certificate Store MMC.

Since Windows Server 2012 and Windows 8 we can quick access the Certificate Store MMC, for Local Computer and Current User, using Command Prompt/PowerShell or the Windows Search:

Local Computer

certlm

Note: Using the Windows Search we need to add the .msc – certlm.msc

Current User

certmgr

Note: Using the Windows Search we need to add the .msc – certmgr.msc 

Please also check the original post:

PKI Tip: Certificate Store Shortcuts
https://blogs.technet.microsoft.com/xdot509/2013/06/10/pki-tip-certificate-store-shortcuts/

Lync/SfB Server: Stop Front End service in Starting state

Some of the cases we work have the Front End service in a Starting state:

In PowerShell the status is StartPending:

We cannot stop it on the Services Management Console:

We can go to Task Manager and try to manual stop the service:

If that doesn’t work we need to Go to Details and End task that is associated with the service:

Another simple way to stop all Starting services is by using the following PowerShell cmdlet. We will get a prompt for each service:

Get-Service | ?{$_.Status -eq “StartPending”} | Stop-process

Get-Service
https://msdn.microsoft.com/en-us/powershell/reference/5.1/microsoft.powershell.management/get-service

Stop-Process
https://msdn.microsoft.com/en-us/powershell/reference/5.1/microsoft.powershell.management/stop-process

Lync/SfB Server: Event 41026, LS Data MCU after May 2017 .NET Framework update

Update 2017/06/28 – In Workaround #1 we also need to request new Front End certificates with Client and Server authentication in the EKU.

Recently we notice that Lync Server 2010/2013 and Skype for Business Server 2015 Front Ends were generating the Events 41025 and immediately after the Event 41026:

Log Name: Lync Server
Source: LS Data MCU
Date: 5/23/2017 5:31:45 PM
Event ID: 41025
Task Category: (1018)
Level: Information
Keywords: Classic
User: N/A
Computer: sfbfe.uclobby.com
Description:
Connection to the Web Conferencing Edge Server has succeeded

Edge Server Machine FQDN: sfbedge.uclobby.com, Port:8057

Log Name: Lync Server
Source: LS Data MCU
Date: 5/23/2017 5:31:45 PM
Event ID: 41026
Task Category: (1018)
Level: Error
Keywords: Classic
User: N/A
Computer: sfbfe.uclobby.com
Description:
No connectivity with any of Web Conferencing Edge Servers. External Skype for Business clients cannot use Web Conferencing modality.

Cause: Service may be unavailable or Network connectivity may have been compromised.
Resolution:
Verify all Web Conferencing Edge Services in the topology are running, and network connectivity is available.

External Users also reported that couldn’t use WhiteBoard, Polls, Q&A or present PowerPoint with the following errors messages:

We can’t connect to the server for sharing right now.

Network issues are keeping you from sharing notes and presenting whiteboards, polls and uploaded PowerPoint files.

While this is still being investigated a KB article was release with the current workarounds:

LS Data MCU events 41025 and 41026 are constantly generated after you install the May 2017 .NET Framework
https://support.microsoft.com/kb/4023993

The issue is OS independent and affects Lync Server 2010, Lync Server 2013 and Skype for Business Server 2015 and here is a list of the .Net Framework KBs:

  • Windows Server 2008 R2

Description of the Security and Quality Rollup for the .NET Framework 3.5.1 for Windows 7 and Windows Server 2008 R2: May 9, 2017 (KB4014504)
Note: Lync Server 2010 only

Description of the Security Only update for the .NET Framework 3.5.1 for Windows 7 Service Pack 1 and Windows Server 2008 R2 Service Pack 1: May 9, 2017 (KB4014579)
Note: Lync Server 2010 only

Description of the Security and Quality Rollup for the .NET Framework 4.5.2 for Windows 7 Service Pack 1, Windows Server 2008 R2 Service Pack 1, and Windows Server 2008 Service Pack 2: May 9, 2017 (KB4014514)

Description of the Security Only update for the .NET Framework 4.5.2 for Windows 7 SP1, Windows Server 2008 R2 SP1, and Windows Server 2008 SP2: May 9, 2017 (KB4014599)

  • Windows Server 2012

Description of the Security and Quality Rollup for the .NET Framework 4.5.2 for Windows Server 2012: May 9, 2017 (KB4014513)

Description of the Security Only update for the .NET Framework 4.5.2 for Windows Server 2012: May 9, 2017 (KB4014597)

  • Windows Server 2012 R2

Description of the Security and Quality Rollup for the .NET Framework 4.5.2 for Windows 8.1 and Windows Server 2012 R2: May 9, 2017 (KB4014512)

Description of the Security Only update for the .NET Framework 4.5.2 for Windows 8.1 and Windows Server 2012 R2: May 9, 2017 (KB4014595)

  • Windows Server 2016

Windows 10 Version 1607 and Windows Server 2016: May 9, 2017—KB4019472 (OS Build 14393.1198)

This .NET Framework update adds an additional check to the certificate on Enhanced Key Usage (EKU), since all Lync/SfB Server by default use Web Server template, they will only have the Server Authentication in the EKU.

As mentioned in the KB4023993 we can use two workarounds:

Workaround #1

Request new Edge Internal and Front End Pool Certificate with Client and Server Authentication

This workaround requires that we request a new certificate on the Edge Server Internal Interface and in all Front End Servers.

Open the Certification Authority snap-in, right click on Certificate Templates, and then select Manage:

Now in the Certificate Templates Console window, locate the Web Server template, right-click it, and then select Duplicate Template:

In the New Template window select General and add a name:

Note: Please take note of Template Name – WebServerClientandServer. We need to use it to request the new certificate.

In the Extensions Tab , select Application Policies and Edit it:

Add the Client Authentication:

Both Authentication should be present:

Back in Certification Authority snap-in, right click on Certificate Templates > New > Certificate Template to Issue:

Select the new template:

Now that we have the template with Client and Server Authentication, we need to request a new Edge Server Internal Certificate with the recently created template.

Request-CsCertificate -New -Type Internal -Template WebServerClientandServer -FriendlyName “Edge Internal with Client and Server Auth” -Output C:\UCLobby\EdgeIntCliSrv.req

Note: We can also use the -PrivateKeyExportable $true switch to allow the private key to be exported.

In the Active Directory Certificate Services select Request a certificate:

Example: http://ca.gears.lab/certsrv/

Advanced certificate request:

Submit a certificate request by using a base-64-encoded CMC or PKCS #10 file, or submit a renewal request by using a base-64-encoded PKCS #7 file.

We need to select the new certificate template and submit:

We download the new certificate and copy it to the Edge Server and import it:

On the Edge Server import and assign the new certificate:

Import-CsCertificate -Path C:\UCLobby\EdgeIntCliSrv.cer
https://technet.microsoft.com/en-us/library/gg398688.aspx

Note: If we specify the -PrivateKeyExportable $true in the Request-CsCertificate we also need to add it to the Import-csCertificate.

Set-CsCertificate -Type Internal -Thumbprint 335d17df1520a5e30beee96406ffa53e20805342
https://technet.microsoft.com/en-us/library/gg398518.aspx

Please also request new certificates for the Front End Servers with Client and Server Authentication.

After restarting the Lync/SfB Edge and Front End Services the issues should be fixed and external users should be able to use WhiteBoard, Polls, Q&A or present PowerPoint without issues.

Workaround #2

Add a registry key to temporary disable the EKU check

On the all Lync/SfB Front Ends disable the check for the Web Conferencing Service.

Please note that these registry keys are for the default install locations. We can use the following script to assist adding the registry key in the correct location:

Lync/SfB Server: Disable EKU check for Web Conferencing Service
https://gallery.technet.microsoft.com/LyncSfB-Server-Disable-EKU-dab6cb88

Lync Server 2010

reg add HKLM\SOFTWARE\Microsoft\.NETFramework\v2.0.50727\System.Net.ServicePointManager.RequireCertificateEKUs /v “C:\Program Files\Microsoft Lync Server 2010\Web Conferencing\DataMCUSvc.exe” /t REG_DWORD /d 0 /f

Note: Lync Server 2010 still uses the .NET 3.5 this is why we use v2.0.50727.

Lync Server 2013

reg add HKLM\SOFTWARE\Microsoft\.NETFramework\v4.0.30319\System.Net.ServicePointManager.RequireCertificateEKUs /v “C:\Program Files\Microsoft Lync Server 2013\Web Conferencing\DataMCUSvc.exe” /t REG_DWORD /d 0 /f

Skype for Business Server 2015

reg add HKLM\SOFTWARE\Microsoft\.NETFramework\v4.0.30319\System.Net.ServicePointManager.RequireCertificateEKUs /v “C:\Program Files\Skype for Business Server 2015\Web Conferencing\DataMCUSvc.exe” /t REG_DWORD /d 0 /f

After adding the registry key simply restart the Web Conferencing Service

PowerShell

Stop-CsWindowsService -InputObject RTCDATAMCU
Start-CsWindowsService -InputObject RTCDATAMCU

services.msc

Now the external users will be able to use WhiteBoard, Polls, Q&A or present PowerPoint without issues.

Lync/SfB Server: OAuthTokenIssuer, Assigned certificate not found or untrusted.

In a recent support case the OAuth certificate was missing in one of the Front Ends:

We also notice the Missing message in the Deployment Wizard Step 3, for the OAuth certificate:

And in PowerShell we had the following error when we tried to check the certificates:

Get-CsCertificate
https://technet.microsoft.com/en-us/library/gg398227.aspx

Get-CsCertificate : OAuthTokenIssuer: Assigned certificate not found or untrusted. Check that the certificate exists
in the certificate store, that it is not expired and that the certificate chain is valid.

Since the OAuth certificate is a Global setting and it’s replicated, we don’t need to request a new one.

To restore the OAuth certificate, we simply need to restart the Lync/SfB Server Replica Replicator Agent:

During start-up the Replica Replicator Agent will add the OAuth certificate again to the Computer Certificate Store:

We can also check the Deployment Wizard Step 3, to confirm that the correct certificate will be displayed:

For reference, here is the PowerShell output:

Get-CsCertificate -Type OAuthTokenIssuer

SfB Server 2015: Cannot install Cumulative Update – Failed to create network share. (-2147467259 xds-replica)

In a recent support case we couldn’t install the Skype for Business Server 2015 Cumulative Update on a Edge Server:

We checked the Skype for Business Core Components log file (OcsCore.msp-EDGESTD-[2017-03-21][15-30-48]_log.txt) located in %TEMP%:

The log showed that the Setup was aborted because it couldn’t create a file share:

CreateSmb: Error 0x80004005: failed to create share: ‘xds-replica’
MSI (s) (78!14) [15:32:19:981]: Product: Skype for Business Server 2015, Core Components — Error 26301. Failed to create network share. (-2147467259 xds-replica)

MSI (s) (78!14) [15:32:19:981]: Closing MSIHANDLE (193) of type 790531 for thread 3860
Error 26301. Failed to create network share. (-2147467259 xds-replica)

After checking the services running on the server we notice that Server Service (LanmanServer) was disabled:

Then we enable and started the service:

With Server Service running we were able to install the Skype for Business Cumulative Update:

Please note that if we try to repair the Skype for Business Server 2015 Core Components when the Server Service is disable we also get the following errors:

Failed to create network share. (-2147467259 xds-replica)

Failed to drop network share. (-2147022782 xds-replica)

Lync Server: Event 41029 LS Data MCU – No connectivity with the Lync Web App

In a recent support case we were working on an issue where sometimes the users couldn’t use the Lync Web App.

The troubleshooting started in the Event Viewer > Lync Server, we notice that we had a few errors:

Log Name: Lync Server
Source: LS Data MCU
Date: 01/03/2017 15:00:43
Event ID: 41029
Task Category: (1018)
Level: Error
Keywords: Classic
User: N/A
Computer: lync2013fe01.gears.lab
Description:
No connectivity with the Lync Web App. Affected Web browser clients cannot use Web Conferencing modality.

Server Machine FQDN: lync2013fe01.gears.lab, Port:8061
Server Type: External-WebApp-Edge [HTTP side error:Unable to connect to the remote server]
If the problem persists this event will be logged again after 20 minutes
Cause: Service may be unavailable or Network connectivity may have been compromised.

Another error was mentioning an issue HTTP connectivity:

Log Name: Lync Server
Source: LS User Services
Date: 01/03/2017 15:04:57
Event ID: 30988
Task Category: (1006)
Level: Error
Keywords: Classic
User: N/A
Computer: lync2013fe01.gears.lab
Description:
Sending HTTP request failed. Server functionality will be affected if messages are failing consistently.

Sending the message to https://lync2013fe01.gears.lab:444/LiveServer/Replication failed. IP Address is 172.20.13.21. Error code is 2EFD. Content-Type is application/replication+xml. Http Error Code is 0.

Cause: Network connectivity issues or an incorrectly configured certificate on the destination server. Check the eventlog description for more information.
Resolution:
Check the destination server to see that it is listening on the same URI and it has certificate configured for MTLS. Other reasons might be network connectivity issues between the two servers.

Both events showed that the server could not establish a connection himself.

Then we check if the server was listening on that port:

netstat -anp TCP
https://technet.microsoft.com/en-us/library/bb490947.aspx

Get-NetTCPConnection -State Listen -LocalPort 80,8080,443,444,4443,8061 | ft -AutoSize
https://technet.microsoft.com/itpro/powershell/windows/tcpip/get-nettcpconnection

The HTTP/HTTPS bindings were only on 127.0.0.1 and this is the loopback address.

Then we run the same on a working server in the same pool:

Note: For Get-NetTCPConnection :: is any available IPV4/IPV6 address.

So, in a working server the binding was on any available IP address, while the non-working was only on the loopback address.

Initially, we thought the issue was in IIS/certificate bindings, but both were properly configured:

Get-WebBinding | ft -AutoSize
https://technet.microsoft.com/en-us/library/hh867866(v=wps.630).aspx

netsh http show sslcert
https://msdn.microsoft.com/en-us/library/windows/desktop/cc307236(v=vs.85).aspx

After checking other parameters available in netsh we found that the non-working had the loopback address configured in the HTTP IP Listen List:

netsh http show iplisten

While the working server we didn’t had any IP address configured:

This was causing the wrong binding, to fix it we only had to remove the loopback address from the list:

netsh http delete iplisten 127.0.0.1

After this change the server started to listen in the correct IP address/ports:

netstat -anp TCP

We also confirmed in the Event Viewer that the Lync Web App was starting: