SfB Server 2015: Event 57005, LS User Store Sync Agent – Could not find stored procedure XdsQueryCriticalDocumentSignatures

While updating our SfB Server 2015 lab, we notice that a recently updated Front End server had multiple errors and warnings in the Event Viewer:

Log Name: Lync Server
Source: LS User Store Sync Agent
Date: 12/09/2017 21:54:33
Event ID: 57005
Task Category: (1061)
Level: Error
Keywords: Classic
User: N/A
Computer: sfbfe03.recore.lab
Description:
Error encountered pushing data to RtcXds Blob Store

#CTX#{ctx:{traceId:10006, activityId:”e40d8197-4293-4146-9d72-03c0c2957f6c”}}#CTX#
Push cycle identifier: [sfbfe03.recore.lab.2fd688f5-0f3a-407f-bab5-3fa5c3757443]
ItemCount: [0]
Error Message: [PushController: XdsQueryCriticalDocumentSignatures failed: System.Data.SqlClient.SqlException (0x80131904): Could not find stored procedure ‘XdsQueryCriticalDocumentSignatures’.
at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
at System.Data.SqlClient.SqlDataReader.TryConsumeMetaData()
at System.Data.SqlClient.SqlDataReader.get_MetaData()
at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)
at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, SqlDataReader ds)
at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, TaskCompletionSource`1 completion, Int32 timeout, Task& task, Boolean asyncWrite)
at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method)
at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior, String method)
at System.Data.SqlClient.SqlCommand.ExecuteReader()
at Microsoft.Rtc.Common.Data.DBCore.Execute(SprocContext sprocContext, SqlConnection sqlConnection, SqlTransaction sqlTransaction)
ClientConnectionId:e597ef79-3a87-4d08-8561-8e8c0db10e37
Error Number:2812,State:62,Class:16]
Cause: Possible issues with back-end database.
Resolution:
Ensure the back-end is functioning correctly.

Log Name: Lync Server
Source: LS User Store Sync Agent
Date: 12/09/2017 21:54:33
Event ID: 57006
Task Category: (1061)
Level: Warning
Keywords: Classic
User: N/A
Computer: sfbfe03.recore.lab
Description:
RtcDb Sync Agent sproc failed

#CTX#{ctx:{traceId:10006, activityId:”e40d8197-4293-4146-9d72-03c0c2957f6c”}}#CTX#
Sproc: [XdsQueryCriticalDocumentSignatures]
Exception: [System.Data.SqlClient.SqlException (0x80131904): Could not find stored procedure ‘XdsQueryCriticalDocumentSignatures’.
at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
at System.Data.SqlClient.SqlDataReader.TryConsumeMetaData()
at System.Data.SqlClient.SqlDataReader.get_MetaData()
at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)
at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, SqlDataReader ds)
at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, TaskCompletionSource`1 completion, Int32 timeout, Task& task, Boolean asyncWrite)
at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method)
at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior, String method)
at System.Data.SqlClient.SqlCommand.ExecuteReader()
at Microsoft.Rtc.Common.Data.DBCore.Execute(SprocContext sprocContext, SqlConnection sqlConnection, SqlTransaction sqlTransaction)
ClientConnectionId:e597ef79-3a87-4d08-8561-8e8c0db10e37
Error Number:2812,State:62,Class:16]

The error message mentions that XdsQueryCriticalDocumentSignatures store procedure is missing from RTCXDS database, after checking the version for this particular database, we notice that a new version was available:

Test-CsDatabase -ConfiguredDatabases -SqlServerFqdn sqlpool.recore.lab | Select SqlServerFqdn, SqlInstanceName, DatabaseName, InstalledVersion, ExpectedVersion | ft -AutoSize

A complete database version list for Skype for Business Server 2015 is available here:

Doug Deitterick’s Blog – How to Verify if Skype for Business Server 2015 Database Updates Completed Successfully
https://blogs.technet.microsoft.com/dodeitte/2015/05/10/how-to-verify-if-skype-for-business-server-2015-database-updates-completed-successfully/

Please note that XdsQueryCriticalDocumentSignatures store procedure was added in the May 2017 Cumulative Update for SfB Server 2015.

After we finish updating the remaining Front End servers and updated the RTCXDS database on the SfB Back End, we didn’t get more Errors/Warnings related to the missing XdsQueryCriticalDocumentSignatures store procedure.

We need to make sure that we follow the steps described for each Lync/SfB Server version:

Updates for Lync Server 2010
http://support.microsoft.com/kb/2493736

Updates for Lync Server 2013
http://support.microsoft.com/kb/2809243

Updates for Skype for Business Server 2015
http://support.microsoft.com/kb/3061064

Lync/SfB Server: Event 41026, LS Data MCU after May 2017 .NET Framework update

Update 2017/06/28 – In Workaround #1 we also need to request new Front End certificates with Client and Server authentication in the EKU.

Recently we notice that Lync Server 2010/2013 and Skype for Business Server 2015 Front Ends were generating the Events 41025 and immediately after the Event 41026:

Log Name: Lync Server
Source: LS Data MCU
Date: 5/23/2017 5:31:45 PM
Event ID: 41025
Task Category: (1018)
Level: Information
Keywords: Classic
User: N/A
Computer: sfbfe.uclobby.com
Description:
Connection to the Web Conferencing Edge Server has succeeded

Edge Server Machine FQDN: sfbedge.uclobby.com, Port:8057

Log Name: Lync Server
Source: LS Data MCU
Date: 5/23/2017 5:31:45 PM
Event ID: 41026
Task Category: (1018)
Level: Error
Keywords: Classic
User: N/A
Computer: sfbfe.uclobby.com
Description:
No connectivity with any of Web Conferencing Edge Servers. External Skype for Business clients cannot use Web Conferencing modality.

Cause: Service may be unavailable or Network connectivity may have been compromised.
Resolution:
Verify all Web Conferencing Edge Services in the topology are running, and network connectivity is available.

External Users also reported that couldn’t use WhiteBoard, Polls, Q&A or present PowerPoint with the following errors messages:

We can’t connect to the server for sharing right now.

Network issues are keeping you from sharing notes and presenting whiteboards, polls and uploaded PowerPoint files.

While this is still being investigated a KB article was release with the current workarounds:

LS Data MCU events 41025 and 41026 are constantly generated after you install the May 2017 .NET Framework
https://support.microsoft.com/kb/4023993

The issue is OS independent and affects Lync Server 2010, Lync Server 2013 and Skype for Business Server 2015 and here is a list of the .Net Framework KBs:

  • Windows Server 2008 R2

Description of the Security and Quality Rollup for the .NET Framework 3.5.1 for Windows 7 and Windows Server 2008 R2: May 9, 2017 (KB4014504)
Note: Lync Server 2010 only

Description of the Security Only update for the .NET Framework 3.5.1 for Windows 7 Service Pack 1 and Windows Server 2008 R2 Service Pack 1: May 9, 2017 (KB4014579)
Note: Lync Server 2010 only

Description of the Security and Quality Rollup for the .NET Framework 4.5.2 for Windows 7 Service Pack 1, Windows Server 2008 R2 Service Pack 1, and Windows Server 2008 Service Pack 2: May 9, 2017 (KB4014514)

Description of the Security Only update for the .NET Framework 4.5.2 for Windows 7 SP1, Windows Server 2008 R2 SP1, and Windows Server 2008 SP2: May 9, 2017 (KB4014599)

  • Windows Server 2012

Description of the Security and Quality Rollup for the .NET Framework 4.5.2 for Windows Server 2012: May 9, 2017 (KB4014513)

Description of the Security Only update for the .NET Framework 4.5.2 for Windows Server 2012: May 9, 2017 (KB4014597)

  • Windows Server 2012 R2

Description of the Security and Quality Rollup for the .NET Framework 4.5.2 for Windows 8.1 and Windows Server 2012 R2: May 9, 2017 (KB4014512)

Description of the Security Only update for the .NET Framework 4.5.2 for Windows 8.1 and Windows Server 2012 R2: May 9, 2017 (KB4014595)

  • Windows Server 2016

Windows 10 Version 1607 and Windows Server 2016: May 9, 2017—KB4019472 (OS Build 14393.1198)

This .NET Framework update adds an additional check to the certificate on Enhanced Key Usage (EKU), since all Lync/SfB Server by default use Web Server template, they will only have the Server Authentication in the EKU.

As mentioned in the KB4023993 we can use two workarounds:

Workaround #1

Request new Edge Internal and Front End Pool Certificate with Client and Server Authentication

This workaround requires that we request a new certificate on the Edge Server Internal Interface and in all Front End Servers.

Open the Certification Authority snap-in, right click on Certificate Templates, and then select Manage:

Now in the Certificate Templates Console window, locate the Web Server template, right-click it, and then select Duplicate Template:

In the New Template window select General and add a name:

Note: Please take note of Template Name – WebServerClientandServer. We need to use it to request the new certificate.

In the Extensions Tab , select Application Policies and Edit it:

Add the Client Authentication:

Both Authentication should be present:

Back in Certification Authority snap-in, right click on Certificate Templates > New > Certificate Template to Issue:

Select the new template:

Now that we have the template with Client and Server Authentication, we need to request a new Edge Server Internal Certificate with the recently created template.

Request-CsCertificate -New -Type Internal -Template WebServerClientandServer -FriendlyName “Edge Internal with Client and Server Auth” -Output C:\UCLobby\EdgeIntCliSrv.req

Note: We can also use the -PrivateKeyExportable $true switch to allow the private key to be exported.

In the Active Directory Certificate Services select Request a certificate:

Example: http://ca.gears.lab/certsrv/

Advanced certificate request:

Submit a certificate request by using a base-64-encoded CMC or PKCS #10 file, or submit a renewal request by using a base-64-encoded PKCS #7 file.

We need to select the new certificate template and submit:

We download the new certificate and copy it to the Edge Server and import it:

On the Edge Server import and assign the new certificate:

Import-CsCertificate -Path C:\UCLobby\EdgeIntCliSrv.cer
https://technet.microsoft.com/en-us/library/gg398688.aspx

Note: If we specify the -PrivateKeyExportable $true in the Request-CsCertificate we also need to add it to the Import-csCertificate.

Set-CsCertificate -Type Internal -Thumbprint 335d17df1520a5e30beee96406ffa53e20805342
https://technet.microsoft.com/en-us/library/gg398518.aspx

Please also request new certificates for the Front End Servers with Client and Server Authentication.

After restarting the Lync/SfB Edge and Front End Services the issues should be fixed and external users should be able to use WhiteBoard, Polls, Q&A or present PowerPoint without issues.

Workaround #2

Add a registry key to temporary disable the EKU check

On the all Lync/SfB Front Ends disable the check for the Web Conferencing Service.

Please note that these registry keys are for the default install locations. We can use the following script to assist adding the registry key in the correct location:

Lync/SfB Server: Disable EKU check for Web Conferencing Service
https://gallery.technet.microsoft.com/LyncSfB-Server-Disable-EKU-dab6cb88

Lync Server 2010

reg add HKLM\SOFTWARE\Microsoft\.NETFramework\v2.0.50727\System.Net.ServicePointManager.RequireCertificateEKUs /v “C:\Program Files\Microsoft Lync Server 2010\Web Conferencing\DataMCUSvc.exe” /t REG_DWORD /d 0 /f

Note: Lync Server 2010 still uses the .NET 3.5 this is why we use v2.0.50727.

Lync Server 2013

reg add HKLM\SOFTWARE\Microsoft\.NETFramework\v4.0.30319\System.Net.ServicePointManager.RequireCertificateEKUs /v “C:\Program Files\Microsoft Lync Server 2013\Web Conferencing\DataMCUSvc.exe” /t REG_DWORD /d 0 /f

Skype for Business Server 2015

reg add HKLM\SOFTWARE\Microsoft\.NETFramework\v4.0.30319\System.Net.ServicePointManager.RequireCertificateEKUs /v “C:\Program Files\Skype for Business Server 2015\Web Conferencing\DataMCUSvc.exe” /t REG_DWORD /d 0 /f

After adding the registry key simply restart the Web Conferencing Service

PowerShell

Stop-CsWindowsService -InputObject RTCDATAMCU
Start-CsWindowsService -InputObject RTCDATAMCU

services.msc

Now the external users will be able to use WhiteBoard, Polls, Q&A or present PowerPoint without issues.

Lync Server: Event 41029 LS Data MCU – No connectivity with the Lync Web App

In a recent support case we were working on an issue where sometimes the users couldn’t use the Lync Web App.

The troubleshooting started in the Event Viewer > Lync Server, we notice that we had a few errors:

Log Name: Lync Server
Source: LS Data MCU
Date: 01/03/2017 15:00:43
Event ID: 41029
Task Category: (1018)
Level: Error
Keywords: Classic
User: N/A
Computer: lync2013fe01.gears.lab
Description:
No connectivity with the Lync Web App. Affected Web browser clients cannot use Web Conferencing modality.

Server Machine FQDN: lync2013fe01.gears.lab, Port:8061
Server Type: External-WebApp-Edge [HTTP side error:Unable to connect to the remote server]
If the problem persists this event will be logged again after 20 minutes
Cause: Service may be unavailable or Network connectivity may have been compromised.

Another error was mentioning an issue HTTP connectivity:

Log Name: Lync Server
Source: LS User Services
Date: 01/03/2017 15:04:57
Event ID: 30988
Task Category: (1006)
Level: Error
Keywords: Classic
User: N/A
Computer: lync2013fe01.gears.lab
Description:
Sending HTTP request failed. Server functionality will be affected if messages are failing consistently.

Sending the message to https://lync2013fe01.gears.lab:444/LiveServer/Replication failed. IP Address is 172.20.13.21. Error code is 2EFD. Content-Type is application/replication+xml. Http Error Code is 0.

Cause: Network connectivity issues or an incorrectly configured certificate on the destination server. Check the eventlog description for more information.
Resolution:
Check the destination server to see that it is listening on the same URI and it has certificate configured for MTLS. Other reasons might be network connectivity issues between the two servers.

Both events showed that the server could not establish a connection himself.

Then we check if the server was listening on that port:

netstat -anp TCP
https://technet.microsoft.com/en-us/library/bb490947.aspx

Get-NetTCPConnection -State Listen -LocalPort 80,8080,443,444,4443,8061 | ft -AutoSize
https://technet.microsoft.com/itpro/powershell/windows/tcpip/get-nettcpconnection

The HTTP/HTTPS bindings were only on 127.0.0.1 and this is the loopback address.

Then we run the same on a working server in the same pool:

Note: For Get-NetTCPConnection :: is any available IPV4/IPV6 address.

So, in a working server the binding was on any available IP address, while the non-working was only on the loopback address.

Initially, we thought the issue was in IIS/certificate bindings, but both were properly configured:

Get-WebBinding | ft -AutoSize
https://technet.microsoft.com/en-us/library/hh867866(v=wps.630).aspx

netsh http show sslcert
https://msdn.microsoft.com/en-us/library/windows/desktop/cc307236(v=vs.85).aspx

After checking other parameters available in netsh we found that the non-working had the loopback address configured in the HTTP IP Listen List:

netsh http show iplisten

While the working server we didn’t had any IP address configured:

This was causing the wrong binding, to fix it we only had to remove the loopback address from the list:

netsh http delete iplisten 127.0.0.1

After this change the server started to listen in the correct IP address/ports:

netstat -anp TCP

We also confirmed in the Event Viewer that the Lync Web App was starting:

 

Lync Server 2013: Event 14497,14517 LS Protocol Stack

This issue dates a while back, but after Lync Server 2013 Cumulative Update 3 (5.0.8308.556 and above) the Edge Server Access Service won’t start with the Event 14497 LS Protocol Stack:

Event14517-14497-02

One or more configuration errors were detected at startup that cannot be mitigated.

Cause: There are serious problems with the server configuration that prevented it from starting up.
Resolution:
Review the previous event log entries to identify failures. Alter the server configuration as required. If problems persist, contact Product Support Services.

Log Name:      Lync Server
Source:        LS Protocol Stack
Date:          07/05/2015 17:41:09
Event ID:      14497
Task Category: (1001)
Level:         Error
Keywords:      Classic
User:          N/A

Computer:      edge01.gears.lab

In a previous error, we found the following in Event Viewer:

Event14517-14497-01

Event 14517, LS Protocol Stack

The server configuration validation mechanism detected some serious problems.

1 errors and 0 warnings were detected.

ERRORS:
The server at FQDN [sipfed.online.lync.com] is configured as both type ‘allowed partner server’ and type ‘IM service provider’.

WARNINGS:
No warnings

Cause: The configuration is invalid and the server might not behave as expected.
Resolution:
Review and correct the errors listed above, then restart the service. You may also wish to review any warnings present.

Log Name:      Lync Server
Source:        LS Protocol Stack
Date:          07/05/2015 10:32:35
Event ID:      14517
Task Category: (1001)
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      edge01.gears.lab

Both the errors helped us to find the issue easily. In this case, we had configured a Lync Online tenant as Allowed Domain in SIP Federated Domains before updating to Lync Server 2013 CU3:

Lync Server Control Panel

Event14517-14497-03

Event14517-14497-04

Lync Server PowerShell

Lync Server 2013: Get-CsAllowedDomain
https://technet.microsoft.com/en-us/library/gg398164(v=ocs.15).aspx

Event14517-14497-03ps

Lync Server 2013: Get-CsHostingProvider
https://technet.microsoft.com/en-us/library/gg413078(v=ocs.15).aspx

Event14517-14497-04ps

To solve this we simply need to remove the Access Edge service (FQDN), which in this case is sipfed.online.lync.com from the Allowed Domain:

Lync Server Control Panel

Event14517-14497-05

Lync Server PowerShell

Lync Server 2013: Set-CsAllowedDomain
https://technet.microsoft.com/en-us/library/gg398931(v=ocs.15).aspx

Event14517-14497-05ps

Before continue, we need to check if the replication was successful:

Lync Server 2013: Get-CsManagementStoreReplicationStatus
https://technet.microsoft.com/en-us/library/gg399052(v=ocs.15).aspx

Event14517-14497-06a

Now we can go to Edge Server and use Start-CsWindowsService to start all Lync Server related services. After that, we can check with Get-CsWindowsService that all services are up and running:

Event14517-14497-06

Notice that after Lync Server 2013 Cumulative Update 3 we cannot add a new Allowed Domain with the same Access Edge service (FQDN) as a Hosting Provider:

Event14517-14497-07

When using Allowed Domains without specifying the Access Edge service (FQDN), make sure that Lync Server will rely on the DNS SRV Record for that specific SIP domain.

The Nslookup.exe Command Line Tool

https://technet.microsoft.com/en-us/library/ee624049(v=ws.10).aspx

nslookup -type=srv _sipfederationtls._tcp.<SIP Domain>
Event14517-14497-08