Cannot remove the Director Pool – Users or Contacts are associated to it.

While trying to decommissioning a Lync Server 2013 Director Pool we got the following error message when we publish the new topology:

This wasn’t expected since a Director Pool shouldn’t have users associated with it.

After troubleshooting the issue, we notice that some users had the attribute msRTCSIP-PrimaryHomeServer associated to the Lync Server 2013 Director Pool.
These users were previously moved to Skype for Business Online, during the move the attribute was updated to the Director Pool that was configured as federation route.
Please note that this behaviour can also happen if we have a Front End Server Pool in the federation route.

Because the msRTCSIP-PrimaryHomeServer attribute isn’t used by Skype for Business Online we can clear it.

The first step is to get the Pool Distinguished Name and the quickest way is using View Logs in the Publishing Wizard:

Then, we Expand all Actions and scroll down to Check Orphaned Users:

In this example the Pool Distinguished Name is:

CN=Lc Services,CN=Microsoft,CN=1:8,CN=Pools,CN=RTC Service,CN=Services,CN=Configuration,DC=uclobby,DC=com

And we assign it to a variable ($PoolDN):

$PoolDN=”CN=Lc Services,CN=Microsoft,CN=1:8,CN=Pools,CN=RTC Service,CN=Services,CN=Configuration,DC=uclobby,DC=com”
$PoolDN

Alternatively, we can use PowerShell to get the Pool Distinguished Name:

Import-module ActiveDirectory
$RTCDN = “AD:\CN=*,CN=Pools,CN=RTC Service,CN=Services,CN=Configuration,”+(Get-ADDomain).distinguishedname
$PoolDN=”CN=Lc Services,CN=Microsoft,”+(Get-ItemProperty -Path $RTCDN -Name dNSHostName,distinguishedname | ?{$_.dNSHostName -eq “<POOL FQDN>“}).distinguishedname

Now we can list the all users that have msRTCSIP-PrimaryHomeServer attribute associated to the pool:

Get-CsUser -LDAPFilter “(msRTCSIP-PrimaryHomeServer=$PoolDN)” | Select SamAccountName,DisplayName,SipAddress,HostingProvider | ft -AutoSize

Note: We can only use this workaround if the HostingProvider is sipfed.online.lync.com.

If we have few users we can simply clear the msRTCSIP-PrimaryHomeServer attribute manually:

Get-ADuser <USERACCOUNT> | Set-ADObject -Identity $_.distinguishedname -Clear “msRTCSIP-PrimaryHomeServer”

However, if we want to clear the attribute for all users associated to the Lync Server 2013 Director Pool we should use the following:

Get-ADObject -LDAPFilter “(& (msRTCSIP-PrimaryHomeServer=$PoolDN)(msRTCSIP-DeploymentLocator=sipfed.online.lync.com))” | Set-ADObject -Clear “msRTCSIP-PrimaryHomeServer”

Note: We added the msRTCSIP-DeploymentLocator since we can only clear the msRTCSIP-PrimaryHomeServer if the users were moved to Skype for Business Online.

After clearing the msRTCSIP-PrimaryHomeServer attribute we successfully remove the Lync Server 2013 Director Pool:

SfB Server – Prerequisite installation failed: SqlInstanceRtcLocal

Recently while adding a new Front End Server to the existing Skype for Business Enterprise Pool we got the following message on SfB Deployment Wizard Step 1:

Prerequisite installation failed: Prerequisite installation failed: SqlInstanceRtcLocal For more information, check your SQL Server log files. Log files are in the folder C:\Program Files\Microsoft SQL Server\MSSQL*.RtcLocal\MSSQL\Log, where the * represents your SQL Server version number. For example, SQL Server 2012 uses this path: C:\Program Files\Microsoft SQL Server\MSSQL11.RtcLocal\MSSQL\Log.

After attempting to run Step 1 a second time the error message was slightly different:

Prerequisite not satisfied: SupportedSqlRtcLocal: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: Shared Memory Provider, error: 40 – Could not open a connection to SQL Server)

The SQL Server (RTCLOCAL) service was installed but stopped:

We tried to start the service without success:

Windows could not start the SQL Server (RTCLOCAL) on Local Computer. For more information, review the System Event Log. If this is a non-Microsoft service, contact the service vendor, and refer to service-specific error code 5023.

And looking in Event Viewer > Windows Logs > System we could find two related errors:

Log Name: System
Source: Schannel
Date: 16/10/2017 18:35:40
Event ID: 36871
Task Category: None
Level: Error
Keywords:
User: SYSTEM
Computer: sfbfe04bck.recore.lab
Description:
A fatal error occurred while creating an SSL client credential. The internal error state is 10013.

Log Name: System
Source: Service Control Manager
Date: 16/10/2017 18:35:41
Event ID: 7024
Task Category: None
Level: Error
Keywords: Classic
User: N/A
Computer: sfbfe04bck.recore.lab
Description:
The SQL Server (RTCLOCAL) service terminated with the following service-specific error:
The group or resource is not in the correct state to perform the requested operation.

The error state 10013 is related to Enabled Protocols, we checked the enabled protocols and on this particular server TLS 1.0 was disabled for client and server:

Get-ChildItem -Path “HKLM:SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS 1.0”

To re-enable TLS 1.0, we modified the following registry keys:

Set-ItemProperty -Path ‘HKLM:\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS 1.0\Client’ -Name DisabledByDefault -Value ‘0’ -Type Dword
Set-ItemProperty -Path ‘HKLM:\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS 1.0\Client’ -Name Enabled -Value ‘1’ -Type Dword

Set-ItemProperty -Path ‘HKLM:\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS 1.0\Server’ -Name DisabledByDefault -Value ‘0’ -Type Dword
Set-ItemProperty -Path ‘HKLM:\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS 1.0\Server’ -Name Enabled -Value ‘1’ -Type Dword

Note: After enable/disable protocols or cipher suites we need to restart the server.

Because the SQL Server Express installation failed, we also had to remove the RTCLOCAL instance by going to Control Panel > Programs > Programs and Features > Uninstall a program, select the SQL Server 2014 and then Uninstall/Change:

Now we use the option to Remove:

We will be prompted to remove the RTCLOCAL:

And we only need to remove the Database Engine Services:

In Ready to Remove we select Remove and wait for the RTCLOCAL to be removed:

Please also make sure that all the database files (*.mdf and *.ldf) related to the RTCLOCAL were removed:

(Get-ChildItem “C:\Program Files\Microsoft SQL Server\*RTCLOCAL” -Include *.mdf,*.ldf -Recurse).count

Since we remove the RTCLOCAL instance we should restart the server again.

Finally, we should be able to successful run Deployment Wizard Step 1:

Please note that currently it’s not supported to disable TLS 1.0 on any role related to Lync Server 2010/2013 and Skype for Business Server 2015.

As announced at Ignite 2017 the support will be available for Skype for Business Server 2015 in a future update.

SfB Server 2015: Event 57005, LS User Store Sync Agent – Could not find stored procedure XdsQueryCriticalDocumentSignatures

While updating our SfB Server 2015 lab, we notice that a recently updated Front End server had multiple errors and warnings in the Event Viewer:

Log Name: Lync Server
Source: LS User Store Sync Agent
Date: 12/09/2017 21:54:33
Event ID: 57005
Task Category: (1061)
Level: Error
Keywords: Classic
User: N/A
Computer: sfbfe03.recore.lab
Description:
Error encountered pushing data to RtcXds Blob Store

#CTX#{ctx:{traceId:10006, activityId:”e40d8197-4293-4146-9d72-03c0c2957f6c”}}#CTX#
Push cycle identifier: [sfbfe03.recore.lab.2fd688f5-0f3a-407f-bab5-3fa5c3757443]
ItemCount: [0]
Error Message: [PushController: XdsQueryCriticalDocumentSignatures failed: System.Data.SqlClient.SqlException (0x80131904): Could not find stored procedure ‘XdsQueryCriticalDocumentSignatures’.
at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
at System.Data.SqlClient.SqlDataReader.TryConsumeMetaData()
at System.Data.SqlClient.SqlDataReader.get_MetaData()
at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)
at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, SqlDataReader ds)
at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, TaskCompletionSource`1 completion, Int32 timeout, Task& task, Boolean asyncWrite)
at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method)
at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior, String method)
at System.Data.SqlClient.SqlCommand.ExecuteReader()
at Microsoft.Rtc.Common.Data.DBCore.Execute(SprocContext sprocContext, SqlConnection sqlConnection, SqlTransaction sqlTransaction)
ClientConnectionId:e597ef79-3a87-4d08-8561-8e8c0db10e37
Error Number:2812,State:62,Class:16]
Cause: Possible issues with back-end database.
Resolution:
Ensure the back-end is functioning correctly.

Log Name: Lync Server
Source: LS User Store Sync Agent
Date: 12/09/2017 21:54:33
Event ID: 57006
Task Category: (1061)
Level: Warning
Keywords: Classic
User: N/A
Computer: sfbfe03.recore.lab
Description:
RtcDb Sync Agent sproc failed

#CTX#{ctx:{traceId:10006, activityId:”e40d8197-4293-4146-9d72-03c0c2957f6c”}}#CTX#
Sproc: [XdsQueryCriticalDocumentSignatures]
Exception: [System.Data.SqlClient.SqlException (0x80131904): Could not find stored procedure ‘XdsQueryCriticalDocumentSignatures’.
at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
at System.Data.SqlClient.SqlDataReader.TryConsumeMetaData()
at System.Data.SqlClient.SqlDataReader.get_MetaData()
at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)
at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, SqlDataReader ds)
at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, TaskCompletionSource`1 completion, Int32 timeout, Task& task, Boolean asyncWrite)
at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method)
at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior, String method)
at System.Data.SqlClient.SqlCommand.ExecuteReader()
at Microsoft.Rtc.Common.Data.DBCore.Execute(SprocContext sprocContext, SqlConnection sqlConnection, SqlTransaction sqlTransaction)
ClientConnectionId:e597ef79-3a87-4d08-8561-8e8c0db10e37
Error Number:2812,State:62,Class:16]

The error message mentions that XdsQueryCriticalDocumentSignatures store procedure is missing from RTCXDS database, after checking the version for this particular database, we notice that a new version was available:

Test-CsDatabase -ConfiguredDatabases -SqlServerFqdn sqlpool.recore.lab | Select SqlServerFqdn, SqlInstanceName, DatabaseName, InstalledVersion, ExpectedVersion | ft -AutoSize

A complete database version list for Skype for Business Server 2015 is available here:

Doug Deitterick’s Blog – How to Verify if Skype for Business Server 2015 Database Updates Completed Successfully
https://blogs.technet.microsoft.com/dodeitte/2015/05/10/how-to-verify-if-skype-for-business-server-2015-database-updates-completed-successfully/

Please note that XdsQueryCriticalDocumentSignatures store procedure was added in the May 2017 Cumulative Update for SfB Server 2015.

After we finish updating the remaining Front End servers and updated the RTCXDS database on the SfB Back End, we didn’t get more Errors/Warnings related to the missing XdsQueryCriticalDocumentSignatures store procedure.

We need to make sure that we follow the steps described for each Lync/SfB Server version:

Updates for Lync Server 2010
http://support.microsoft.com/kb/2493736

Updates for Lync Server 2013
http://support.microsoft.com/kb/2809243

Updates for Skype for Business Server 2015
http://support.microsoft.com/kb/3061064

SfB Server: Cannot install KB2982006 – This update is not applicable to your computer.

Update 2018/01/31 – Please note that the fix for this is included in the Skype for Business January 2018 Cumulative Update (CU6 HF1)

Prerequisite (KB2982006) not satisfied when you try to install Skype for Business Server 2015
https://support.microsoft.com/kb/4056288

In a recent deployment with all the Windows Updates installed, including the KB2919442 and KB2919355,  we couldn’t install the KB2982006:

We also got the same error in the Event Viewer > Windows Logs > Setup:

Log Name: Setup
Source: Microsoft-Windows-WUSA
Date: 04/09/2017 18:48:44
Event ID: 3
Task Category: None
Level: Error
Keywords:
User: RECORE\Administrator
Computer: sfbstd.recore.lab
Description:
Windows update could not be installed because of error 2149842967 “” (Command line: “”C:\Windows\system32\wusa.exe” “C:\UCLobby\Windows8.1-KB2982006-x64.msu” “)

The good news is we can use the DISM tool to manually add the KB2982006.

Before using DISM we need to make sure that KB2919442 and KB2919355 are already installed:

Get-Hotfix KB2919442,KB2919355,KB2982006

Please note that we cannot use a .msu file if we use the Online switch with DISM:

dism /Online /Add-Package /PackagePath:C:\UCLobby\Windows8.1-KB2982006-x64.msu

We have to expand the .msu file first using the steps described here:

How to use DISM to install a hotfix from within Windows
https://blogs.technet.microsoft.com/askcore/2011/02/15/how-to-use-dism-to-install-a-hotfix-from-within-windows/

An easy way is to create a folder and copy the Windows8.1-KB2982006-x64.msu file and also create a KB2982006 sub-folder:

Then we expand the .msu with the following cmdlet:

Expand -F:* C:\UCLobby\Windows8.1-KB2982006-x64.msu C:\UCLobby\KB2982006
https://technet.microsoft.com/en-gb/library/cc722332(v=ws.10).aspx

Since we have a .cab file we can proceed and manually add the KB2982006 with DISM:

dism /Online /Add-Package /PackagePath:C:\UCLobby\KB2982006\Windows8.1-KB2982006-x64.cab

With Get-Hotfix we confirm that KB2919442, KB2919355 and KB2982006 are installed:

Get-Hotfix KB2919442,KB2919355,KB2982006

Finally, we run Step 2 and confirm that the check for KB2982006 is successful:

 

Lync/SfB: Quickly access the Certificate Store

In a previous post we wrote about the Checks to do in the Lync/Skype for Business Server Certificate Store, however, sometimes we might also want to manually check it using the Certificate Store MMC.

Since Windows Server 2012 and Windows 8 we can quick access the Certificate Store MMC, for Local Computer and Current User, using Command Prompt/PowerShell or the Windows Search:

Local Computer

certlm

Note: Using the Windows Search we need to add the .msc – certlm.msc

Current User

certmgr

Note: Using the Windows Search we need to add the .msc – certmgr.msc 

Please also check the original post:

PKI Tip: Certificate Store Shortcuts
https://blogs.technet.microsoft.com/xdot509/2013/06/10/pki-tip-certificate-store-shortcuts/

Lync/SfB Server: Stop Front End service in Starting state

Some of the cases we work have the Front End service in a Starting state:

In PowerShell the status is StartPending:

We cannot stop it on the Services Management Console:

We can go to Task Manager and try to manual stop the service:

If that doesn’t work we need to Go to Details and End task that is associated with the service:

Another simple way to stop all Starting services is by using the following PowerShell cmdlet. We will get a prompt for each service:

Get-Service | ?{$_.Status -eq “StartPending”} | Stop-process

Get-Service
https://msdn.microsoft.com/en-us/powershell/reference/5.1/microsoft.powershell.management/get-service

Stop-Process
https://msdn.microsoft.com/en-us/powershell/reference/5.1/microsoft.powershell.management/stop-process

SfB Server 2015: Pool Pairing with CMS and AlwaysON

We already publish guides to Deploying SQL Server AlwaysOn Availability Group for Skype for Business Server 2015 and also SfB Server: Moving Central Management to a pool with SQL Server AlwaysOn BackEnd.

However, we were asked to create another guide when we want to pair two SfB Enterprise Pools where the Primary Pool is hosting the Central Management Store (CMS).

Please note that in this scenario we use the SQL Server Defaults Paths.

Step 1 – Create CMS database in secondary pool back end

First, we need to take note of which SQL Server node is Primary in the SfB Backup Pool. In the following example, SQL01BCK is the active node:

Now in a Skype for Business PowerShell execute the following cmdlet:

Install-CsDatabase -CentralManagementDatabase -SqlServerFqdn SQL01BCK.recore.lab -SqlInstanceName SFBBEBCK -UseDefaultSqlPaths
https://technet.microsoft.com/en-us/library/gg399044(v=ocs.16).aspx

Note: We need to specify the FQDN of the SQL Server active node and not the AlwaysOn SQL Listener.

The databases are created but not part of the AlwaysOn Availability Group:

Step 2 – Add the CMS databases to the AlwaysOn Availabilty Group

Open a PowerShell on the active SQL Server in the Backup Pool Back End, and set the Recovery to Full and Perform a Full Backup:

Invoke-Sqlcmd -ServerInstance SQL01BCK\SFBBEBCK -Query “ALTER DATABASE [xds] SET RECOVERY FULL WITH NO_WAIT;”
Invoke-Sqlcmd -ServerInstance SQL01BCK\SFBBEBCK -Query “ALTER DATABASE [lis] SET RECOVERY FULL WITH NO_WAIT;”

Backup-SqlDatabase -ServerInstance SQL01BCK\SFBBEBCK -Database xds
Backup-SqlDatabase -ServerInstance SQL01BCK\SFBBEBCK -Database lis

Since in this scenario we use the SQL Server Defaults Paths, we don’t need to copy the folder structures using RoboCopy.

Now in SQL Management Studio, right click in the existent AlwaysOn Availability Group and Add Database:

In the Wizard, select both CMS databases:

Like when we configured AlwaysOn we need to specify a temporary shared folder:

Make sure all check in the validation are successful:

And finally the CMS databases will be added to the AlwaysOn Availability Group:

Step 3 – Add the necessary permissions to the secondary SQL Server node

In the previous guides related to AlwaysOn it was suggested to change the topology builder, however, we can simplify this without republishing the topology.

In the SQL Management Studio failover the AlwaysOn Availability Group:

Select the New Primary Replica:

After connecting to replica, the failover should be successful:

Back in the Skype for Business PowerShell and we execute the following cmdlet:

Install-CsDatabase -Update -CentralManagementDatabase -SqlServerFqdn SQL02BCK.recore.lab -SqlInstanceName SFBBEBCK -UseDefaultSqlPaths

Step 4 – Configure Pool Pairing

In the Topology Builder, edit the Primary Pool and associate the Backup Pool:

Now we publish the topology but unchecked the CMS creation since we already manually created it:

Here is the to-do list:

Update Skype for Business Server with the changes defined in the topology by running local Setup on each server in the following list.
Important: Server changes made in Topology Builder must replicate to the servers in your topology. Please confirm that replication has been successful before proceeding setup.
Server FQDN: sfbfe01.recore.lab, Pool FQDN: sfbpool.recore.lab
Server FQDN: sfbfe02.recore.lab, Pool FQDN: sfbpool.recore.lab
Server FQDN: sfbfe03.recore.lab, Pool FQDN: sfbpool.recore.lab
Server FQDN: sfbfe01bck.recore.lab, Pool FQDN: sfbpoolbck.recore.lab
Server FQDN: sfbfe02bck.recore.lab, Pool FQDN: sfbpoolbck.recore.lab
Server FQDN: sfbfe03bck.recore.lab, Pool FQDN: sfbpoolbck.recore.lab

The databases listed are not part of an AlwaysOn Availability Group. You can use the New Availability Group Wizard in the SQL Server Management Studio to create an Availability Group. You should make sure that the databases are installed before running the ‘New Availability Group Wizard’.
SQL Server instance: sqlpoolbck.recore.lab\sfbbebck, Stores: CentralMgmt

Run the Invoke-CsBackupServiceSync cmdlet to ensure conferencing data is replicated.
Invoke-CsBackupServiceSync -PoolFqdn sfbpool.recore.lab
Invoke-CsBackupServiceSync -PoolFqdn sfbpoolbck.recore.lab

On all SfB Front End servers that are part of both pools we need to run SfB Deployment Wizard Step 2:

After Step 2, the Backup Service will be installed on the Front End Servers that belong to the Primary Pool:

And in the Front End Servers that are part of Backup Pool will have Backup, FTA and Master Replica Services:

Start the stopped services, invoke the backup sync and verify that it was successful:

Invoke-CsBackupServiceSync -PoolFqdn sfbpool.recore.lab
Invoke-CsBackupServiceSync -PoolFqdn sfbpoolbck.recore.lab
https://technet.microsoft.com/en-us/library/jj205374(v=ocs.16).aspx

Get-CsBackupServiceStatus -PoolFqdn sfbpool.recore.lab | fl
Get-CsBackupServiceStatus -PoolFqdn sfbpoolbck.recore.lab | fl
https://technet.microsoft.com/en-us/library/jj205032(v=ocs.16).aspx