SfB Server – Prerequisite installation failed: SqlInstanceRtcLocal

Recently while adding a new Front End Server to the existing Skype for Business Enterprise Pool we got the following message on SfB Deployment Wizard Step 1:

Prerequisite installation failed: Prerequisite installation failed: SqlInstanceRtcLocal For more information, check your SQL Server log files. Log files are in the folder C:\Program Files\Microsoft SQL Server\MSSQL*.RtcLocal\MSSQL\Log, where the * represents your SQL Server version number. For example, SQL Server 2012 uses this path: C:\Program Files\Microsoft SQL Server\MSSQL11.RtcLocal\MSSQL\Log.

After attempting to run Step 1 a second time the error message was slightly different:

Prerequisite not satisfied: SupportedSqlRtcLocal: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: Shared Memory Provider, error: 40 – Could not open a connection to SQL Server)

The SQL Server (RTCLOCAL) service was installed but stopped:

We tried to start the service without success:

Windows could not start the SQL Server (RTCLOCAL) on Local Computer. For more information, review the System Event Log. If this is a non-Microsoft service, contact the service vendor, and refer to service-specific error code 5023.

And looking in Event Viewer > Windows Logs > System we could find two related errors:

Log Name: System
Source: Schannel
Date: 16/10/2017 18:35:40
Event ID: 36871
Task Category: None
Level: Error
Keywords:
User: SYSTEM
Computer: sfbfe04bck.recore.lab
Description:
A fatal error occurred while creating an SSL client credential. The internal error state is 10013.

Log Name: System
Source: Service Control Manager
Date: 16/10/2017 18:35:41
Event ID: 7024
Task Category: None
Level: Error
Keywords: Classic
User: N/A
Computer: sfbfe04bck.recore.lab
Description:
The SQL Server (RTCLOCAL) service terminated with the following service-specific error:
The group or resource is not in the correct state to perform the requested operation.

The error state 10013 is related to Enabled Protocols, we checked the enabled protocols and on this particular server TLS 1.0 was disabled for client and server:

Get-ChildItem -Path “HKLM:SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS 1.0”

To re-enable TLS 1.0, we modified the following registry keys:

Set-ItemProperty -Path ‘HKLM:\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS 1.0\Client’ -Name DisabledByDefault -Value ‘0’ -Type Dword
Set-ItemProperty -Path ‘HKLM:\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS 1.0\Client’ -Name Enabled -Value ‘1’ -Type Dword

Set-ItemProperty -Path ‘HKLM:\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS 1.0\Server’ -Name DisabledByDefault -Value ‘0’ -Type Dword
Set-ItemProperty -Path ‘HKLM:\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS 1.0\Server’ -Name Enabled -Value ‘1’ -Type Dword

Note: After enable/disable protocols or cipher suites we need to restart the server.

Because the SQL Server Express installation failed, we also had to remove the RTCLOCAL instance by going to Control Panel > Programs > Programs and Features > Uninstall a program, select the SQL Server 2014 and then Uninstall/Change:

Now we use the option to Remove:

We will be prompted to remove the RTCLOCAL:

And we only need to remove the Database Engine Services:

In Ready to Remove we select Remove and wait for the RTCLOCAL to be removed:

Please also make sure that all the database files (*.mdf and *.ldf) related to the RTCLOCAL were removed:

(Get-ChildItem “C:\Program Files\Microsoft SQL Server\*RTCLOCAL” -Include *.mdf,*.ldf -Recurse).count

Since we remove the RTCLOCAL instance we should restart the server again.

Finally, we should be able to successful run Deployment Wizard Step 1:

Please note that currently it’s not supported to disable TLS 1.0 on any role related to Lync Server 2010/2013 and Skype for Business Server 2015.

As announced at Ignite 2017 the support will be available for Skype for Business Server 2015 in a future update.

SfB Server 2015: Pool Pairing with CMS and AlwaysON

We already publish guides to Deploying SQL Server AlwaysOn Availability Group for Skype for Business Server 2015 and also SfB Server: Moving Central Management to a pool with SQL Server AlwaysOn BackEnd.

However, we were asked to create another guide when we want to pair two SfB Enterprise Pools where the Primary Pool is hosting the Central Management Store (CMS).

Please note that in this scenario we use the SQL Server Defaults Paths.

Step 1 – Create CMS database in secondary pool back end

First, we need to take note of which SQL Server node is Primary in the SfB Backup Pool. In the following example, SQL01BCK is the active node:

Now in a Skype for Business PowerShell execute the following cmdlet:

Install-CsDatabase -CentralManagementDatabase -SqlServerFqdn SQL01BCK.recore.lab -SqlInstanceName SFBBEBCK -UseDefaultSqlPaths
https://technet.microsoft.com/en-us/library/gg399044(v=ocs.16).aspx

Note: We need to specify the FQDN of the SQL Server active node and not the AlwaysOn SQL Listener.

The databases are created but not part of the AlwaysOn Availability Group:

Step 2 – Add the CMS databases to the AlwaysOn Availabilty Group

Open a PowerShell on the active SQL Server in the Backup Pool Back End, and set the Recovery to Full and Perform a Full Backup:

Invoke-Sqlcmd -ServerInstance SQL01BCK\SFBBEBCK -Query “ALTER DATABASE [xds] SET RECOVERY FULL WITH NO_WAIT;”
Invoke-Sqlcmd -ServerInstance SQL01BCK\SFBBEBCK -Query “ALTER DATABASE [lis] SET RECOVERY FULL WITH NO_WAIT;”

Backup-SqlDatabase -ServerInstance SQL01BCK\SFBBEBCK -Database xds
Backup-SqlDatabase -ServerInstance SQL01BCK\SFBBEBCK -Database lis

Since in this scenario we use the SQL Server Defaults Paths, we don’t need to copy the folder structures using RoboCopy.

Now in SQL Management Studio, right click in the existent AlwaysOn Availability Group and Add Database:

In the Wizard, select both CMS databases:

Like when we configured AlwaysOn we need to specify a temporary shared folder:

Make sure all check in the validation are successful:

And finally the CMS databases will be added to the AlwaysOn Availability Group:

Step 3 – Add the necessary permissions to the secondary SQL Server node

In the previous guides related to AlwaysOn it was suggested to change the topology builder, however, we can simplify this without republishing the topology.

In the SQL Management Studio failover the AlwaysOn Availability Group:

Select the New Primary Replica:

After connecting to replica, the failover should be successful:

Back in the Skype for Business PowerShell and we execute the following cmdlet:

Install-CsDatabase -Update -CentralManagementDatabase -SqlServerFqdn SQL02BCK.recore.lab -SqlInstanceName SFBBEBCK -UseDefaultSqlPaths

Step 4 – Configure Pool Pairing

In the Topology Builder, edit the Primary Pool and associate the Backup Pool:

Now we publish the topology but unchecked the CMS creation since we already manually created it:

Here is the to-do list:

Update Skype for Business Server with the changes defined in the topology by running local Setup on each server in the following list.
Important: Server changes made in Topology Builder must replicate to the servers in your topology. Please confirm that replication has been successful before proceeding setup.
Server FQDN: sfbfe01.recore.lab, Pool FQDN: sfbpool.recore.lab
Server FQDN: sfbfe02.recore.lab, Pool FQDN: sfbpool.recore.lab
Server FQDN: sfbfe03.recore.lab, Pool FQDN: sfbpool.recore.lab
Server FQDN: sfbfe01bck.recore.lab, Pool FQDN: sfbpoolbck.recore.lab
Server FQDN: sfbfe02bck.recore.lab, Pool FQDN: sfbpoolbck.recore.lab
Server FQDN: sfbfe03bck.recore.lab, Pool FQDN: sfbpoolbck.recore.lab

The databases listed are not part of an AlwaysOn Availability Group. You can use the New Availability Group Wizard in the SQL Server Management Studio to create an Availability Group. You should make sure that the databases are installed before running the ‘New Availability Group Wizard’.
SQL Server instance: sqlpoolbck.recore.lab\sfbbebck, Stores: CentralMgmt

Run the Invoke-CsBackupServiceSync cmdlet to ensure conferencing data is replicated.
Invoke-CsBackupServiceSync -PoolFqdn sfbpool.recore.lab
Invoke-CsBackupServiceSync -PoolFqdn sfbpoolbck.recore.lab

On all SfB Front End servers that are part of both pools we need to run SfB Deployment Wizard Step 2:

After Step 2, the Backup Service will be installed on the Front End Servers that belong to the Primary Pool:

And in the Front End Servers that are part of Backup Pool will have Backup, FTA and Master Replica Services:

Start the stopped services, invoke the backup sync and verify that it was successful:

Invoke-CsBackupServiceSync -PoolFqdn sfbpool.recore.lab
Invoke-CsBackupServiceSync -PoolFqdn sfbpoolbck.recore.lab
https://technet.microsoft.com/en-us/library/jj205374(v=ocs.16).aspx

Get-CsBackupServiceStatus -PoolFqdn sfbpool.recore.lab | fl
Get-CsBackupServiceStatus -PoolFqdn sfbpoolbck.recore.lab | fl
https://technet.microsoft.com/en-us/library/jj205032(v=ocs.16).aspx

SfB Server: Moving Central Management to a pool with SQL Server AlwaysOn BackEnd

In a previous post, we described how to configure AlwaysOn Availability Groups for Skype for Business Server 2015:

Deploying SQL Server AlwaysOn Availability Group for Skype for Business Server 2015
https://uclobby.wordpress.com/2015/05/08/deploying-sql-server-alwayson-availability-group-for-skype-for-business-server-2015/

That same post covers a green field deployment, but in this one we are going to work on a scenario where the xds and lis aren’t initially added to the AlwaysOn Availability Group.

Step 1  Install the Database on one of the SQL nodes

First, we need to check the active node in the AlwaysOn Availability Group, as we should use the SQL node FQDN instead of the SQL pool FQDN. If we try to use the SQL pool FQDN, we get this error:

“Install-CsDatabase : An error occurred while creating or updating the database for feature CentralMgmtStore. (…)”

As we can see, the Install-CsDatabase cmdlet is trying to use the \\sqlpool.borderlands.lab\C$. However, since this is the listener FQDN, it doesn’t connect to the SQL node file share and the cmdlet fails.

To check the active node we can use the SQL Server Management Studio. Connect to one of the SQL nodes and then expand AlwaysOn High Availability > Availability Groups > Group Name > Availability Replicas:

In this case, the SQL01 is the primary. Now we go back to the Skype for Business Server PowerShell and we can install the Central Management Database:

Install-CsDatabase -CentralManagementDatabase -SqlServerFqdn sql01.borderlands.lab -SqlInstanceName S4B_BE

Step 2  Add the CMS databases to the AlwaysOn Availability Group

Before adding the database to the AlwaysOn Availability Group, we need to change the database recovery to full and also perform a full backup. This can be achieved by running the following PowerShell cmdlets on the SQL01 PowerShell:

Invoke-Sqlcmd -Query “ALTER DATABASE [xds] SET RECOVERY FULL WITH NO_WAIT;” -ServerInstance “SQL01\S4B_BE”
Invoke-Sqlcmd -Query “ALTER DATABASE [lis] SET RECOVERY FULL WITH NO_WAIT;” -ServerInstance “SQL01\S4B_BE”

Backup-SqlDatabase -ServerInstance SQL01\S4B_BE -Database xds
Backup-SqlDatabase -ServerInstance SQL01\S4B_BE -Database lis

We should also copy the directory structure to the second SQL node:

robocopy C:\CsData \\SQL02\C$\CsData /e /xf *
https://technet.microsoft.com/en-GB/library/cc733145.aspx

Now everything is ready to add xds and lis to the existing AlwaysOn Availability Group. To do this we need to right click on the Availability Group and select Add Database…:

In the first step, we click Next:

We select the two databases (xds and lis):

We will need a temp file share for the initial sync:

Then we need to connect to the other SQL node:

If successful connected, the Connected As should change to the user that we use:

And then the Wizard will perform a validation check:

In the Summary, we click Finish:

Now the Wizard will execute the tasks to add the databases to the Availability Group:

After the Wizard finishes, the databases are shown as part of the Availability Group:

Step 3  Move the Central Management Store

This will be the normal procedure to move the CMS. Before moving, it’s recommended to perform a backup:

Export-CsConfiguration
https://technet.microsoft.com/en-us/library/gg398627.aspx

Export-CsLisConfiguration
https://technet.microsoft.com/en-us/library/gg398539.aspx

Next, in one of the Skype for Business Server pool Front End run:

Move-CsManagementServer

If the move is successful, we need to run Deployment Wizard > Step 2 on the remaining pool front ends and then start the newly added services: Master Replicator Agent and File Transfer Agent.

Finally, we should also run Deployment Wizard > Step 2 on the servers that previously had the Central Management role.

 

Deploying SQL Server AlwaysOn Availability Group for Skype for Business Server 2015

In Lync Server 2013, there were requests regarding an alternative to SQL Mirroring for SQL Server High Availability. This was related to the fact that SQL Mirroring was marked as a feature to be removed in future SQL Server versions:

This feature will be removed in a future version of Microsoft SQL Server. Avoid using this feature in new development work, and plan to modify applications that currently use this feature. Use AlwaysOn Availability Groups instead.
in SQL Server 2014 – Database Mirroring (SQL Server) – https://msdn.microsoft.com/en-us/library/ms189852.aspx

In Lync Server 2013, it was common to have SQL Server High Availability using SQL Mirroring. The reason for this was that Topology Builder did all the hard work for us. Another supported scenario was to use SQL failover clustering, but in this case we need to manually deploy it:

Database software support in Lync Server 2013
https://technet.microsoft.com/en-us/library/gg398990.aspx

The good news is Skype for Business Server 2015 comes with AlwaysOn Availability Groups:

Note: AlwaysOn Availability Groups requires SQL Server 2012/2014 Enterprise Edition.

For other supported scenarios, check the following:

Back End Server high availability in Skype for Business Server
https://technet.microsoft.com/en-us/library/jj205248.aspx

To deploy AlwaysOn Availability Groups for Skype for Business Server 2015, we need to follow specific steps. In this tutorial, we consider a lab environment with one Front End server and two SQL Server 2014 Enterprise Edition servers, which is a new environment without any previous Lync Server/OCS deployments.

Let’s start by installing and configuring the clustering service on both SQL Servers (SQL01 and SQL02). We can add new features by using the following PowerShell cmdlet:

Add-WindowsFeature Net-Framework-Core, Failover-Clustering, RSAT-Clustering-Mgmt,RSAT-Clustering-PowerShell -Source d:\sources\sxs
Note: The reason to use the source switch is that Windows Server 2012 R2 doesn’t install the source files. So, if your server doesn’t have internet access, you need to specify the path. In this case, the DVD is D:

Now that we have both servers with the necessary Windows Features, we can create the cluster. Before creating the cluster, we should test the configuration:

Test-Cluster -Node sql01,sql02
https://technet.microsoft.com/en-us/library/hh847274.aspx

In a lab environment, these warnings can be ignored, but in a production environment we need to check them before continue.
The Test-Cluster cmdlet will generate a Failover Cluster Validation Report:

After the test, we can create the cluster. For that, we can also use a PowerShell cmdlet. Since we don’t have DHCP in our lab subnet, we need a valid IP Address in the SQL Servers subnet:

New-Cluster -Name sqlcluster -Node sql01,sql02 -NoStorage -StaticAddress 172.20.15.8
https://technet.microsoft.com/en-us/library/hh847246.aspx

The New-Cluster will generate a Create Cluster report:

Before installing SQL Server we also need to configure the Cluster Quorum, we can use a File Share Witness:

Set-ClusterQuorum -Cluster sqlcluster -NodeAndFileShareMajority “\\dc01.gears.lab\SQLClusterWitness”
https://technet.microsoft.com/en-us/library/ee461013.aspx

Note: For additional information please go to Configure and Manage the Quorum in a Windows Server 2012 Failover Cluster

Now that we have the cluster with basic configuration, we can proceed and install SQL Server 2014 on both servers:

In Instance Features select at least Database Engine Services:

We can use Default instance for both servers:

Or change it to a different name. If you change it to a Named Instance, make sure both servers use the same instance name:

In Service Accounts, change the Account Name to a custom service account and use it on both SQL Servers:

After completing the installation, we need to enable AlwaysOn Availability Groups. On each server, we need to open SQL Server Configuration Manager, then right click on SQL Server Service and open Properties:

Select the AlwaysOn High Availability tab and tick Enable AlwaysOn Availability Groups:

For the changes to be applied, we need to restart SQL Server Service:

Skype4B-AOAG14

Select SQL Server Service, then click on the Restart service icon:

An additional step is to create a DNS A record for sqlpool.halo.lab. This is our Availability Group Listener FQDN:

In the Skype for Business Server 2015 Topology Builder, we add a new SQL Server Store with the following configuration:

Notice that we use the SQL01 server FQDN. This is normal and we will change it later on.

Now we publish the topology:

In SQL Server Management Studio, we can check that the Skype for Business Server 2015 related databases were successfully created in SQL01:

To create a new Availability Group, right click AlwaysOn High Availability and open New Availability Group Wizard…:

Fill the Availability Group name:

The wizard will check for prerequisites and will let us know that, before we proceed, the database recovery needs to be changed to full and also perform a full backup:

To make things easier, we can use the following PowerShell SQL cmdlets:

Back End databases:

Invoke-Sqlcmd -Query “ALTER DATABASE [cpsdyn] SET RECOVERY FULL WITH NO_WAIT;” -ServerInstance “SQL01\S4B_BackEnd”
Invoke-Sqlcmd -Query “ALTER DATABASE [rgsconfig] SET RECOVERY FULL WITH NO_WAIT;” -ServerInstance “SQL01\S4B_BackEnd”
Invoke-Sqlcmd -Query “ALTER DATABASE [rgsdyn] SET RECOVERY FULL WITH NO_WAIT;” -ServerInstance “SQL01\S4B_BackEnd”
Invoke-Sqlcmd -Query “ALTER DATABASE [rtcab] SET RECOVERY FULL WITH NO_WAIT;” -ServerInstance “SQL01\S4B_BackEnd”
Invoke-Sqlcmd -Query “ALTER DATABASE [rtcshared] SET RECOVERY FULL WITH NO_WAIT;” -ServerInstance “SQL01\S4B_BackEnd”
Invoke-Sqlcmd -Query “ALTER DATABASE [rtcxds] SET RECOVERY FULL WITH NO_WAIT;” -ServerInstance “SQL01\S4B_BackEnd”

Backup-SqlDatabase -ServerInstance SQL01\S4B_BackEnd -Database cpsdyn
Backup-SqlDatabase -ServerInstance SQL01\S4B_BackEnd -Database rgsconfig
Backup-SqlDatabase -ServerInstance SQL01\S4B_BackEnd -Database rgsdyn
Backup-SqlDatabase -ServerInstance SQL01\S4B_BackEnd -Database rtcab
Backup-SqlDatabase -ServerInstance SQL01\S4B_BackEnd -Database rtcshared
Backup-SqlDatabase -ServerInstance SQL01\S4B_BackEnd -Database rtcxds

CMS Databases:

Invoke-Sqlcmd -Query “ALTER DATABASE [xds] SET RECOVERY FULL WITH NO_WAIT;” -ServerInstance “SQL01\S4B_BackEnd”
Invoke-Sqlcmd -Query “ALTER DATABASE [lis] SET RECOVERY FULL WITH NO_WAIT;” -ServerInstance “SQL01\S4B_BackEnd”

Backup-SqlDatabase -ServerInstance SQL01\S4B_BackEnd -Database xds
Backup-SqlDatabase -ServerInstance SQL01\S4B_BackEnd -Database lis

Monitoring Databases:

Invoke-Sqlcmd -Query “ALTER DATABASE [LcsCDR] SET RECOVERY FULL WITH NO_WAIT;” -ServerInstance “SQL01\S4B_BackEnd”
Invoke-Sqlcmd -Query “ALTER DATABASE [QoEMetrics] SET RECOVERY FULL WITH NO_WAIT;” -ServerInstance “SQL01\S4B_BackEnd”

Backup-SqlDatabase -ServerInstance SQL01\S4B_BackEnd -Database LcsCDR
Backup-SqlDatabase -ServerInstance SQL01\S4B_BackEnd -Database QoEMetrics

Archiving Database:

Invoke-Sqlcmd -Query “ALTER DATABASE [LcsLog] SET RECOVERY FULL WITH NO_WAIT;” -ServerInstance “SQL01\S4B_BackEnd”

Backup-SqlDatabase -ServerInstance SQL01\S4B_BackEnd -Database LcsLog

Another requirement is that we copy the directory structure to the second SQL server:

robocopy C:\CsData \\SQL02\C$\CsData /e /xf *
https://technet.microsoft.com/en-GB/library/cc733145.aspx

Go back to the wizard, click Refresh and select the databases:

On the next step, click Add Replica…:

Change the server name and connect to the second SQL Server:

Select both SQL Instances in the Replicas tab:

We also need to create a listener, thus select the Listener tab and then select Create an availability group listener:

Note: As mentioned before, we don’t have DHCP on this Lab subnet, so we use a static address (different from the cluster).

Click Next and specify a temporary file share:

The wizard will run additional availability group validation checks:

And if everything goes okay, we get the following messages:

In AlwaysOn High Availability, we can check if the selected databases were included in the group:

Almost done. If we compare Security Logins for both servers, we can notice that some logins are missing from SQL02:

To add all the necessary permissions, we need to change the Primary Replica to the second SQL Server, right click on Availability Group and select Failover:

In the wizard, click Next:

We need to connect to the server:

If Failover is successful, we get this:

We can see that the Primary Replica is now the second SQL Server:

Time to go back to Topology Builder, select SQL Server Store and Edit Properties… :

Change the SQL Server FQDN to the second SQL Server:

Publish the topology.

In the Skype for Business Server 2015 server, open the PowerShell and run:

Install-CsDatabase -Update -ConfiguredDatabases -SqlServerFqdn sqlpool.halo.lab -Verbose

After completion, the necessary logins are also added to the second SQL server:

Finally, let’s change SQL Server Store in Topology Builder to the final value:

After publishing the topology, we now have Skype for Business Server 2015 with an AlwaysOn Availability Group configured.

Additional resource:

Chris Lehr experienced a few errors during the deployment and published the notes on his blog:

Chris and Robin’s Technology blog – SQL 2014 AlwaysOn Deployment for Skype for Business Server 2015 http://blog.chrislehr.com/2015/06/sql-2014-alwayson-deployment-for-skype.html