While troubleshooting an issue where the Front End Service wasn’t starting, we notice that the Event Viewer > Applications and Services Logs > Lync Server had references to the local database unavailability:
Log Name: Lync Server
Source: LS User Services
Date: 02/01/2019 14:18:40
Event ID: 32122
Task Category: (1006)
Level: Warning
Keywords: Classic
User: N/A
Computer: sfbfe01.recore.lab
Description:
Skype for Business Server Service Startup Pending as Local BackEnd is not yet available.
Service Startup will be delayed until the Local BackEnd becomes available.
Cause: Possible Issues with the Local BackEnd.
Resolution:
Ensure that the Local BackEnd SQL Server is running on this machine.
Both SQL Services were up and running, so we looked at the events in the Event Viewer > Windows Logs > Application and here we had errors while accessing the SQL data files:
Log Name: Application
Source: MSSQL$RTCLOCAL
Date: 02/01/2019 14:11:50
Event ID: 17207
Task Category: Server
Level: Error
Keywords: Classic
User: N/A
Computer: sfbfe01.recore.lab
Description:
FileMgr::StartLogFiles: Operating system error 2(The system cannot find the file specified.) occurred while creating or opening file ‘D:\CsData\RtcDatabaseStore\rtclocal\LogPath\rtc.ldf’. Diagnose and correct the operating system error, and retry the operation.
Log Name: Application
Source: MSSQL$RTCLOCAL
Date: 02/01/2019 14:17:29
Event ID: 18456
Task Category: Logon
Level: Information
Keywords: Classic,Audit Failure
User: NETWORK SERVICE
Computer: sfbfe01.recore.lab
Description:
Login failed for user ‘NT AUTHORITY\NETWORK SERVICE’. Reason: Failed to open the explicitly specified database ‘rtc’. [CLIENT: <local machine>]
Two drives failed and had to be replaced, then we tried to rebuild the databases with the Deployment Wizard > Step 2 (bootstrap):
An error occurred while applying SQL script for the feature RtcDatabaseStore. For details, see the log file ‘C:\Users\Administrator.RECORE\AppData\Local\Temp\2\Create-RtcDatabaseStore-sfbfe01.recore.lab_rtclocal-[2019_01_02][14_52_01].log’
Database ‘lyss’ exists but not accessible. It must be repaired manually or dropped and a new one created. if you want to preserve data, you must use this product’s backup/export restore/import solution. Examine the product documentation for instructions.
Database ‘lyss’ exists but not accessible. It must be repaired manually or dropped and a new one created. if you want to preserve data, you must use this product’s backup/export restore/import solution. Examine the product documentation for instructions.
We also tried rebuilding the local databases using PowerShell:
Install-CsDatabase -Clean -LocalDatabases
Since we didn’t had a backup, we decided to rebuild the SQL Local databases from scratch.
To rebuild we need to manually drop the local databases, we can achieve this using PowerShell on the affected Front End Server:
Invoke-Sqlcmd -ServerInstance “.\RTCLOCAL” -Query “Drop database rtc, rtcdyn;”
Invoke-Sqlcmd -ServerInstance “.\LYNCLOCAL” -Query “Drop database lyss;”
Now we can rebuild the local databases using the following PowerShell cmdlet:
Install-CsDatabase -DatabasePaths D:\CsData -LocalDatabases
Note: In this particular case the SQL data files were located on the D drive.
After rebuilding the SQL Local Databases it’s recommended to reboot the server.
Now the Front End Service can connect to the local database and the service should start, please note that it might take some time since the server needs to recreate all the data:
Log Name: Lync Server
Source: LS User Services
Date: 02/01/2019 15:16:02
Event ID: 30960
Task Category: (1006)
Level: Information
Keywords: Classic
User: N/A
Computer: sfbfe01.recore.lab
Description:
Successfully connected to database “rtc” using the connection string of
driver={SQL Server Native Client 11.0};Trusted_Connection=yes;AutoTranslate=no;server=(local)\rtclocal;database=rtc;
