Lync/SfB Server: OAuthTokenIssuer, Assigned certificate not found or untrusted.

In a recent support case the OAuth certificate was missing in one of the Front Ends:

We also notice the Missing message in the Deployment Wizard Step 3, for the OAuth certificate:

And in PowerShell we had the following error when we tried to check the certificates:

Get-CsCertificate
https://technet.microsoft.com/en-us/library/gg398227.aspx

Get-CsCertificate : OAuthTokenIssuer: Assigned certificate not found or untrusted. Check that the certificate exists
in the certificate store, that it is not expired and that the certificate chain is valid.

Since the OAuth certificate is a Global setting and it’s replicated, we don’t need to request a new one.

To restore the OAuth certificate, we simply need to restart the Lync/SfB Server Replica Replicator Agent:

During start-up the Replica Replicator Agent will add the OAuth certificate again to the Computer Certificate Store:

We can also check the Deployment Wizard Step 3, to confirm that the correct certificate will be displayed:

For reference, here is the PowerShell output:

Get-CsCertificate -Type OAuthTokenIssuer

SfB Server 2015: Cannot install Cumulative Update – Failed to create network share. (-2147467259 xds-replica)

In a recent support case we couldn’t install the Skype for Business Server 2015 Cumulative Update on a Edge Server:

We checked the Skype for Business Core Components log file (OcsCore.msp-EDGESTD-[2017-03-21][15-30-48]_log.txt) located in %TEMP%:

The log showed that the Setup was aborted because it couldn’t create a file share:

CreateSmb: Error 0x80004005: failed to create share: ‘xds-replica’
MSI (s) (78!14) [15:32:19:981]: Product: Skype for Business Server 2015, Core Components — Error 26301. Failed to create network share. (-2147467259 xds-replica)

MSI (s) (78!14) [15:32:19:981]: Closing MSIHANDLE (193) of type 790531 for thread 3860
Error 26301. Failed to create network share. (-2147467259 xds-replica)

After checking the services running on the server we notice that Server Service (LanmanServer) was disabled:

Then we enable and started the service:

With Server Service running we were able to install the Skype for Business Cumulative Update:

Please note that if we try to repair the Skype for Business Server 2015 Core Components when the Server Service is disable we also get the following errors:

Failed to create network share. (-2147467259 xds-replica)

Failed to drop network share. (-2147022782 xds-replica)

SfB Server 2015: SIP/2.0 503 Service Unavailable

While working on a PSTN integration with Skype for Business Server 2015, we notice that the inbound calls were being rejected immediately with SIP/2.0 503 Service Unavailable:

TL_INFO(TF_PROTOCOL) [festd\festd]6984.23A4::03/06/2017-22:56:29.167.0000245B (S4,SipMessage.DataLoggingHelper:sipmessage.cs(801)) [4223311279]
>>>>>>>>>>>>Outgoing SipMessage c=[<SipTcpConnection_1ACDDA8>], 172.20.20.40:5068->172.20.15.39:56945
SIP/2.0 503 Service Unavailable
FROM: “Joule Adams”<sip:09920156101;phone-context=PrivateUnknown@recore.lab;user=phone>;tag=f0d14b5566;epid=8CE36BBBAB
TO: <sip:+449920207101@172.20.20.40;user=phone>;tag=9837f16c5d;epid=14123C3787
CSEQ: 12538 INVITE
CALL-ID: 49bfd2ea-8753-4651-a8aa-d7b48d066d93
VIA: SIP/2.0/TCP 172.20.15.39:56945;branch=z9hG4bKdab2e468
CONTENT-LENGTH: 0
SERVER: RTCC/6.0.0.0 MediationServer

The related SIP messages didn’t had any information why the Mediation Service immediately rejected the call.

However, in the Trace tab we found an error related to this call:

TL_ERROR(TF_COMPONENT) [festd\festd]6984.5B3C::03/06/2017-22:56:29.075.00002454 (MediationServer,ProxyCall.ProcessGatewayCallStateChanged:proxycall.cs(1446)) [25728949]
25728949Exception: System.MissingMethodException: Method not found: ‘Boolean Microsoft.Rtc.Internal.Collaboration.Media.SdpMediaDescription.ValidateMediaDescription()’.
at Microsoft.RTC.MediationServerCore.ProxySdpProcessor.InternalTryParseSdp[TGlobalDescription,TMediaDescription](SdpContentDescription cd, ProxyOfferAnswerInfo& offerAnswer, String& parsingError)
at Microsoft.RTC.MediationServerCore.ProxySdpProcessor.TryParseOffer(SdpContentDescription offer, Boolean bypassOffer)
at Microsoft.RTC.MediationServerCore.ProxyCall.ProcessGatewayCallStateChanged(Object sender, CallChangeEventArgs args)NULL

TL_INFO(TF_PROTOCOL) [festd\festd]6984.5B3C::03/06/2017-22:56:29.168.0000245C (MediationServer,ProxyCall.TerminateSession:proxycall.cs(3822)) [25728949]
25728949Terminating Proxy side call with session state: Idle, sip-response: 503, ms-diag: 10014, ms-diag reason: An internal exception received while processing the incoming request.NULL

This happened because the Enterprise Voice and Dialin Conferencing features were added after the Front End server update:

Then we update these components with the Skype for Business Server 2015 Update Installer:

Finally the inbound call will work without issues:

Lync Server: Event 41029 LS Data MCU – No connectivity with the Lync Web App

In a recent support case we were working on an issue where sometimes the users couldn’t use the Lync Web App.

The troubleshooting started in the Event Viewer > Lync Server, we notice that we had a few errors:

Log Name: Lync Server
Source: LS Data MCU
Date: 01/03/2017 15:00:43
Event ID: 41029
Task Category: (1018)
Level: Error
Keywords: Classic
User: N/A
Computer: lync2013fe01.gears.lab
Description:
No connectivity with the Lync Web App. Affected Web browser clients cannot use Web Conferencing modality.

Server Machine FQDN: lync2013fe01.gears.lab, Port:8061
Server Type: External-WebApp-Edge [HTTP side error:Unable to connect to the remote server]
If the problem persists this event will be logged again after 20 minutes
Cause: Service may be unavailable or Network connectivity may have been compromised.

Another error was mentioning an issue HTTP connectivity:

Log Name: Lync Server
Source: LS User Services
Date: 01/03/2017 15:04:57
Event ID: 30988
Task Category: (1006)
Level: Error
Keywords: Classic
User: N/A
Computer: lync2013fe01.gears.lab
Description:
Sending HTTP request failed. Server functionality will be affected if messages are failing consistently.

Sending the message to https://lync2013fe01.gears.lab:444/LiveServer/Replication failed. IP Address is 172.20.13.21. Error code is 2EFD. Content-Type is application/replication+xml. Http Error Code is 0.

Cause: Network connectivity issues or an incorrectly configured certificate on the destination server. Check the eventlog description for more information.
Resolution:
Check the destination server to see that it is listening on the same URI and it has certificate configured for MTLS. Other reasons might be network connectivity issues between the two servers.

Both events showed that the server could not establish a connection himself.

Then we check if the server was listening on that port:

netstat -anp TCP
https://technet.microsoft.com/en-us/library/bb490947.aspx

Get-NetTCPConnection -State Listen -LocalPort 80,8080,443,444,4443,8061 | ft -AutoSize
https://technet.microsoft.com/itpro/powershell/windows/tcpip/get-nettcpconnection

The HTTP/HTTPS bindings were only on 127.0.0.1 and this is the loopback address.

Then we run the same on a working server in the same pool:

Note: For Get-NetTCPConnection :: is any available IPV4/IPV6 address.

So, in a working server the binding was on any available IP address, while the non-working was only on the loopback address.

Initially, we thought the issue was in IIS/certificate bindings, but both were properly configured:

Get-WebBinding | ft -AutoSize
https://technet.microsoft.com/en-us/library/hh867866(v=wps.630).aspx

netsh http show sslcert
https://msdn.microsoft.com/en-us/library/windows/desktop/cc307236(v=vs.85).aspx

After checking other parameters available in netsh we found that the non-working had the loopback address configured in the HTTP IP Listen List:

netsh http show iplisten

While the working server we didn’t had any IP address configured:

This was causing the wrong binding, to fix it we only had to remove the loopback address from the list:

netsh http delete iplisten 127.0.0.1

After this change the server started to listen in the correct IP address/ports:

netstat -anp TCP

We also confirmed in the Event Viewer that the Lync Web App was starting: