깡민쓰~IT Story

In my Previous Article we discussed about the detailed call flow when a Skype for Business Desktop Client tries to sign in.

Its time to jump to the troubleshooting phase, where we are going to discuss on step by step approach, Collecting required logs and Using different tools that could help in identifying the issue cause.

Typical sign in issue would be that users not being able to sign in and getting one of below errors:

And many other errors, depending on what’s causing the issue:

Depending on the Error, issue cause could be in one of below:

1. SFB Client or Computer issues
2. Authentication or provisioning issues
3. Network/Connectivity related issues
4. Server related issues

Troubleshooting, Step by Step Approach:

Below approach could help us isolate the issue cause in a systematic manner.

1 : Verify if Server discovery is succeeding or not:

Manual vs Automatic Sign in

Simpler way to identify if DNS Records are the problem is by testing Manual Sign in vs Automatic Sign in method. To do that, we can edit the Advanced settings and manually provide Skype for Business Server/Pool name like below:

Skype for Business Client Settings -> Tools -> Personal -> Advanced

For Manual Sign in, provide the Skype for Business Pool name or Server name and the Port number manually in Internal or External Server name Box (depending on whether we are testing internal Sign in or External Remote Connectivity)

Internal Server Name : Front End Pool name & Port

External Server Name : External Access Edge FQDN & Port

If manual sign in worked, we know that Client is able to sign in when its connecting directly to Server/Pool using Manual settings, it’s just the Automatic sign in that is not functional, basically client is not able to determine where to connect to sign in.

For Automatic sign in to work, there are certain DNS Records (refer to Server Discovery step in previous article) that needs to be configured so that client can make DNS Query and get the Pool Info.

Skype for Business Client Log

Since we know from above step that automatic sign in is failing, we can look at client logs to see what all DNS Queries it made and whether it’s able to resolve DNS records or not.

If we have Logging Enabled on the client, we can open client Side Log ‘Lync-UccApi.Uccapilog’ present in Location “C:\Users\Mouli\AppData\Local\Microsoft\Office\1X.0\Lync\Tracing” using a notepad or any editor, below are sample log where client queries DNS for

Lyncdiscover Records:

SRV Records:

09/12/2016|19:30:07.502 1098:E10 INFO :: QueryDNSSrv - DNS Name[_sipinternaltls._tcp.contoso.com]09/12/2016|19:30:07.502 1098:E10 INFO :: QueryDNSSrv - DNS Name[_sip._tls.contoso.com]09/12/2016|19:30:07.502 1098:112C TRACE :: SIP_MSG_PROCESSOR::OnDnsResolutionComplete[0D2F4C30] Entered host lync2010-se.contoso.com09/12/2016|19:30:07.502 1098:112C TRACE :: SIP_MSG_PROCESSOR::OnDnsResolutionComplete get DNS result server: lync2010-se.contoso.com IP: 192.168.2.50:5061

Host A Records:

09/12/2016|19:34:00.56 1064 ERROR ResolveHostNameUsingGetAddrInfo - getaddrinfo(sipinternal.contoso.com)09/12/2016|19:34:00.567 1064 ERROR :: ResolveHostNameUsingGetAddrInfo - getaddrinfo(sip.contoso.com)09/12/2016|19:34:00.567 1064 ERROR :: ResolveHostNameUsingGetAddrInfo - getaddrinfo(sipexternal.contoso.com)

Nslookup Tool:

We could also make use of windows Builtin tool Nslookup to check the DNS Resolution.

Nslookup Lyncdiscover.contoso.com [For SRV Record we need to run ‘Set type=SRV’ ]

From above check we will know if DNS Records exists or not, based on that we could configure DNS Records to fix the issue.

Host File Entries:

Sometimes even if DNS Record is resolvable, automatic sign in might still be failing, as DNS Records might be pointing to a Load Balancer or Reverse Proxy or Firewall or different Skype for Business Pool which is having issues.

In such situation, we could use host file entries (admin rights is needed) to override DNS and point user to hit the Front end or Edge Server IP Address directly and test it out. Host file entries overrides DNS Queries, I, e, Client prefers host file entries over DNS query response.

By default, host file is located in “C:\Windows\System32\drivers\etc”

At the End of this stage, we will be in a state where we will know if the issue is specific to user's home server/Pool or whether its with different pool and due to the Redirection failure, based on the outcome we should take necessary actions to fix the issue.

2 : Verify if Network Connectivity is succeeding or not:

Now that we confirmed that DNS isn’t the problem, or takes steps to fix the DNS Issue, if sign in is still failing, next thing to check if whether client is able to make network connection to the Skype for business server and whether it was able to establish a TLS Session successfully or not.

We could make use of Telnet or Port Query tool (needs to be installed) to check if required ports are open and whether client is able to make successful connection. (if telnet is successful, you will see a Blank window in cmd prompt)

Telnet Sip.contoso.com 443

If telnet or Port Query is blocked on firewall (due to security restrictions), we can collect a Network trace from client to see whether connectivity check is succeeding or not.

If it’s able to connect, then we should see TCP 3 Way Handshake (A -> A,S -> A), if not we would see Retransmits, where client to trying to connect over the required port & its failing.

Based on the outcome, we could proceed further or we could work with Security team to get the required ports accessible for Client to be able to make connection to Server.

Other than above basic connectivity check, for client to be able to connect to Skype for Business Server, TLS Check is necessary. After successful Port connectivity check, client will try and establish a secure TLS connection, for this to work, below are some of things to consider:

Client should trust the Certificate issued on the Server (Client should contain the Root/Intermediate Certs)

FQDN/ URL that we are trying to access should be part of Certificate SAN entry on the Server where we are connecting to.

Required TLS Protocol & TLS cipher suites should be allowed in Firewall (If any)

3 : Check for Authentication or Server side Failures:

If we have surpassed above steps, then we should see SIP REGISTER Request being sent by client to initiate sign in process. This is another trick to identify, if all the above checks are successful is by opening UCCAPI Logs using Snooper Tool, if we see any REGISTER Messages being sent, which means client has performed all previous steps and then only sending SIP Requests.

Using Snooper Tool (Needs to be installed) we can open the Skype for Business Client side logs located in “C:\Users\Mouli\AppData\Local\Microsoft\Office\1X.0\Lync\Tracing” to see what's happening at the application level.

Depending the issue cause, sometimes Client logs might reveal the exact cause of the issue (for Ex : "Destination URI either not enabled for SIP or does not exist") and sometimes it would just give a generic Error (For Ex : 500 Internal Server Error )

Once we see which server is the source for generating this Error, we could take a look at server side Logs to see what's going on. Some of the Logs that we can look on the Server side are below:

Lync Server Event Logs (for any Server side or user specific Error, if logged)

Security Event Logs (for any Authentication failures, failure security audit)

Skype for Business CLS Logs (for detailed information on what's happening at SIP level)

Centralized Logging Service Logging is something that we can use to collect Server side logs while reproducing the issue. we could use the CLSLogger or the CLS Logging commands and collect the logs for IMAndPresence scenario (SIPStack & User Services) to get detailed information on what's going on when server gets the REGISTER request from the Skype for Business Client.

CLS Logging commands:

Start-CsClsLogging -Scenario "IMAndPresence" -Pools "SFB-SE.Contoso.com"

Stop-CsClsLogging -Scenario "IMAndPresence" -Pools "SFB-SE.Contoso.com"

Search-CsClsLogging -Pools "SFB-SE.Contoso.com" -Computers "SFB-SE.Contoso.com" -Components "SIPStack,UserServices" -StartTime "10/14/2016 6:09:51 PM" -EndTime "10/14/2016 6:14:51 PM" -LogLevel "All" -MatchAll -OutputFilePath "C:\Logs\SFBLog.txt"

4 : Troubleshooting Tools:

Fiddler Tool

If we are suspecting issues where users is not able to connect to HTTPS services like Lyncdiscover or Certificate Provisioning web services, to see client side request and response, we can collect Fiddler trace from client machine when we try to sign in.

Network Monitor

If we are suspecting Network connectivity or TLS Session establishment issue, we can use Network monitor Tool to collect network capture. Sometimes simultaneous capture from client and server would help identify if there are any firewall between blocking the packets

Snooper Tool

To view Client side UCCAPI logs or Server side CLS logs, we could make use of Snooper Tool to see SIP requests and responses.

Hope the above information helps, Happy Troubleshooting !

https://blogs.technet.microsoft.com/praj/2016/10/14/troubleshooting-skype-for-business-client-sign-in-issues/

Skype for Business Desktop client sign in issue is one of the most common scenario for Helpdesk or Admins or Support Folks who are working in Messaging or Unified communication field. While there are lot of awesome blogs right from the OCS Days explaining about the client sign in call flow, troubleshooting, Log Analysis and etc. I always use to prefer my OneNote page created by taking bits and pieces from different places that covers all these details. I thought sharing the info here might help in getting all the details in one go.

Before entering the troubleshooting phase, one should first understand the Skype for Business Client Sign in process flow to identity what’s expected and act accordingly. In this article, we will focus mainly on the Call flow when Skype for business Desktop Client login.

For simplicity, we could divide the entire Skype for Business Client Sign in process into below 5 steps:

1. Server Discovery
2. Connectivity Checks
3. Authentication
4. Optional Redirection
5. Retrieve Settings and Policies

1 : Server Discovery

Skype for Business Client is hardcoded to query certain DNS records to locate the Skype for business server information, which is required for Automatic Client sign in, below are the list of DNS records that client would query in order for Server discovery.

Lyncdiscover Records

Lyncdiscoverinternal.contoso.com

Lyncdiscover.contoso.com

SRV Records

_sipinternaltls._tcp.contoso.com

_sip._tls.contoso.com

DNS A Records

Sip.domain.com

Sipinternal.domain.com

Sipexternal.domain.com

At the End of this step, if we have DNS Records configured, skype for business client will get the FQDN/IP Address & Port combination of Skype for business server where it can reach to login.

2 : Connectivity Checks

Once Skype for Business client identified the Server Information, Client performs Network Connectivity checks to see if it can reach the server on identified IP address & Port combination and also it verifies if it can establish a TLS secure connection to the FQDN that it got in first step.

Port Connectivity Checks

Client attempts Network connectivity check to see if it can reach server on required port

In Networking terms this is termed as TCP 3 Way Handshake [ACK–SYNC–ACK]

TLS connectivity Checks

Client attempts to check if it can establish a secure connection with the server

In this, basically client will check if the certificate presented by server is being trusted on client or not; and it also includes Cipher Selection.

In order for Client to be able to trust the presented certificate, client should have the Root CA Cert of the Certification authority that has issued the certificate to the server in its Certificate Trusted Root Store.

3 : Authentication

This is the actual step where client interacts with the Skype for business server using SIP protocol and authenticates itself. Overall process involves Client learning the set of supported Authentication mechanism on the Skype for Business Registrar Servers and Selecting appropriate Authentication methods and getting authenticated.

Firstly, Client sends an Unauthenticated REGISTER Request to the Skype for Business Server:

In response to this REGISTER request, Skype for Business Server would send the list of Authentication mechanisms available for Authentication in 401 Unauthorized:

Client would then select one of the authentication methods and gets authenticated (depending on whether signing in internally or externally, first time sign in or subsequent sign in). By default, Skype for Business Registrar Configurations has below 3 Authentications enabled:

If Client is signing in Internally, then all 3 above listed Authentication methods [Kerberos/NTLM/TLS-DSK] will be available.

If Client is signing in Externally, then only 2 authentication methods will be available [NTLM/TLS-DSK] will be available.

If client uses Kerberos authentication

This is the default one that client uses internally during first time sign in I.e. when they don’t have user certificate to sign in using TLS DSK.

Client would reach out to AD Server and gets authentication ticket (Kerberos ticket) for accessing service on Skype for Business Server. Once it gets the Kerberos ticket, it submits that to Server in next REGISTER request, and server would authenticate the user and signs the user.(In Kerberos method, there is an interaction between clients and AD Servers, this is the primary reason why Kerberos Authentication isn't available for Remote Sign in)

When using Kerberos, in client side logs we will see 2 REGISTER Request/Responses between client and the skype server.

If client uses NTLM Authentication

This is the default one that client uses externally during first time sign in I.e. when they don’t have user certificate to sign in using TLS DSK

Client would send information/details required for authentication in the next REGISTER Requests to the skype for business server, skype for business server in turn talks to AD Server and validates the submitted information/details.

If the Validation succeeds then, skype for business server would consider user authentication as valid/genuine and signs the user. (In NTLM method, All the interaction is between client and the Skype for business server and Skype for business Server to Active directory, but no interaction from client directly with Active Directory)

When using NTLM, In Client side logs we will see 3 REGISTER Request/Responses between client and the skype server.

If Client uses TLS-DSK Authentication

In order for client to use this authentication, client should have user certificate issued by the Skype for business server.

Below is how client connects to the Skype for Business Web Services and gets the user certificate issued.

Client would get the location/URL of web services to get the user certificate from, this would be sent by server in first response for anonymous REGISTER sent. (Ex for URL : https://SFB-SEWeb.Contoso.com:443/CertProv/CertProvisioningService.svc)

Client would connect to skype for business Web Services and authenticates itself using Kerberos/NTLM depending on whether its connecting internally or externally and gets the user certificate issued.

Once user certificate is issued, Client would submit the user certificate details to the skype for business server in the next REGISTER and authenticates itself.

When using TLS-DSK, In Client side logs we will see 4 REGISTER Request/Responses between client and the skype server.

4 : Optional Redirection:

This step is optional, we might or might not see, depending on which server client reached.

If client reached out to Front End Server, where he is homed, then we wouldn’t see this step at all, however, if client reaches out and authenticates against any other front end server (within same pool or different pool) we will see this step, where server would identify where user is homed and redirects to the home Pool Accordingly.

In the above Example, Client reached SFB-FE1.contoso.com (since DNS was pointing here), which inturn redirected to User's home server "SFB-SE.Contoso.com"

5 : Retrieve Settings and Policies

This Step is post Successful Authentication where client/user retrieves different information such as Server side Settings, Policies applied to the user which includes details like normalization rules, what all features allowed, URLs to use when client needs to leverage certain services or modalities and so on.

In this step we would see client sending SERVICE/SUBSCRIBE SIP requests and getting required responses.

SERVICE Requesting for Normalization rules (Location Profile).

SUBSCRIBE Requesting for contact lists

SUBSCRIBE Requesting for Server side configurations/policies (Inband)

SUBSCRIBE Requesting for Presence info of users in the contact list

If we understood above call flow, then below story will sum it up on what happens when user tries to sign in:

1. User provides the sip uri Mouli@contoso.com on the skype for business desktop client and clicks on sign in.
2. Client first performs “Server Discovery step” and tries Lyncdiscover Records, if it’s not available, then it will failback to SRV Records and then host A Records.
3. At the end of Server discovery stage, client would get “Server FQDN/IP and the Port to be connecting to”.
4. Client would now attempt and check if it can establish Network connectivity and TLS Connectivity.
5. Post successful connectivity checks, client would now send the first anonymous sign in Request “REGISTER” to the server, to learn what all authentication methods are available.
6. Server would present the available Authentication methods and respective target/service name and other info.
7. Client now selects one of the Authentication methods depending on whether signing in internally or externally, first time sign in or subsequent sign in.
8. Upon successful authentication, if it’s not a home server, then Authenticated server will determine the user’s home server and redirect the user to go user’s home pool/Server.
9. Post successful sign in, client would send subsequent SERVICE/SUBSCRIBE requesting for inband configuration, policies & etc.

In my Next Article, I will be discussing in detail on Troubleshooting approach when dealing with Skype for business client sign in issues.

https://blogs.technet.microsoft.com/praj/2016/10/14/skype-for-business-client-sign-in-call-flow-detailed/

Deep Dive into the Microsoft Lync 2013 Client Sign-in Process

Whether you're using Lync Server, hybrid, hosted or online, the c

lient still needs to sign in to get everything started. Join this session to understand the internals of this process across the Lync clients, from Lyncdiscover to Edge—this session is technical and detailed around understanding the protocol flows and troubleshooting when things go sideways. 

Exploring Default Installation and SQL Paths in Lync Server 2013

http://howdouc.blogspot.com/2012/11/exploring-default-installation-and-sql.html

SfB Topology Builder – Select Database File Location

http://silbers.net/sfb-topology-builder-select-database-file-location/

How to Install Lync SQL Express to a Non-System Drive

https://guybachar.net/2014/03/29/how-to-install-lync-sql-express-to-a-non-system-drive/

Script: Set-Cs2013Features.ps1 – Easily Install Prerequisites and Tools for Microsoft Lync Server 2013

https://www.ucunleashed.com/1697

Lync Server 2013: Using the DatabasePathMap Parameter to Deploy Databases

https://blogs.technet.microsoft.com/nexthop/2012/11/20/lync-server-2013-using-the-databasepathmap-parameter-to-deploy-databases/

해결 방법 

SQL Express 명령어로 설치

SQLEXPR_x64.exe /IACCEPTSQLSERVERLICENSETERMS /HIDECONSOLE /ACTION=Install /FEATURES=SQLEngine,Tools /INSTANCENAME=RTCLOCAL /TCPENABLED=1 /SQLSVCACCOUNT=”NT AUTHORITY\NetworkService” /SQLSYSADMINACCOUNTS=”Builtin\Administrators” /BROWSERSVCSTARTUPTYPE=”Automatic” /AGTSVCACCOUNT=”NT AUTHORITY\NetworkService” /SQLSVCSTARTUPTYPE=Automatic

증상 : SQL Express 설치 안됨.

SqlInstanceRtcLocal with a failure code of -2067922934.

download : https://www.microsoft.com/en-us/download/confirmation.aspx?id=47263

Skype for Business Server 2015, Debugging Tools is a collection of additional tools for use by IT Admins to aid in troubleshooting Skype for Business Server 2015 deployments. The collection of tools include:

• Snooper
• CLSLogger
• CLSScenarioEdit.psm1

Stats Man and KHI released for Skype for Business Server 2019

Setup Step-by-Step For Real-Time Stats Manager

https://blogs.technet.microsoft.com/dodeitte/2015/10/24/skype-for-business-server-real-time-statistics-manager/

https://blogs.technet.microsoft.com/dodeitte/2016/04/25/skype-for-business-server-real-time-statistics-manager-1-1-available/

Skype for Business Server 실시간 통계 관리자에 대한 업데이트를 사용할 수 있습니다. 릴리스 1.1에서 다음과 같은 알려진 문제점이 수정되었습니다.

• UI / 서버 / 에이전트 - 신뢰성 및 성능 향상

• UI - 기본 필터 제어가 이제 다른 경우와 정확하게 정렬됩니다 (사람들이 특정 서버가 시스템에 없을 때 시스템에 있다고 생각하게 만들었습니다)

• 서버 - 이제 서버 구성 요소가 영어가 아닌 서버에 설치됩니다.

• 서버 / 에이전트 - 특정 버전의 .NET 4.0 때문에 에이전트 및 서버 구성 요소가 .NET 오류로 설치되지 않는 경우가 있습니다. 이 문제가 해결되었습니다.

• 에이전트 - Statsman 에이전트에 확장 이벤트 로깅이 추가되었습니다. 토폴로지가 아닌 서버에 에이전트를 설치하면 더 이상 크래시가 발생하지 않으며 가능한 다른 많은 오류 조건과 함께 이벤트 로그에 기록됩니다.

• UI - Chrome 브라우저를 사용하는 웹 클라이언트는 Statistics Manager 웹 서버와 동일한 작업 그룹 또는 도메인에 가입되지 않은 클라이언트 컴퓨터를 사용할 때 여러 로그인 프롬프트를 보게됩니다. 이제 세션 당 하나의 로그인 만 필요합니다.

Lync 2013 - Skype for Business Server 마이그레이션 Check List

 

After much searching you will find there isn’t much in the way of  guidance provided on TechNet when migrating from Lync 2013 to Skype for Business Server as there is when migrating from Lync 2010 to Lync 2013.  I was working on a Lync 2013 to Skype for Business Migration and wanted to make sure we didn’t forget anything when moving from a Lync 2013 Pool to a Skype for Business Pool. The following should help.  Essentially the process is the same so refer to https://technet.microsoft.com/en-us/library/jj205369%28v=ocs.15%29.aspx?f=255&MSPPError=-2147217396 for more details.