Hello all,
we seem to have multiple reports of Skype for Business online sign-in issues.
From an affected Customer log we see:
1217114726|curl |1|00|HEADER_IN : report-to: {"group":"network-errors","max_age":86400,"endpoints":[{"url":"https://identity.nelreports.net/api/report?catId=GW+estsfd+dub2"}]}
1217114726|curl |1|00|HEADER_IN : nel: {"report_to":"network-errors","max_age":86400,"success_fraction":0.001,"failure_fraction":1.0}
1217114726|curl |1|00|HEADER_IN : Set-Cookie: fpc=Ag6hGzYAUaZIjGHUbIAugW4; expires=Sat, 16-Jan-2021 11:47:27 GMT; path=/; secure; HttpOnly; SameSite=None
1217114726|curl |1|00|HEADER_IN : Set-Cookie: x-ms-gateway-slice=estsfd; path=/; secure; httponly
1217114726|curl |1|00|HEADER_IN : Set-Cookie: stsservicecookie=estsfd; path=/; secure; httponly
1217114726|curl |1|00|HEADER_IN : Content-Disposition: inline; filename=userrealm.json
1217114726|curl |1|00|HEADER_IN : Date: Thu, 17 Dec 2020 11:47:26 GMT
1217114726|curl |1|00|HEADER_IN : Content-Length: 175
Checking the URL https://identity.nelreports.net/api/report?catId=GW+estsfd+dub2 we get:
To verify your issue is the same:
Settings > Logging > Global Settings > Global Log Level Limit > DebugSettings > Logging > Global Settings > Global Log Level Limit > Log to Standard Output > EnableSettings > Logging > Global Settings > Global Log Level Limit > Log File Size (Kbytes) > Trio 8300 & VVX after 5.5.0 = 1000Settings > Logging > Global Settings > Global Log Level Limit > Log File Size (Kbytes) > Trio or CCX 10240
Settings > Logging > Module Log Level Limits > CURL > Event 1
Settings > Logging > Module Log Level Limits > SIP > Debug
We will update this once we know more
Best Regards
Steffen Baier
If official support is required please check how to phone or open a case here
----------------Thanks for the update, this morning everyone started emailing that they can't seem to login to the phones.
VVX 401 phones are not registering, authentication failed.
Please update ASAP, or if we need to perform a firmware upgrade.
Thank you
Hello @plateplus
at present, it does not look to be a Poly issue and we are working with our Partner Microsoft on this.
Its a global issue affecting certain tenants. The Microsoft Ticket reference is 23363356 in case you want to open your own ticket.
Have a look at this post:
Jun 05, 2018 Question: Do Poly Phones support a REST API?
Resolution: Please => here <=
The above has some examples on how to use the REST API in case you need phones to reboot so they can sign in again once Microsoft has identified the root cause.
Best Regards
Steffen Baier
If official support is required please check how to phone or open a case here
----------------Hello all,
our Partner Microsoft is still investigating the root cause on this. It is now stated in the Office 365 Admin Center
As a workaround, once the fault is fixed, we assume phones have to be restarted.
Workaround 1:
Prerequisite the phones must have the REST API already enabled. This covers users on Poly RealPresence Resource Manager or PDMS-E
Get a list of all phones IP addresses from the DHCP server as an example and store them in a text file called ip.txt
Copy the below and ensure to change the password 789 to what the actual admin password of the phone is.
@echo off
echo Reading ip.txt
for /F "tokens=1" %i in (ip.txt) do (
curl -i -X POST -H "Content-Type: application/json" -d "" -k https://Polycom:789@%i/api/v1/mgmt/safeRestart
timeout /t 5 /NOBREAK
)
Store the above as reboot.bat and store it in the same folder as ip.txt. Open a MS-Dos command line and change to the directory where both files have been stored. run reboot.bat
(It only works in newer versions of Windows 10 that have CURL)
Workaround 2:
Below is a PERL script which can be used in a local network. The default username is Polycom and the Password can be changed from 789:
#!/usr/bin/perl -w
# RebootScript.pl - Reboot range of Polycom phones using the Web Interface
# Initial Script (???) - bm
# Updated (???) - sb
# Update (28 April 2017) - mcab
# - Set the admin username and password at the top of the file. (Script will generate the cookie automatically now)
# - Removed dependency on CGI.pm
# Update (28 April 2017 #2) - mcab
# - add IO::Socket::SSL
# - ignore validating TLS certs from phone
use strict;
use LWP::UserAgent;
use IO::Socket;
use Socket;
use IO::Socket::SSL;
use MIME::Base64;
my $username = "Polycom";
my $password = "789";
#####################################
# Netmask 255.255.255.0 = 0xFFFFFF00#
# Netmask 255.255.0.0 = 0xFFFF0000#
# Netmask 255.0.0.0 = 0xFF000000#
# Netmask 0.0.0.0 = 0x00000000#
#####################################
my $netmask = 0xFFFFFF00; # 255.255.255.0
#########################################
# Define IP Address Start Range $Start #
# Define IP Address End Range $finish #
#########################################
my $start_address = unpack 'N', inet_aton( '10.252.149.57' );
my $finish_address = unpack 'N', inet_aton( '10.252.149.58' );
for ( my $address = $start_address; $address <= $finish_address; ++$address ) {
next if ( $address & $netmask ) == $address or ( $address & ~$netmask ) == ~$netmask;
my $ua = LWP::UserAgent->new( );
$ua->ssl_opts( verify_hostname => 0, SSL_verify_mode => SSL_VERIFY_NONE );
my $destination = inet_ntoa( pack 'N', $address);
#############################
# Define Reboot or Restart #
# Simply remove # and add # #
#############################
my $restartUrl = "https://$destination/form-submit/Reboot";
# Create request object and add the authentication header and the expected cookie
my $req = HTTP::Request->new(POST => $restartUrl);
$req->header('Cookie' => 'Authorization=Basic ' . encode_base64("$username:$password"));
$req->authorization_basic( $username, $password );
# Send the reset request to the user agent
my $res = $ua->request( $req );
print $res->status_line, "\n";
sleep (5)
}
Store this on a PC and run it (needs Perl installed)
This simple and basic Perl script can enable the Reboot of a range of Phones
How does it work?
2 Actions are required within the Script itself:
- Define the Password ($Password)
my $password = "789";
- Define the IP Address Range (aka start / stop)
Example: 10.252.149.57 till 10.252.149.58
my $start_address = unpack 'N', inet_aton( '10.252.149.57' );
my $finish_address = unpack 'N', inet_aton( '10.252.149.58' );
How do I run it?
- Download and install ActivePerl for Windows PC'script
- Download RebootScript.pl.txt and store as RebootScript.pl into the C:\Perl64\bin directory
- Open a CMD window and browse to C:\Perl64\bin and Edit to suit the above actions
- perl RebootScript.pl and hit enter. It will not send any warnings to anyone and wait 5 Seconds between Phone IP
If official support is required please check how to phone or open a case here
----------------Hello all,
this seems to be resolved via a change on our Partner Microsofts end. Please work with Microsoft for any follow-up questions.
Best regards
Steffen Baier
If official support is required please check how to phone or open a case here
----------------The official explanation from the O365 Admin Centre (LY230423):
Title: Users can't sign into Poly devices
User Impact: Users may have been unable to sign into Poly devices.
More info: This was affecting Skype for Business and Teams-only Azure Active Directory (AAD) federated users logging into Poly devices.
Final status: We've rolled back the recently deployed update and confirmed with affected users that impact was remediated.
Scope of impact: Impact was specific to users signing into Poly devices.
Start time: Thursday, December 17, 2020, 12:59 AM (12:59 AM UTC)
End time: Thursday, December 17, 2020, 6:00 PM (6:00 PM UTC)
Root cause: A recently deployed update implemented telemetry enhancements for AAD that resulted in users being unable to sign into Poly devices.
Next steps:
- We're reviewing our update procedures to better identify similar Poly device authentication issues during our development and testing cycles.
This is the final update for the event.
We are working with our Partner Microsoft so any change like this in the future should not affect the end-user.
If official support is required please check how to phone or open a case here
----------------