IDPv3 / ECP / O365 / ActiveSync failure

[Dan Malone]

Some of this just doesn't make any sense yet, so I'm throwing it out here to see if something similar has happened...

Yesterday we deployed our IDPv3. All new VMs, so the cut-over was a simple DNS change.
Since we had the new environment up and running, we were able to test everything except ECP using the hosts file trick.
7am on Sunday happened, and everything was happy.
We tested our ECP clients, Thunderbird, iPhone native, Android native, etc., and they were happy too.
While we had some O365 servers that didn't honor the 300s DNS TTL, I was able to verify via logs that our testing was hitting the IDPv3 servers.
By 9am, all the old servers were doing was metadata transfers, so we cheered and went about our Sunday.

About 6:30pm, the president calls the CIO saying he can't get email on his iPhone.

Troubleshooting results in the following:
Logs do not show any errors and are actually showing successful release to MicroSoft via ECP:

    2016-11-13 19:23:30,615 - INFO [Shibboleth-Audit.SSO:241] - 20161114T032330Z|urn:oasis:names:tc:SAML:2.0:bindings:SOAP|_639a0cd2-8808-4a25-a47c-cfb0ef976ce9|urn:federation:MicrosoftOnline|http://shibboleth.net/ns/profiles/saml2/sso/ecp|https://idp.calpoly.edu/idp/shibboleth|urn:oasis:names:tc:SAML:2.0:bindings:PAOS|_b7e64d804125
    c75c0b840a1a52157d25|dmalone|urn:oasis:names:tc:SAML:2.0:ac:classes:PasswordProtectedTransport|MicrosoftOnline:UserID,MicrosoftOnline:ImmutableID|AAhzZWNyZXQ2OSmDFz3SzupZqiMAq73hpdHndV93NlKUDRYME+NHAuNM2oXWmTAOblcDqwpCEMuWBeBpnnj4pQ1JSwYaiRTd7htrvzh1ZycJZi5oiziPWU8JfdvrmH/cHktq6R1nIBRAXTxAOi4=|_48b271d2302cf78f502c817cfe4a1598|

Further testing showed that Thunderbird and other IMAP clients are working without issue.
The clients with problems were all setup using ActiveSync.

Running the test at https://testconnectivity.microsoft.com/ , showed we were failing the Autodiscover and Exchange ActiveSync test.

Knowing the first question we would ask is "did anything change?", we made the decision to fallback to IDPv2.
The servers were still up and running, so it was a simple DNS change.

Again it took some time for the O365 servers to update their DNS, but slowly they did, and like magic, iPhones started working...

Running the https://testconnectivity.microsoft.com/  again and we are now passing the Autodiscover test, showing successful connection to autodiscover.calpoly.edu which is a CNAME to autodiscover.outlook.com.

This name leads us to: https://autodiscover-s.outlook.com/Autodiscover/Autodiscover.xml
Which prompts for authentication that is verified against our IDP via ECP.

Everything I saw tells me it was just coincidental that things started working when we rolled back.
Now I need to convince management and MicroSoft.
So, has anyone seen anything similar?

Thanks,
Dan
-- 
Dan Malone
Lead Identity Management Architect
Information Technology Services
California Polytechnic State University
San Luis Obispo, California

Direct 805-756-6326
dmalone at calpoly.edu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://shibboleth.net/pipermail/users/attachments/20161114/af7c9136/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4207 bytes
Desc: S/MIME Cryptographic Signature
URL: <http://shibboleth.net/pipermail/users/attachments/20161114/af7c9136/attachment.p7s>