<!DOCTYPE html>
<html>
<head>
<meta http-equiv="Content-Type" content="text/xhtml; charset=utf-8">
<style>
div.plaintext { white-space: normal; }
body { font-family: sans-serif; }
h1 { font-size: 1.4em; }
h2 { font-size: 1.2em; }
h3 { font-size: 1.1em; }
blockquote { margin: 0 0 5px; padding-left: 5px; border-left: 2px solid #777777; color: #777777; }
blockquote blockquote { border-left-color: #999999; color: #999999; }
blockquote blockquote blockquote { border-left-color: #BBBBBB; color: #BBBBBB; }
a { color: #3983C4; }
blockquote a { color: #777777; }
blockquote blockquote a { color: #999999; }
blockquote blockquote blockquote a { color: #BBBBBB; }
math[display="inline"] > mrow { padding:5px; }
div.footnotes li p { margin: 0.2em 0; }
</style>
</head>
<body>
<div class="plaintext"><p dir="auto">FYI 2</p>
<p dir="auto">Forwarded message:</p>
<blockquote><p dir="auto">From: Nicole Roy <nroy@internet2.edu><br>
To: inc-ops-notifications@incommon.org<br>
Subject: Re: [InCommon NOTICE] InCommon MDQ metadata retrieval issues for some clients<br>
Date: Fri, 08 Oct 2021 14:04:39 -0600<br>
<br>
We are seeing signs that some of the CloudFront edge locations that were formerly failing may be starting to return correct MDQ results now. We have not received any confirmation from AWS that this is the case, but we are cautiously optimistic. I’ve updated the outage wiki page, linked below. We will communicate more as we receive more information from AWS.<br>
<br>
Thank you,<br>
<br>
Nicole<br>
<br>
On 8 Oct 2021, at 12:23, Nicole Roy wrote:<br>
</p>
<blockquote><p dir="auto">Hello,<br>
<br>
We continue to pursue this issue with AWS engineers. They have identified a problem with encoding certain characters from a request, which causes them not to be able to correctly resolve an entity descriptor within the S3 origin location that backs our MDQ service. They are working on a fix. As of this update, we do not have an ETA from them, but they assure us that this has the highest internal priority within AWS.<br>
<br>
Updates will continue to be posted at: <a href="https://spaces.at.internet2.edu/display/federationops/2021-10-07+MDQ+intermittent+outage">https://spaces.at.internet2.edu/display/federationops/2021-10-07+MDQ+intermittent+outage</a><br>
<br>
Best Regards,<br>
<br>
Nicole<br>
<br>
On 7 Oct 2021, at 14:34, Nicole Roy wrote:<br>
</p>
<blockquote><p dir="auto">Another update, we have identified a method of temporarily pinning MDQ requests to a specific, currently-known-good CloudFront edge location. Details at: <a href="https://spaces.at.internet2.edu/display/federationops/2021-10-07+MDQ+intermittent+outage">https://spaces.at.internet2.edu/display/federationops/2021-10-07+MDQ+intermittent+outage</a><br>
<br>
Deployers are advised to test this approach locally, and ensure that whichever CloudFront edge location they pick, actually works before updating hosts files on their affected systems.<br>
<br>
Best,<br>
<br>
Nicole<br>
<br>
On 7 Oct 2021, at 12:35, Nicole Roy wrote:<br>
</p>
<blockquote><p dir="auto">All future status updates will be posted in the Internet2 wiki, at: <a href="https://spaces.at.internet2.edu/display/federationops/2021-10-07+MDQ+intermittent+outage">https://spaces.at.internet2.edu/display/federationops/2021-10-07+MDQ+intermittent+outage</a><br>
<br>
As of 12:30 MDT, AWS support is able to reproduce this issue and is escalating it within AWS.<br>
<br>
Best,<br>
<br>
Nicole<br>
<br>
On 7 Oct 2021, at 10:06, Nicole Roy wrote:<br>
</p>
<blockquote><p dir="auto">Update 2020-10-07 10:00 MDT<br>
<br>
We have opened a support case with Amazon Web Services with regard to the edge cache locations which appear to be failing:<br>
<br>
13.32.208.20<br>
13.32.208.84<br>
13.32.208.58<br>
13.32.208.38<br>
<br>
On 7 Oct 2021, at 9:31, Nicole Roy wrote:<br>
</p>
<blockquote><p dir="auto">As of 9:15 a.m. MDT on October 7, we are able to reproduce the issue and it appears to be isolated to a number of CloudFront servers. We are working to further identify the problem with these distribution points.<br>
<br>
Best Regards,<br>
<br>
Nicole<br>
<br>
On 7 Oct 2021, at 9:06, Nicole Roy wrote:<br>
</p>
<blockquote><p dir="auto">Hello,<br>
<br>
This morning, InCommon operations was notified of some intermittent issues by some metadata clients, in retrieving SAML metadata from <a href="https://mdq.incommon.org">https://mdq.incommon.org</a>. We are actively investigating this issue and will share more information when we know more. In the meantime, if you are experiencing issues with the MDQ service, please send email to help@incommon.org, including (if possible) detailed information about the client you’re using (SAML software and version), IP address of the client, physical location (approximate), log data if share-able, etc.<br>
<br>
Best Regards,<br>
<br>
Nicole Roy<br>
Pronouns: She/her/hers<br>
Director of Technology and Strategy<br>
InCommon / Internet2 Trust and Identity Services</p>
</blockquote></blockquote></blockquote></blockquote></blockquote></blockquote></blockquote></div>
</body>
</html>