<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
<title></title>
<style type="text/css">
<!--
body{margin-left:10px;margin-right:10px;margin-top:10px;margin-bottom:10px;}
-->
</style>
</head>
<body marginleft="10" marginright="10" margintop="10" marginbottom="10">
<div align="left" style="text-align:left;"><font face="Calibri" size="+1" color="#000000" style="font-family:Calibri;font-size:14pt;color:#000000;">So many good replies!</font></div>
<br />
<div align="left" style="text-align:left;"><font face="Calibri" size="+1" color="#000000" style="font-family:Calibri;font-size:14pt;color:#000000;">I did very little today apart from look through log and config files - then it was working.</font></div>
<br />
<div align="left" style="text-align:left;"><font face="Calibri" size="+1" color="#000000" style="font-family:Calibri;font-size:14pt;color:#000000;">And yes! My thoughts exactly on the expiration! Why would the metadata file by offline for 2 weeks then suddenly back? Of course I don't know that these things are valid for 2 weeks - I only know the new on is. I don't know that the one I have with today's expiration date wasn't downloaded yesterday.</font></div>
<br />
<div align="left" style="text-align:left;"><font face="Calibri" size="+1" color="#000000" style="font-family:Calibri;font-size:14pt;color:#000000;">So I had the InCommon-metadata.xml file restored from last night's backup and sure enough, yet there it is: </font><font face="Courier New" size="+0" color="#000000" style="font-family:Courier New;font-size:12pt;color:#000000;">validUntil="</font><font face="Courier New" size="+0" color="#DD0000" style="font-family:Courier New;font-size:12pt;color:#DD0000;">2015-02-25T10:00:00Z</font><font face="Courier New" size="+0" color="#000000" style="font-family:Courier New;font-size:12pt;color:#000000;">"</font></div>
<div align="left" style="text-align:left;"><font face="Calibri" size="+1" color="#000000" style="font-family:Calibri;font-size:14pt;color:#000000;">And right on schedule - at 5:02am (10:02am GMT) - the first "entire metadata document has expired" message appears in the idp-process.log.</font></div>
<br />
<div align="left" style="text-align:left;"><font face="Calibri" size="+1" color="#000000" style="font-family:Calibri;font-size:14pt;color:#000000;">This morning while I was troubleshooting and saw the Windows server having a hard time downloading the InCommon-metadata.xml file, I tried it in curl for more insight into what was happening:</font></div>
<br />
<div align="left" style="text-align:left;"><font face="Calibri" size="+1" color="#000000" style="font-family:Calibri;font-size:14pt;color:#000000;">#####</font></div>
<div align="left" style="text-align:left;"><font face="Courier New" size="+0" color="#000000" style="font-family:Courier New;font-size:10pt;color:#000000;">08:58:12:/home/cburton> curl -v "http://wayf.incommonfederation.org/InCommon/InCommon-metadata.xml" | m</font></div>
<div align="left" style="text-align:left;"><font face="Courier New" size="+0" color="#000000" style="font-family:Courier New;font-size:10pt;color:#000000;">></font></div>
<div align="left" style="text-align:left;"><font face="Courier New" size="+0" color="#000000" style="font-family:Courier New;font-size:10pt;color:#000000;"> % Total % Received % Xferd Average Speed Time Time Time Current</font></div>
<div align="left" style="text-align:left;"><font face="Courier New" size="+0" color="#000000" style="font-family:Courier New;font-size:10pt;color:#000000;"> Dload Upload Total Spent Left Speed</font></div>
<div align="left" style="text-align:left;"><font face="Courier New" size="+0" color="#000000" style="font-family:Courier New;font-size:10pt;color:#000000;"> 0 0 0 0 0 0 0 0 --:--:-- 0:00:05 --:--:-- 0< </font><font face="Courier New" size="+0" color="#DD0000" style="font-family:Courier New;font-size:10pt;color:#DD0000;">HTTP/1.1 301 Moved Permanently</font></div>
<div align="left" style="text-align:left;"><font face="Courier New" size="+0" color="#000000" style="font-family:Courier New;font-size:10pt;color:#000000;">< Date: Wed, 25 Feb 2015 13:58:38 GMT</font></div>
<div align="left" style="text-align:left;"><font face="Courier New" size="+0" color="#000000" style="font-family:Courier New;font-size:10pt;color:#000000;">< Server: Apache</font></div>
<div align="left" style="text-align:left;"><font face="Courier New" size="+0" color="#000000" style="font-family:Courier New;font-size:10pt;color:#000000;">< </font><font face="Courier New" size="+0" color="#DD0000" style="font-family:Courier New;font-size:10pt;color:#DD0000;">Location: <a href="http://md.incommon.org/InCommon/InCommon-metadata-" target="_blank">http://md.incommon.org/InCommon/InCommon-metadata-</a></font><font face="Courier New" size="+0" color="#DD0000" style="font-family:Courier New;font-size:12pt;color:#DD0000;"><u><b>fallback</b></u></font><font face="Courier New" size="+0" color="#DD0000" style="font-family:Courier New;font-size:10pt;color:#DD0000;">.xml</font></div>
<div align="left" style="text-align:left;"><font face="Courier New" size="+0" color="#000000" style="font-family:Courier New;font-size:10pt;color:#000000;">< Content-Length: 347</font></div>
<div align="left" style="text-align:left;"><font face="Courier New" size="+0" color="#000000" style="font-family:Courier New;font-size:10pt;color:#000000;">< Connection: close</font></div>
<div align="left" style="text-align:left;"><font face="Courier New" size="+0" color="#000000" style="font-family:Courier New;font-size:10pt;color:#000000;">< Content-Type: text/html; charset=iso-8859-1</font></div>
<div align="left" style="text-align:left;"><font face="Courier New" size="+0" color="#000000" style="font-family:Courier New;font-size:10pt;color:#000000;"><</font></div>
<div align="left" style="text-align:left;"><font face="Courier New" size="+0" color="#000000" style="font-family:Courier New;font-size:10pt;color:#000000;">{ [data not shown]</font></div>
<div align="left" style="text-align:left;"><font face="Courier New" size="+0" color="#000000" style="font-family:Courier New;font-size:10pt;color:#000000;">115 347 115 347 0 0 68 0 0:00:05 0:00:05 --:--:-- 7229* Closing connection #0</font></div>
<br />
<div align="left" style="text-align:left;"><font face="Courier New" size="+0" color="#000000" style="font-family:Courier New;font-size:10pt;color:#000000;"><!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"></font></div>
<div align="left" style="text-align:left;"><font face="Courier New" size="+0" color="#000000" style="font-family:Courier New;font-size:10pt;color:#000000;"><html><head></font></div>
<div align="left" style="text-align:left;"><font face="Courier New" size="+0" color="#000000" style="font-family:Courier New;font-size:10pt;color:#000000;"><title>301 Moved Permanently</title></font></div>
<div align="left" style="text-align:left;"><font face="Courier New" size="+0" color="#000000" style="font-family:Courier New;font-size:10pt;color:#000000;"></head><body></font></div>
<div align="left" style="text-align:left;"><font face="Courier New" size="+0" color="#000000" style="font-family:Courier New;font-size:10pt;color:#000000;"><h1>Moved Permanently</h1></font></div>
<div align="left" style="text-align:left;"><font face="Courier New" size="+0" color="#000000" style="font-family:Courier New;font-size:10pt;color:#000000;"><p>The document has moved <a href="http://md.incommon.org/InCommon/InCommon-metadata-fallback.xml">here</a>.</p></font></div>
<div align="left" style="text-align:left;"><font face="Courier New" size="+0" color="#000000" style="font-family:Courier New;font-size:10pt;color:#000000;"><hr></font></div>
<div align="left" style="text-align:left;"><font face="Courier New" size="+0" color="#000000" style="font-family:Courier New;font-size:10pt;color:#000000;"><address>Apache Server at wayf.incommonfederation.org Port 80</address></font></div>
<div align="left" style="text-align:left;"><font face="Courier New" size="+0" color="#000000" style="font-family:Courier New;font-size:10pt;color:#000000;"></body></html></font></div>
<br />
<div align="left" style="text-align:left;"><font face="Calibri" size="+1" color="#000000" style="font-family:Calibri;font-size:14pt;color:#000000;">#####</font></div>
<br />
<div align="left" style="text-align:left;"><font face="Calibri" size="+1" color="#000000" style="font-family:Calibri;font-size:14pt;color:#000000;">There shouldn't have been any firewall changes; certainly none on the Shib server itself and changes that'd affect this would be extremely rare in the university firewall - and the timing for the process "fixing itself" sure would be improbable.</font></div>
<br />
<div align="left" style="text-align:left;"><font face="Calibri" size="+1" color="#000000" style="font-family:Calibri;font-size:14pt;color:#000000;">I'm not on that Shibboleth Notifications mailing list but hopefully I will be in a few minutes.</font></div>
<br />
<div align="left" style="text-align:left;"><font face="Calibri" size="+1" color="#000000" style="font-family:Calibri;font-size:14pt;color:#000000;">Also, I'll be looking into "logback" (this one, right? <a href="http://logback.qos.ch/" target="_blank">http://logback.qos.ch/</a> ). It's trying to be the successor for log4j, which I've seen used before, but I've never really dug into myself.</font></div>
<br />
<div align="left" style="text-align:left;"><font face="Calibri" size="+1" color="#000000" style="font-family:Calibri;font-size:14pt;color:#000000;">Thanks all for the responses,</font></div>
<div align="left" style="text-align:left;"><font face="Calibri" size="+1" color="#000000" style="font-family:Calibri;font-size:14pt;color:#000000;">--Clayton</font></div>
<br />
<br />
<br />
<div align="left" style="text-align:left;"><font face="Arial" size="+0" color="#000000" style="font-family:Arial;font-size:10pt;color:#000000;"><b>"Shib Users users-at-shibboleth.net |Shib|" <<a href="mailto:3unnche4it@sneakemail.com">3unnche4it@sneakemail.com</a>> on Wednesday, February 25, 2015 at 15:34 -0500 wrote:</b></font></div>
<div align="left" style="text-align:left;"><span style="background-color:#d0d0d0;"><font face="Times New Roman" size="+0" color="#000000" style="font-family:Times New Roman;font-size:12pt;color:#000000;">On Wed, Feb 25, 2015 at 2:22 PM, Clayton wrote:</font></span></div>
<div align="left" style="text-align:left;"><span style="background-color:#d0d0d0;"><font face="Times New Roman" size="+0" color="#000000" style="font-family:Times New Roman;font-size:12pt;color:#000000;">></font></span></div>
<div align="left" style="text-align:left;"><span style="background-color:#d0d0d0;"><font face="Times New Roman" size="+0" color="#000000" style="font-family:Times New Roman;font-size:12pt;color:#000000;">> This morning I got a call from our Help Desk telling me that there were Shib</font></span></div>
<div align="left" style="text-align:left;"><span style="background-color:#d0d0d0;"><font face="Times New Roman" size="+0" color="#000000" style="font-family:Times New Roman;font-size:12pt;color:#000000;">> SSO errors. Eventually I figured out that the problem was only with the SPs</font></span></div>
<div align="left" style="text-align:left;"><span style="background-color:#d0d0d0;"><font face="Times New Roman" size="+0" color="#000000" style="font-family:Times New Roman;font-size:12pt;color:#000000;">> which are part of the InCommon federation. About the time I was coming to</font></span></div>
<div align="left" style="text-align:left;"><span style="background-color:#d0d0d0;"><font face="Times New Roman" size="+0" color="#000000" style="font-family:Times New Roman;font-size:12pt;color:#000000;">> this conclusion and finding that I was having a difficult time downloading</font></span></div>
<div align="left" style="text-align:left;"><span style="background-color:#d0d0d0;"><font face="Times New Roman" size="+0" color="#000000" style="font-family:Times New Roman;font-size:12pt;color:#000000;">> the HTTP-file-backed metadata file from InCommon, the problem cleared up. I</font></span></div>
<div align="left" style="text-align:left;"><span style="background-color:#d0d0d0;"><font face="Times New Roman" size="+0" color="#000000" style="font-family:Times New Roman;font-size:12pt;color:#000000;">> think they fixed it on their end (by switching to a "fall back" server or</font></span></div>
<div align="left" style="text-align:left;"><span style="background-color:#d0d0d0;"><font face="Times New Roman" size="+0" color="#000000" style="font-family:Times New Roman;font-size:12pt;color:#000000;">> file).</font></span></div>
<div align="left" style="text-align:left;"><span style="background-color:#d0d0d0;"><font face="Times New Roman" size="+0" color="#000000" style="font-family:Times New Roman;font-size:12pt;color:#000000;">That's probably the wrong conclusion but let's hold that thought as we</font></span></div>
<div align="left" style="text-align:left;"><span style="background-color:#d0d0d0;"><font face="Times New Roman" size="+0" color="#000000" style="font-family:Times New Roman;font-size:12pt;color:#000000;">try to get to the bottom of this.</font></span></div>
<div align="left" style="text-align:left;"><span style="background-color:#d0d0d0;"><font face="Times New Roman" size="+0" color="#000000" style="font-family:Times New Roman;font-size:12pt;color:#000000;">If you manage an IdP in the InCommon Federation but you're NOT a</font></span></div>
<div align="left" style="text-align:left;"><span style="background-color:#d0d0d0;"><font face="Times New Roman" size="+0" color="#000000" style="font-family:Times New Roman;font-size:12pt;color:#000000;">Federation Site Administrator, you probably want to subscribe to the</font></span></div>
<div align="left" style="text-align:left;"><span style="background-color:#d0d0d0;"><font face="Times New Roman" size="+0" color="#000000" style="font-family:Times New Roman;font-size:12pt;color:#000000;">inc-ops-notifications mailing list:</font></span></div>
<div align="left" style="text-align:left;"><span style="background-color:#d0d0d0;"><font face="Times New Roman" size="+0" color="#000000" style="font-family:Times New Roman;font-size:12pt;color:#000000;"><a href="https://lists.incommon.org/sympa/info/inc-ops-notifications" target="_blank">https://lists.incommon.org/sympa/info/inc-ops-notifications</a></font></span></div>
<div align="left" style="text-align:left;"><span style="background-color:#d0d0d0;"><font face="Times New Roman" size="+0" color="#000000" style="font-family:Times New Roman;font-size:12pt;color:#000000;">All Federation Site Administrators are automatically subscribed to</font></span></div>
<div align="left" style="text-align:left;"><span style="background-color:#d0d0d0;"><font face="Times New Roman" size="+0" color="#000000" style="font-family:Times New Roman;font-size:12pt;color:#000000;">this list but it's an open subscription list so....</font></span></div>
<div align="left" style="text-align:left;"><span style="background-color:#d0d0d0;"><font face="Times New Roman" size="+0" color="#000000" style="font-family:Times New Roman;font-size:12pt;color:#000000;">> Specifically we started seeing symptoms b/c of this first line in the</font></span></div>
<div align="left" style="text-align:left;"><span style="background-color:#d0d0d0;"><font face="Times New Roman" size="+0" color="#000000" style="font-family:Times New Roman;font-size:12pt;color:#000000;">> InCommon metadata file:</font></span></div>
<div align="left" style="text-align:left;"><span style="background-color:#d0d0d0;"><font face="Times New Roman" size="+0" color="#000000" style="font-family:Times New Roman;font-size:12pt;color:#000000;">></font></span></div>
<div align="left" style="text-align:left;"><span style="background-color:#d0d0d0;"><font face="Times New Roman" size="+0" color="#000000" style="font-family:Times New Roman;font-size:12pt;color:#000000;">> </font></span></div>
<div align="left" style="text-align:left;"><span style="background-color:#d0d0d0;"><font face="Times New Roman" size="+0" color="#000000" style="font-family:Times New Roman;font-size:12pt;color:#000000;">That implies your metadata refresh process has been broken for two</font></span></div>
<div align="left" style="text-align:left;"><span style="background-color:#d0d0d0;"><font face="Times New Roman" size="+0" color="#000000" style="font-family:Times New Roman;font-size:12pt;color:#000000;">weeks since the InCommon metadata file has a two-week expiration</font></span></div>
<div align="left" style="text-align:left;"><span style="background-color:#d0d0d0;"><font face="Times New Roman" size="+0" color="#000000" style="font-family:Times New Roman;font-size:12pt;color:#000000;">window.</font></span></div>
<div align="left" style="text-align:left;"><span style="background-color:#d0d0d0;"><font face="Times New Roman" size="+0" color="#000000" style="font-family:Times New Roman;font-size:12pt;color:#000000;">> The error has been in the log file for a while:</font></span></div>
<div align="left" style="text-align:left;"><span style="background-color:#d0d0d0;"><font face="Times New Roman" size="+0" color="#000000" style="font-family:Times New Roman;font-size:12pt;color:#000000;">Two weeks, right?</font></span></div>
<div align="left" style="text-align:left;"><span style="background-color:#d0d0d0;"><font face="Times New Roman" size="+0" color="#000000" style="font-family:Times New Roman;font-size:12pt;color:#000000;">> Error</font></span></div>
<div align="left" style="text-align:left;"><span style="background-color:#d0d0d0;"><font face="Times New Roman" size="+0" color="#000000" style="font-family:Times New Roman;font-size:12pt;color:#000000;">> [org.Opensaml.Saml2.Metadata.Provider.Httpmetadataprovider:262] - error</font></span></div>
<div align="left" style="text-align:left;"><span style="background-color:#d0d0d0;"><font face="Times New Roman" size="+0" color="#000000" style="font-family:Times New Roman;font-size:12pt;color:#000000;">> retrieving metadata from ...</font></span></div>
<div align="left" style="text-align:left;"><span style="background-color:#d0d0d0;"><font face="Times New Roman" size="+0" color="#000000" style="font-family:Times New Roman;font-size:12pt;color:#000000;">>From what location? Exactly what location are you requesting?</font></span></div>
<div align="left" style="text-align:left;"><span style="background-color:#d0d0d0;"><font face="Times New Roman" size="+0" color="#000000" style="font-family:Times New Roman;font-size:12pt;color:#000000;">My guess is that your IdP is requesting metadata at the following location:</font></span></div>
<div align="left" style="text-align:left;"><span style="background-color:#d0d0d0;"><font face="Times New Roman" size="+0" color="#000000" style="font-family:Times New Roman;font-size:12pt;color:#000000;"><a href="http://md.incommon.org/InCommon/InCommon-metadata.xml" target="_blank">http://md.incommon.org/InCommon/InCommon-metadata.xml</a></font></span></div>
<div align="left" style="text-align:left;"><span style="background-color:#d0d0d0;"><font face="Times New Roman" size="+0" color="#000000" style="font-family:Times New Roman;font-size:12pt;color:#000000;">which is what it should be doing but exactly two weeks ago we</font></span></div>
<div align="left" style="text-align:left;"><span style="background-color:#d0d0d0;"><font face="Times New Roman" size="+0" color="#000000" style="font-family:Times New Roman;font-size:12pt;color:#000000;">introduced the MD-RPI schema into production metadata and I'll bet</font></span></div>
<div align="left" style="text-align:left;"><span style="background-color:#d0d0d0;"><font face="Times New Roman" size="+0" color="#000000" style="font-family:Times New Roman;font-size:12pt;color:#000000;">that broke your metadata refresh process somehow.. Something you did</font></span></div>
<div align="left" style="text-align:left;"><span style="background-color:#d0d0d0;"><font face="Times New Roman" size="+0" color="#000000" style="font-family:Times New Roman;font-size:12pt;color:#000000;">today must have brought it back to life.</font></span></div>
<div align="left" style="text-align:left;"><span style="background-color:#d0d0d0;"><font face="Times New Roman" size="+0" color="#000000" style="font-family:Times New Roman;font-size:12pt;color:#000000;">> Is there anything I can do to be alerted to this kind of thing?</font></span></div>
<div align="left" style="text-align:left;"><span style="background-color:#d0d0d0;"><font face="Times New Roman" size="+0" color="#000000" style="font-family:Times New Roman;font-size:12pt;color:#000000;">> I suppose I could have a script grep through the log file every 5 minutes</font></span></div>
<div align="left" style="text-align:left;"><span style="background-color:#d0d0d0;"><font face="Times New Roman" size="+0" color="#000000" style="font-family:Times New Roman;font-size:12pt;color:#000000;">> for new errors and send me an email when they're found.</font></span></div>
<div align="left" style="text-align:left;"><span style="background-color:#d0d0d0;"><font face="Times New Roman" size="+0" color="#000000" style="font-family:Times New Roman;font-size:12pt;color:#000000;">> But is there something more elegant?</font></span></div>
<div align="left" style="text-align:left;"><span style="background-color:#d0d0d0;"><font face="Times New Roman" size="+0" color="#000000" style="font-family:Times New Roman;font-size:12pt;color:#000000;">That's a Shibboleth question that I can't answer, maybe someone else</font></span></div>
<div align="left" style="text-align:left;"><span style="background-color:#d0d0d0;"><font face="Times New Roman" size="+0" color="#000000" style="font-family:Times New Roman;font-size:12pt;color:#000000;">has a suggestion?</font></span></div>
<div align="left" style="text-align:left;"><span style="background-color:#d0d0d0;"><font face="Times New Roman" size="+0" color="#000000" style="font-family:Times New Roman;font-size:12pt;color:#000000;">Tom</font></span></div>
<div align="left" style="text-align:left;"><span style="background-color:#d0d0d0;"><font face="Times New Roman" size="+0" color="#000000" style="font-family:Times New Roman;font-size:12pt;color:#000000;">-- </font></span></div>
<br />
<br/>
</body>
</html>