InCommon metadata problem now fixed - how to detect in the future

Clayton upnyhgb8v6 at snkmail.com
Wed Feb 25 16:49:40 EST 2015


So many good replies!

I did very little today apart from look through log and config files -
then it was working.

And yes!  My thoughts exactly on the expiration!  Why would the metadata
file by offline for 2 weeks then suddenly back? Of course I don't know
that these things are valid for 2 weeks - I only know the new on is.  I
don't know that the one I have with today's expiration date wasn't
downloaded yesterday.

So I had the InCommon-metadata.xml file restored from last night's backup
and sure enough, yet there it is:  validUntil="2015-02-25T10:00:00Z"
And right on schedule - at 5:02am (10:02am GMT) - the first "entire
metadata document has expired" message appears in the idp-process.log.

This morning while I was troubleshooting and saw the Windows server having
a hard time downloading the InCommon-metadata.xml file, I tried it in curl
for more insight into what was happening:

#####
08:58:12:/home/cburton> curl -v
"http://wayf.incommonfederation.org/InCommon/InCommon-metadata.xml" | m
>
  % Total    % Received % Xferd  Average Speed   Time    Time     Time 
Current
                                 Dload  Upload   Total   Spent    Left 
Speed
  0     0    0     0    0     0      0      0 --:--:--  0:00:05 --:--:--  
  0< HTTP/1.1 301 Moved Permanently
< Date: Wed, 25 Feb 2015 13:58:38 GMT
< Server: Apache
< Location: http://md.incommon.org/InCommon/InCommon-metadata-fallback.xml
< Content-Length: 347
< Connection: close
< Content-Type: text/html; charset=iso-8859-1
<
{ [data not shown]
115   347  115   347    0     0     68      0  0:00:05  0:00:05 --:--:-- 
7229* Closing connection #0

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>301 Moved Permanently</title>
</head><body>
<h1>Moved Permanently</h1>
<p>The document has moved <a
href="http://md.incommon.org/InCommon/InCommon-metadata-fallback.xml">here</a>.</p>
<hr>
<address>Apache Server at wayf.incommonfederation.org Port 80</address>
</body></html>

#####

There shouldn't have been any firewall changes; certainly none on the Shib
server itself and changes that'd affect this would be extremely rare in
the university firewall - and the timing for the process "fixing itself"
sure would be improbable.

I'm not on that Shibboleth Notifications mailing list but hopefully I will
be in a few minutes.

Also, I'll be looking into "logback" (this one, right? 
http://logback.qos.ch/ ).  It's trying to be the successor for log4j,
which I've seen used before, but I've never really dug into myself.

Thanks all for the responses,
--Clayton



"Shib Users users-at-shibboleth.net |Shib|" <3unnche4it at sneakemail.com> on
Wednesday, February 25, 2015 at 15:34 -0500 wrote:
>On Wed, Feb 25, 2015 at 2:22 PM, Clayton  wrote:
>>
>> This morning I got a call from our Help Desk telling me that there were
>Shib
>> SSO errors. Eventually I figured out that the problem was only with the
>SPs
>> which are part of the InCommon federation. About the time I was coming
>to
>> this conclusion and finding that I was having a difficult time
>downloading
>> the HTTP-file-backed metadata file from InCommon, the problem cleared
>up. I
>> think they fixed it on their end (by switching to a "fall back" server
>or
>> file).
>That's probably the wrong conclusion but let's hold that thought as we
>try to get to the bottom of this.
>If you manage an IdP in the InCommon Federation but you're NOT a
>Federation Site Administrator, you probably want to subscribe to the
>inc-ops-notifications mailing list:
>https://lists.incommon.org/sympa/info/inc-ops-notifications
>All Federation Site Administrators are automatically subscribed to
>this list but it's an open subscription list so...
>> Specifically we started seeing symptoms b/c of this first line in the
>> InCommon metadata file:
>>
>> 
>That implies your metadata refresh process has been broken for two
>weeks since the InCommon metadata file has a two-week expiration
>window.
>> The error has been in the log file for a while:
>Two weeks, right?
>> Error
>> [org.Opensaml.Saml2.Metadata.Provider.Httpmetadataprovider:262] - error
>> retrieving metadata from ...
>>From what location? Exactly what location are you requesting?
>My guess is that your IdP is requesting metadata at the following
>location:
>http://md.incommon.org/InCommon/InCommon-metadata.xml
>which is what it should be doing but exactly two weeks ago we
>introduced the MD-RPI schema into production metadata and I'll bet
>that broke your metadata refresh process somehow. Something you did
>today must have brought it back to life.
>> Is there anything I can do to be alerted to this kind of thing?
>> I suppose I could have a script grep through the log file every 5
>minutes
>> for new errors and send me an email when they're found.
>> But is there something more elegant?
>That's a Shibboleth question that I can't answer, maybe someone else
>has a suggestion?
>Tom
>-- 


-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://shibboleth.net/pipermail/users/attachments/20150225/539efeea/attachment-0001.html 


More information about the users mailing list