Configuring a file-backed HTTP resource for attribute filters

[Wessel, Keith]

Well, now, this generated quite a bit of discussion. That'll teach me to choose my time more carefully for when I go to lunch.

Where shall we start? First, this file cache you referred to. That's an in-memory cache? Something that gets cleaned out by a restart? 'Cause this error occurs on jetty/IDP start-up. So, if it's in-memory, I should be starting with a clean cache and without the local backin file existing. BTW, when I do suck downthe file manually before starting the IDP, the warnings go away and the IDP happily accepts the 304 return code.

Shall I try turning up logging as Brent suggested and see what that tells me? And perhaps, if there's nothing obvious in the output, put that output in Gira?

Keith


From: users [mailto:users-bounces at shibboleth.net] On Behalf Of Brent Putman
Sent: Tuesday, July 21, 2015 12:09 PM
To: users at shibboleth.net
Subject: Re: Configuring a file-backed HTTP resource for attribute filters


On 7/21/15 12:51 PM, Cantor, Scott wrote:





Ok, admittedly I interpreted "no backup file yet" as equivalent to the file cache not being populated either, which aren't necessarily equivalent. So that is the first question to ask, and one can always flush that directory.

Looking at the code, the backup file referenced is definitely ours.  The root issue happens before that, something is going wrong with the FileBacked- class call to super.getInputStream() (the regular HttpResource).

What would be very informative is to see:

1) the DEBUG output from net.shibboleth.ext.spring.resource.HTTPResource.  That will report what the caching status of the file as reported by HttpClient.

2) the ERROR level output reporting exactly what HTTP status code was effectively returned from the client call.  My previous testing and understanding what that if the HttpClient served the response from cache, then you get back a 200, but perhaps that's not always the case.  The superclass getInputStream() has the code block:

        if (httpStatusCode != HttpStatus.SC_OK) {
            String errMsg =
                    "Non-ok status code " + httpStatusCode + " returned from remote resource " + resourceURL;
            log.error(errMsg);
            throw new IOException(errMsg);
        }

I'd like to confirm that that is what's happening to throw the IOException caught by the FileBacked- subclass, and what the exact code is.  Perhaps we are misusing the HttpClient there, i.e. maybe codes other than SC_OK should be treated as success.  Just guessing. Or maybe there really is a bug in HttpClient somewhere.

Also interesting would be DEBUG output from org.apache.http.  (Warning: that will be a lot, since org.apache.http.wire does a wire level dump of all the traffic).

Another thing to look at might be the max cache entry size:  We have for the FileCachingHttpClient default of:

p:maxCacheEntrySize="%{idp.httpclient.filecaching.maxCacheEntrySize:10485760}

so 10MB.  Could the resource being fetched here be bigger than that? I have thought that if the resource is bigger, it simply isn't cached, but perhaps it's more complicated and/or there is a bug around that (ours or theirs).

Finally, might be informative to see if instead switching to the shibboleth.MemoryCachingHttpClient resolves the issue.  Might hint where the problem is.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://shibboleth.net/pipermail/users/attachments/20150721/a9d17c13/attachment-0001.html>