Performing conditional gets with the metadata aggregator

[Wessel, Keith]

Tom,

After letting my new process run overnight, I discovered another small issue: cget.sh doesn't clean up after itself. As a result, I had a couple GB of copies of the InCommon metadata file in tmpdir/cget/http_resource_*.xml. The conditional_get_* files, though only one of each, were also left behind.

I added an rm of the temp file to cget.sh right after it's cat'd near the end of the script as a uick fix, but I suppose it could be argued that tmpdir (default /tmp/cget) could be removed completely at the end of the run instead of just its contents. At the least, the conditional_get_* files probably need to be removed along with the http_resource_* file.

Keith


-----Original Message-----
From: Wessel, Keith 
Sent: Thursday, April 06, 2017 1:22 PM
To: Shib Users <users at shibboleth.net>
Subject: RE: Performing conditional gets with the metadata aggregator

Ah, yes. Now that I look at those lines more closely, I see they're using $TMPDIR. Sorry, I missed that.

So, yes, an update to the documentation would be all that's needed.

Thanks very much for pointing me to this code and to I2 for creating this useful tool. I'm off to implement it!

Keith


-----Original Message-----
From: users [mailto:users-bounces at shibboleth.net] On Behalf Of Tom Scavo
Sent: Thursday, April 06, 2017 12:51 PM
To: Shib Users <users at shibboleth.net>
Subject: Re: Performing conditional gets with the metadata aggregator

On Thu, Apr 6, 2017 at 1:04 PM, Wessel, Keith <kwessel at illinois.edu> wrote:
> Finally getting back to this, and Tom, the bash library you mentioned looks like it'll do exactly what we need. Thank you!
>
> One question: it looks like there are some hard-coded refrences to /tmp in ilb/md_tools.sh on lines 348, 469, and 584. If you're like us and have a very tiny /tmp and are trying to fetch in the InCommon metadata in full, this quickly runs out of disk space.

No, there aren't any hardcoded references to /tmp in the code but
there are references to a semi-standard environment variable called
TMPDIR. Not all Linux flavors provide that environment variable so the
calling script must check for that and create a temporary directory if
one doesn't already exist.

I'll document this inline and commit an update later today.

Thanks Keith.

Tom

> -----Original Message-----
> From: users [mailto:users-bounces at shibboleth.net] On Behalf Of Tom Scavo
> Sent: Tuesday, March 21, 2017 9:31 AM
> To: Shib Users <users at shibboleth.net>
> Subject: Re: Performing conditional gets with the metadata aggregator
>
> On Tue, Mar 21, 2017 at 9:58 AM, Wessel, Keith <kwessel at illinois.edu> wrote:
>> Sadly, yes, we’re just running the cli every five minutes to produce a
>> hybrid document from the different sources. If there’s a better approach to
>> produce the same end result, but with caching to enable conditional gets,
>> I’m not opposed.
>
> Not sure if this meets your needs but this bash library includes a
> conditional GET client:
> https://github.internet2.edu/InCommon/bash-library
>
> Tom
>
>> From: users [mailto:users-bounces at shibboleth.net] On Behalf Of Ian Young
>> Sent: Tuesday, March 21, 2017 7:03 AM
>> To: Shib Users <users at shibboleth.net>
>> Subject: Re: Performing conditional gets with the metadata aggregator
>>
>>
>>
>>
>>
>> On 20 Mar 2017, at 21:31, Wessel, Keith <kwessel at illinois.edu> wrote:
>>
>>
>>
>> So, with that said, Ian, what would the Spring code look like to perform a
>> conditional get in comparison to what's in the IdP?
>>
>>
>>
>> I took a quick look at this today. One thing to watch out for is that the
>> java-support cacheing client builders only cache documents up to a given
>> size, and that size is in both cases too low for the InCommon metadata
>> specifically. There are properties for these things, though.
>>
>>
>>
>> The other question, though, goes back to your original post:
>>
>>
>>
>> Wondered how one would configure the MDA to do a conditional get of metadata
>> in a bean using the file backed HTTP resource class.
>>
>>
>>
>> How are you using the MDA in your application? Are you using the provided
>> CLI interface, the mdq-server code or some other framework to invoke the MDA
>> stages?
>>
>>
>>
>> I ask because the cacheing provided by the java-support http client builders
>> doesn't seem at first glance to persist the document metadata, just its
>> body. That means that if you're just running the CLI once every five
>> minutes, you're not going to get any cacheing. Just want to make sure I am
>> not chasing down a blind alley here.
>>
>>
>>
>> Cheers,
>>
>>
>>     -- Ian
>>
>>
>>
>>
>>
>>
>>
>>
>> --
>> To unsubscribe from this list send an email to
>> users-unsubscribe at shibboleth.net
> --
> To unsubscribe from this list send an email to users-unsubscribe at shibboleth.net
> --
> To unsubscribe from this list send an email to users-unsubscribe at shibboleth.net
-- 
To unsubscribe from this list send an email to users-unsubscribe at shibboleth.net