IdP OutOfMemoryError on refresh of UKFederation Metadata

[Matthew Slowe]

Morning Rob,

On 2021-11-13 19:13, Robert Hardy (r.hardy) wrote:
> Our IdP periodically stops responding to requests, showing a Java heap 
> space error when refreshing the UKFed metadata. This seems to be at 
> random intervals, and it can go weeks without showing the error. The 
> system always come back on restarting the service (whereon the refresh 
> appears to succeed). This started about three months ago.
> 
> The issue does not seem to be a lack of memory - shibd_idpw shows  max 
> memory pool set to 2048MB.

If memory serves, the OutOfMemory behaviour doesn't kill off the service 
but does cause the metadata refresh to fail leaving the previous 
metadata "in service" -- eventually that previous metadata expires and 
"real" services stop.

> We are still on IdP version 3.3.0.

Get in touch with the UK Federation Service Desk if you would like some 
help with getting that upgraded...!

> Also, at these times, although the IdP is not responding to requests 
> (externally or internally), "Get-Service shibd_idp" shows the service to 
> be running, and the output of both status.bat and aacli.bat are as if 
> the system were up.

Giving the service more Java heap to play with should buy you some time 
but, once you're past v3.4 I'd strongly suggest you look at MDQ as a way 
to receive the federation metadata rather than the (every growing) 
aggregate. It switches you over to an on-demand type of metadata access 
massively reducing your periodic memory & compute footprint.

https://www.ukfederation.org.uk/content/Documents/MDQ

https://shibboleth.atlassian.net/wiki/spaces/IDP30/pages/2527461422/MetadataQuery

Hope that helps,
-- 
Matthew Slowe (GPG: 0x6BE0CF7D04600314)
Senior Technical Consultant and Support specialist, Jisc
Team: 0300 300 2212, option 2
Lumen House, Library Avenue, Harwell Oxford, Didcot, OX11 0SG