Very slow processing of attribute-filter.xml with many AttributeFilterPolicy elements

[Lukas Hämmerle]

Hello

When the eduGAIN metadata recently grew considerably by several hundred
SPs, this apparently caused issues for a few of our (and as I've heard
HEANet's) IdPs due to the increase in size of the attribute filter files.

The attribute files we generate contain for each SP an attribute filter
policy rule like (plus a few xml comments):

>     <AttributeFilterPolicy id="afp_for:https://issues.shibboleth.net/shibboleth">
>         <PolicyRequirementRule xsi:type="basic:AttributeRequesterString" value="https://issues.shibboleth.net/shibboleth" />
>         <AttributeRule attributeID="email">
>             <PermitValueRule xsi:type="basic:ANY" />
>         </AttributeRule>
>         <AttributeRule attributeID="eduPersonTargetedID">
>             <PermitValueRule xsi:type="basic:ANY" />
>         </AttributeRule>
>         <AttributeRule attributeID="displayName">
>             <PermitValueRule xsi:type="basic:ANY" />
>         </AttributeRule>
>     </AttributeFilterPolicy>

A sample file to play around would be (temporarily) here:
http://www.switch.ch/aai/downloads/attribute-filter.test.xml

So, in total there are now almost 1500 AttributeFilterPolicy elements
and about 6700 AttributeRule elements. The attribute-filter.xml file
size about 2.1MB (650kB without comments/indentations).

The problem is that the IdP (2.4.2, 1GB heap, 2GB total memory) now
takes about several minutes to load this attribute filter file. During
that time the IdP is blocked and login hardly possible. Compared to the
situation where we had only about half the SPs, the processing time
seems to have increased exponentially. Our findings seem to indicate
that minimizing the number of AttributeFilterPolicy elements seems to
considerably improve the processing time. So, one way out of this would
be to have one AttributeFilterPolicy for each attribute and then list
the SPs where a particular attribute is released to instead of a per-SP
AttributeFilterPolicy.

We intend to do some optimization for the filter files but still, why
does loading and processing a 650kB XML file need so much time and what
optimizations would we best perform? Given the interfederation metadata
file is almost four times as large and does not take more than a few
seconds to load and process, it must have to do something with the
internal representation of attribute filter rules.


Best Regards
Lukas

-- 
SWITCH
Lukas Hämmerle, Central Solutions
GÉANT Project Task Leader "Enabling Users"
Werdstrasse 2, P.O. Box, 8021 Zurich, Switzerland
phone +41 44 268 15 05, direct +41 44 268 15 64
lukas.haemmerle at switch.ch, http://www.switch.ch