IdP and attribute resolver DataConnector monitoring

[Chris Reeves]

Hi,

I have two (vaguely) related questions, to which I'll give a little more
background below:

What recommendations do you have for monitoring the IdP in order to determine
that it is happy and healthy (e.g. all db pools are functioning/not broken)?

Why does the status page always show that none of our DataConnectors have ever
failed, even when we know (and at least something within the IdP knows) that
they are broken?


The IdP status page is used by both our monitoring systems and our
load-balancer in order to check the health of the IdP process on any given
server. In particular, they check for the string 'attribute_resolver_valid:
true' as we've found that in our organisation the most common cause for the
IdP to have issues is when one of the core databases that it uses for
attribute resolution goes away, and this is a fairly reliable indicator that
all is well.

We are about to migrate to v3, however we have noted that in v3 that string no
longer appears. Instead the status of each individual data connector is
listed, for example: 
  DataConnector db1: has never failed
In principle we could update the logic in our monitoring systems to ensure all
individual connectors are (recently) healthy, or modify the status page itself
to report a combined value as it did before, however we appear to be having
issues with detecting failures on individual connectors (see my second
question). Hence my first question: what recommendations do you have for
monitoring the IdP in order to determine that it is happy and healthy (e.g.
all db pools are functioning/not broken)?


As for detecting failures on individual connectors, the configuration for our
database connectors in the attribute resolver is essentially unchanged from
the v2 configuration - here is an example:
  <resolver:DataConnector id="db1" xsi:type="RelationalDatabase" xmlns="urn:mace:shibboleth:2.0:resolver:dc" queryTimeout="PT10S">
    <ApplicationManagedConnection jdbcDriver="oracle.jdbc.driver.OracleDriver" poolMaxSize="8"
        jdbcURL="jdbc:oracle:thin:@dbserver.uni.ac.uk:1521:people" jdbcUserName="dbuser" jdbcPassword="dbpass" />
    <QueryTemplate>
        <![CDATA[
  select distinct attribute_value from shib.user_attributes where attribute_name = 'eduPersonEntitlement' and username = '$requestContext.principalName'
        ]]>
    </QueryTemplate>
    <Column columnName="ATTRIBUTE_VALUE" attributeID="eduPersonEntitlement" />
  </resolver:DataConnector>

In order to test a failure scenario we performed a number of successful
authentications and then modified the firewall on our test server to reject
all connections to the database server. As expected, attribute resolution
fails at this point and we see the following error in idp-process.log:
  2016-09-05T14:34:19.606+01:00 - ERROR [net.shibboleth.idp.profile.impl.ResolveAttributes:257] - Profile Action ResolveAttributes: Error resolving attributes
  net.shibboleth.idp.attribute.resolver.ResolutionException: Data Connector 'db1': Unable to execute SQL query
          at net.shibboleth.idp.attribute.resolver.dc.rdbms.impl.RDBMSDataConnector.retrieveAttributes(RDBMSDataConnector.java:170)
  Caused by: java.sql.SQLException: An SQLException was provoked by the following failure: com.mchange.v2.resourcepool.ResourcePoolException: Attempted to use a closed or broken resource pool
          at com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:118)
  Caused by: com.mchange.v2.resourcepool.ResourcePoolException: Attempted to use a closed or broken resource pool
          at com.mchange.v2.resourcepool.BasicResourcePool.ensureNotBroken(BasicResourcePool.java:1735)

However, if we look at the status page there are no failures logged for this
database connector:
  DataConnector db1: has never failed

We see the same behaviour if we start the IdP in the same failed state
(rejecting all connections of the database server). The IdP starts up and
throws errors indicating that it couldn't validate its datasource, but the
status page still reports that none of the data connectors have ever failed.

Why does the status page always show that none of our DataConnectors have ever
failed, even when we know (and at least something within the IdP knows) that
they are broken? I've found a thread from June 2015[1], and the resulting
IDP-743, which shed some light on both of my questions, however it doesn't
explain why the status page reports that no connectors have ever failed. Is
the reporting on the status page only valid for datasources using something
more modern like BeanManagedConnection (according to IDP-743 and status.jsp it
should only list connectors which support DataSourceEx)?

IdP 3.2.1/Ubuntu 16.04/Tomcat 8


Thanks in advance for any advice/suggestions.

Regards,
    Chris
    
[1] https://shibboleth.net/pipermail/users/2015-June/022207.html