Metrics follow-up thoughts

[Cantor, Scott]

A few ruminations after the dev call.

It's worth noting that supporting Metrics itself doesn't preclude people doing outside-in measuring with other tools, particularly if it's easy to turn off the Metrics piece. I also think that's a much better way of addressing *performance monitoring* as distinct from behavioral/statistical analysis or reporting on internal configuration state.

We could do timers for some very significant application events (like the start/stop of flows), but I think that's a far cry from actually introducing timers into every action bean, though we could do that in the action base classes if we really wanted to.

On the subject of turning this on/off, I wonder if there's a possibility of leveraging the logging API to tune the behavior of any measuring code, whether it's specifically using the Metrics API or not. Couldn't it be acting based on the level of a matching logging category?

This is a bad example because it's too lightweight to bother turning off, but if you had a Counter metric called "authn.ldap.failure" in the "net.shibboleth.idp.metric" metric registry, you could check a logger for "net.shibboleth.idp.metric.authn.ldap.failure" and then:

if (metriclevel.isInfoEnabled()) {
	failures.inc();
}

You could even have discrete log levels assigned to different measuring code based on the cost so you'd get general performance impacts based on the log level for net.shibboleth.idp.metric.

That makes the tuning reloadable, of course, since the log levels are.

This potentially extends to the reporting level if we wanted it to (hiding/ignoring a metric that's "off"), though most of the real performance impact is really measuring, not the reporting.

Separately...

A consideration with some of this is that our actions can't actually be maintaining their own metric objects anyway, since they're prototypes. That does complicate a lot of this no matter how we do it, since that means in a lot of cases, there's no way to track the information in a way that's internal to the actions that might be recording it. The authn case is an example of that. We have no choice but to store the data independently of the actions. So either we just build a collector class ourselves for that data, or we inject a Metrics object into the action(s) to maintain it, or they access the global shared object to get the metric by name (in which case we have to tell it the name).

As I think I aluded to on the call, mostly for Brent, I think it would be useful to consider the Dynamic resolver code as an example where there are a number of metrics we'd want to be able to report, and since that's in OpenSAML, it would be a good case study for how this should be handled. Maybe that would help point the way towards a way of insulating the IdP code more than I had envisioned.

-- Scott