Instrumenting metadata resolvers

[Cantor, Scott]

> I guess I don't have any definite thoughts yet.  It would be probably helpful
> to talk about actual concrete metrics use cases.  What types of things were
> you thinking?  I'm assuming some timer(s) around certain operations, for
> example 1) total resolution time (literally bookending the resolve() method)
> 2) time for executing remote fetches.  Any gauges? Maybe 1) # of entities
> currently "live" 2) number of remote fetches.

That's the big stuff, probably also some ratios on cache hits/misses. Might also be good to look at instrumenting the HttpClient as part of this.

> I haven't looked yet at where you wound up with the notion of using Metrics
> interfaces directly vs having some kind of library-independent abstraction.
> But I guess if we wind up with the former that's not the end of the world.

It isn't practical to wrap everything, there are too many interdependent classes (Histograms, Reservoirs). The work to wrap it would be more than the work to import all the code in the future and just take it over. What I like about Metrics is it's so brain-dead. It's basically the bare minimum you'd implement to do these things.

All I dead was implement stubs for disabled metrics and a lot of work on managing them.

> As far as how these are actually used: I liked your idea of using
> presence/absence of stuff in the context tree to control whether collection
> was on/off.  But of course in the metadata resolvers, unlike the profile
> actions and message handlers, we don't actually have that.

Yes, there are many places it would be tricky to do that. Nor did I do it for the authentication counters. It's just silly to introduce overhead for such simple reporting that's generally useful. I don't mind overhead to track the number of times some action runs because that's not a common need.

OTOH, counters for how often flows run probably are needed by lots of people. I'm just trying to work out what to do about the back button. Counting the number of times stale errors occur isn't interesting.

> I guess maybe
> kinda sorta you could stuff criteria into the input CriteriaSet to do a similar
> thing, but I'm not sure how I feel about that yet, vs just hardwiring things
> right in the resolver.  (As a side note, that would seem sort of like what we
> mused about for encryption logging in OSJ-156 (which btw I was probably
> going to work on very soon). So maybe it's not crazy.)

I suppose it's similar, yes. But CriteriaSet only takes us so far, we'll run into something that doesn't have that kind of input model. Either way, the basic issue is the API. We either have to create the dependency, or report the data out generically so I can wrap the resolver(s) like I just did with the services.

-- Scott