non Fast-fail, LDAP and the testbed.. And Guages.

[Rod Widdowson]

For reasons which I have since forgotten I wanted to enable LDAP attribute fetch for the testbed, at which stage I rapidly fell down
a rabbit hole.

The first thing I discovered is that within the testbed, the LDAP server is started up after the attribute resolver service.  So the
initial load of the attribute resolver fails.  Ahah! I thought, why not turn up the connect timeout?  But things still failed, they
just took longer.

This of course is because all the services are started serially and sequentially and as part of every service initialize we try do
one a one-off load of the configuration, failing immediately if we are failfast.  So by turning up the connect timeout all I was
doing was delaying the attribute service in it's initial (failing) load, and thus the start of the LDAP server.

So my first question is whether we shouldn't actually change the service code and _only_ do the initial load (inside the service
initialization) if this is a failfast service? Otherwise we can just start the reload task and tell it to run a reload immediately
(but in a separate thread).    This is sort of like what we did inside the MetadataResolvers.  It will and it make for a faster
reload in many cases, albeit that the IdP will probably be initially unusable until all the services have done their initial load.
Or is there some implied ordering which would be broken (for instance the Logging Service might a still be initializing while we
start the Attribute Resolver Service.

But the real reason why the testbed failed was because even though we had started up the Attribute Resolver Service, when we went to
create the Attribute Resolver GaugeSet  that failed because at that stage the attribute resolver service still hadn't done a
successful initial load.  This is because in two of our ReloadableServiceGaugeSet deployments (Attribute Resolver and Metadata
Resolver) we are pretty draconian and we require that when we start the GuageSet the service MUST be running, and it MUST been
loaded - this by the way is why people who have misconfigurations always quote lines about the GuageSet not loading.

I can see the motivation - if someone does manage to configure and attribute filter service as the attribute resolver service then
the guage code will quietly sit doing nothing about it specialism (it won't have a class cast issue because we are safe against
that).  But quite frankly that is probably the least of the issues that the deployers will have at this point.

In my case I had a perfectly valid configuration file, it just hadn't started yet.    Of course in a real world deployment people
would be quite likely to have a fallback data connector and maybe that is what I should be doing in the case of the testbed.  

But I am still wondering whether the tests are too draconian?  In my case if I allowed null (but still failed against non null of
the wrong type) allows the testbed to start. 

Thoughts?