Modular Installation in the IdP

[Cantor, Scott]

On 5/15/20, 6:08 AM, "dev on behalf of Etienne Dysli Metref" <dev-bounces at shibboleth.net on behalf of etienne.dysli-metref at switch.ch> wrote:

> As an IdP extension developer, I appreciate this effort toward more
> modularity. :) What I still don't like about it, is the WAR rebuild step
> which has to happen outside of my usual Java development workflow and
> with a different tool.

I don't undertand why this is a problem *for deployment*, but I'm sure we could work on some kind of tool that's better or an alternative. But I really would have to think that anything fully scriptable should be relatively interchangeable as an approach.

We also might want to get clearer about what you mean by "development workflow". We obviously develop and we do not build warfiles to rrun and debug the code locally. That doesn't obviate the need to be able to debug into running containers, which is a different problem, but in terms of *development* workflow, we don't do that, though it's not trivial obviously to avoid it.

I sort of assumed this was more about *deployment* workflow. Maybe we need to step back and understand the specific problem of having a warfile, which I don't think is this particular plugin discussion either.

> This brings me to another idea I've been kicking around and I'll summarise it with "own
> your servlet container". The IdP could ship it's own servlet container
> by embedding Jetty or Tomcat akin to what Spring Boot does, but without
> the complicated JAR-in-JAR packaging.

Well, I think you acknowledged this is a very orthogonal discussion, but if you want to have it...

Embedding a container only "solves" things by requiring proxying from Apache or some other front end. With us owning the servlet container, customizing it adequately for real world use becomes intractable.

If you really meant "no WAR file" in the "run a web server inside our code" sense, that's even more out of bounds IMHO as a *production* deployment approach without assuming a proxy. And I rather imagine that's what you're assuming?

It's a debateable point worthy of discussing as to whether it's better for us in the end to ship a non-viable web story that has to be instantly proxied by Apache or a load balancer for production use, but gives us the ability to not actually lay out the code in servlet container form.

I don't personally buy that it's simpler in ways that should really matter but I can accept that others may feel it is.

> This greatly simplifies installation because deployers no longer have to maintain
> Jetty or Tomcat themselves (likely for the sole purpose of running the IdP) 

That's just not the case IMHO. They have to maintain *a* web front-end regardless, and now they have two to run (adding overhead and brittleness and security exposures, as all proxying does). Obviously if you know the other web server better, that's attractive, but the added work of deploying a container that isn't doing any of the real front-end work is just not that substantial either.

> Embedding your own web server is also a tendency I can observe among the
> web applications I operate as a developer: Jenkins, SonarQube, Sonatype
> Nexus. They all run their own embedded servlet containers. It also makes
> running these in [Docker] containers much easier.

All of them are either much more trivial in their web requirements, security disasters I would never expose to the Internet, or are simply punting on the problem. We are simply honest enough to admit that packaging a web server doesn't give you a web server, it just gives you an open HTTP port you have to proxy with a real web server (and/or load balancer, etc.)

And even so I'm still not opposed to it, but maybe the real question is, if you can't use the existing Internet2 packaging that uses Docker, doesn't that suggest that any single deployment package of this sort is probably unattainable? Everybody thinks they have the answer and that answer seems to be unique to every person.

My suggestion is that if this is the direction people want, we need people with the necessary skill sets to join the project to make it happen. We have resources to fund work if it's valued by the members. But we don't have the right people to do that work.

What I would say right now is that we don't intend or desire to make things worse with the plugin proposal. All we're trying to do is solve right now is the module management problem so functionality can be maintained more independently without making the installation or upgrade process any harder. In a perfect world maybe we'd have started with OSGi but we're just too far past that point.

I don't think any of that work is going to prevent any future directions that are taken related to the servlet container or use of the on-disk WAR assumption. If it does, I think we need to prevent that.

-- Scott