User sessions mixed up when Java app deployed to Glassfish is fronted with Apache httpd

Thu Oct 30 09:42:33 EDT 2014

On Thu, Oct 30, 2014 at 12:17 AM, Cantor, Scott <cantor.2 at osu.edu> wrote:
> On 10/29/14, 6:47 PM, "Philip Durbin" <philip_durbin at harvard.edu> wrote:
>>
>>Users' sessions are getting mixed up!
>
> Unless that session is the one the SP is managing, the SP can't be
> involved in the issue. That session would be the one that has the headers
> populated by the SP and that's really the only manifestation of that
> session. It has nothing to do with the Java container's session, for
> example. And if you're doing anything to try and bind them together, it's
> distinctly possible that is being done incorrectly in some way.

I'm having trouble understanding what you mean. I agree that the
scrambled sessions probably has nothing to do with the Shibboleth SP
"shibd" process that's running. In fact, I'm pretty sure that if I
were to stop shibd and uninstall mod_shib and the rest of the
Shibboleth package that the problem would continue. When I decided to
evaluate mod_shib as a solution I knew that I'd need to introduce
Apache into our setup, that I'd need to front Glassfish with Apache to
use mod_shib. If memory serves, I introduced Apache first (months ago)
and we started seeing the problem, even without Shibboleth installed.

I hope I'm not unintentionally overloading the term "session". When I
say users' sessions are mixed up what I mean is that an anonymous user
is suddenly logged in as someone else. From what I understand, since
we're using Java, what controls this is the JSESSIONID cookie that
looks something like this:

Set-Cookie: JSESSIONID=1190b53dc5cf345521d7e6e2ad5a

>>This is just a demo/beta site but obviously this is a show stopper for
>>us moving forward with mod_shib in production. Even on the demo site
>>I'm tempted to pull Apache httpd out of our setup because I don't want
>>people are looking at the demo site to get scared by suddenly seeing
>>that they are logged in as someone else! We never see this problem
>>when we run only Glassfish. The scrambled user sessions has something
>>to do with introducing Apache httpd into the mix.
>
> Maybe, but I'm highly skeptical of that unless Glassfish itself is doing
> something to expose a bug. That's exactly how everybody else has used the
> software for a decade with nothing like this ever being reported.

Certainly, when I mention this problem to people who use Glassfish,
they think I'm crazy. "we use apache here to front our systems, and
I've never seen this," they say at
http://www.evanchooly.com/logs/%2523glassfish/2014-10-29

And yet, it seems like scrambled sessions is a relatively common
problem when Apache fronts Java apps. In addition to the links I
provided initially, here are some more examples:

1. JIRA: "The session spontaneously switches to another user. The
JSESSIONID cookie of the victim is set for the "perpetrator" (in
complete ignorance as well as innocence), leading to the
session-stealing behavior."
https://confluence.atlassian.com/display/JIRAKB/Request+Assumes+Identity+of+Another+User+who+Logs+in+Concurrently+Due+to+Apache+CacheIgnoreHeaders

2. Jenkins: "Running Jenkins behind Apache: mod_proxy with HTTPS...
issue is I'm an admin and random developer will login and see that
there user id is mine and can admin jenkins"
https://issues.jenkins-ci.org/browse/JENKINS-12585

>><Location /shib.xhtml>
>>  AuthType shibboleth
>>  ShibRequestSetting requireSession 1
>>  require valid-user
>></Location>
>
> What's the relevance of that piece?

I don't protect the entire application behind Shibboleth. The general
public should be able to browse the site. People with Shibboleth
accounts click "Log In", select their Institution (using EDS), log in
via their IDP, and are re-directed to /shib.xhtml which processes the
Shib attributes and either logs users in or prompts them to confirm
the creation of their account in our app.

The user experience is described here with screenshots:
http://datascience.iq.harvard.edu/blog/try-out-single-sign-shibboleth-40-beta

>>I'm running on CentOS 6.5 and using httpd-2.2.15 which means I'm using
>>mod_proxy_ajp, as the wiki recommends (rather than mod_jk). I'm
>>running GlassFish Server Open Source Edition 4.0 (build 89).
>
> I know absolutely nothing about Glassfish other than that it exists and it
> seems to be a giant pain every time it comes up.

[snip]

>>Any advice is welcome! Please tell me what I'm doing wrong!
>
> I don't see anything wrong, that's pretty standard stuff. But there's
> nothing wrong with the SP used in this combination with every other
> container I've seen used. AJP is used routinely with both Tomcat and Jetty
> without any issues like this.

All of our experience is with Glassfish and it would be quite an
effort to attempt to switch to another "Full Platform" Java Enterprise
Edition (EE) application server such as Wildfly (the open source
version of JBoss). We can't use Tomcat or Jetty because our
application requires many of the Java EE APIs that are only
implemented in "Full Platform" Java EE servers, such as Java Message
Service (JMS).

> I suppose one thing you could do is switch to a mod_proxy_http approach,
> as one has to use with WebLogic. That can be used with any container,
> really, since it's just a reverse proxy to the container. It's not the
> better option, but I guess it might highlight something.

I'm not sure I completely understand the suggestion to try
mod_proxy_http instead. Are you saying that we would no longer use AJP
at all? I can't seem to find much information on this.

>>p.s. People reporting similar problems (and how tricky it is to
>>reproduce reliably):
>
> Hmm. Well, those seem to in part indict mod_jk, which is not code anybody
> should use at this point, but you're not using it. They also seem to
> suggest Glassfish is a common component, which suggests to me Glassfish
> has a broken AJP implementation perhaps.

Glassfish certainly could be the problem. I don't know. I was hoping
it's a simple misconfiguration on my part.

>>http://stackoverflow.com/questions/14845493/spring-security-jsf-hibernate-
>>accidental-session-hijacking-on-tomcat
>
> That last one seems to clearly be an application level issue.

I'm currently investigating if there's something wrong with our code.
Yesterday I had one person who knows Java EE quite well say that there
doesn't seem to be a problem with our code. I had another person say,
""DataverseSession is injected to stateless PermissionServiceBean
which makes DataverseSession an instance variable of
PermissionServiceBean. Stateless bean is thread safe but does it
guarantee the initialization of DataverseSession using the Dataverse
Session of a user. What happens when a pooled PermissionServiceBean
instance is given by container to process a request from a user? Is it
going to get the DataverseSession of earlier user?" At this point I
have no idea who is right but I'll continue to investigate this, of
course. Again I'll re-iterate that we don't have any problem with
scrambled sessions when we use only Glassfish, when there is no Apache
in the mix.

> As I said initially, it's critical to identify exactly what session is
> really involved here.

Yes, I agree. In #glassfish the advice was "I would try to see if you
can get better logging... so you can track IP to Cookies, see where it
switched".

> There's actually a Shib-Session-ID variable/header used that does pass in
> the session ID coming from the SP cookie. It might help to see if that's
> actually showing a switch.

Ok, good to know. At the moment I'm not even looking hard at the
Shibboleth angle. I think the SP is innocent, which is why I didn't
even bother giving the version number (2.5.3 for the curious). Mostly
I'm appealing to the kind people on this list because others might be
introducing Apache into their Java app server setup.

Thanks for reading this far. I wish I could say I'm enjoying my
experience adding Shibboleth support to our Java application. It's
been a long, hard road already and we're not even there yet. Again,
this session scrambling bug is hard to reproduce yet critically
important to squash.

Thanks,

Phil

-- 
Philip Durbin
Software Developer for http://dataverse.org
http://www.iq.harvard.edu/people/philip-durbin