Shibboleth, Tomcat, AJP and UTF-8 conversion

[Brent Putman]

On 10/19/16 8:55 PM, Mark Juszczec wrote:
>
> Tomcat is configured to use AJP 1.3 for communication.
>
> The problem is the UTF-8 data is corrupted once it comes off the AJP
> port. 
>
> I have debugged into the Tomcat AJP classes and see the bytes are
> correct.  They arrive in an object called ByteChunk and are then
> converted to a String and the charset is ISO-8859-1.
>
> The String is incorrect.
>
> ISO-8859-1 is the default used by ByteChunk and I've verified it is
> not reset/changed to UTF-8 despite having specified it in server.xml
> per Tomcat documentation.
>
>

I don't think I've fully groked all the details in this thread, but
wanted to share 2 things which I haven't seen yet which might be relevant:

1) In the servlet spec (section 3.11) it does say:

> The default encoding of a request the container uses to create the
> request reader and parse POST data must be “ISO-8859-1” if none has
> been specified
> by the client request.

which means mainly the charset on an entity body Content-Type type. 
But read on..


2) Since many requests don't include a request character encoding, and
lots of things want it to be UTF-8, Tomcat comes with servlet filter
which looks at the HttpServletRequest's characterEncoding property, and
if null, sets it explicitly to UTF-8 (by default, actually you can
configure any encoding you want).  This is again mainly about
processing POSTed data, according to the comments, but since the
request encoding is always set/available, it might have some relevance
to this issue.  You can find this filter in the global conf/web.xml as
"setCharacterEncodingFilter".  It's commented out by default.

As someone else pointed out, there are some params on the AJP
connector, but those appear to be about the URI query params only.  The
servlet filter approach might work for this case, since it globally
sets the whole request encoding.

(Backstory: I recently had an issue with non-ISO-8859-1 characters in
POST params on an signed LTI launch request.  The interpretation of
them as ISO-8859-1 in the Java app was causing the signature to be
broken.  The servlet filter solution seemed to fix.  Really IMHO the
signer needs to put the charset in the posted Content-Type, but that's
another matter...)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://shibboleth.net/pipermail/users/attachments/20161020/022c9d2e/attachment.html>