<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1.0">
<base href="https://issues.shibboleth.net/jira">
<title>Message Title</title>
</head>
<body class="jira" style="color: #333333; font-family: Arial, sans-serif; font-size: 14px; line-height: 1.429">
<table id="background-table" cellpadding="0" cellspacing="0" width="100%" style="border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt; background-color: #f5f5f5; border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt" bgcolor="#f5f5f5">
<!-- header here -->
<tbody>
<tr>
<td id="header-pattern-container" style="padding: 0px; border-collapse: collapse; padding: 10px 20px">
<table id="header-pattern" cellspacing="0" cellpadding="0" border="0" style="border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt">
<tbody>
<tr>
<td id="header-avatar-image-container" valign="top" style="padding: 0px; border-collapse: collapse; vertical-align: top; width: 32px; padding-right: 8px" width="32"> <img id="header-avatar-image" class="image_fix" src="cid:jira-generated-image-avatar-e0837683-5f24-4821-94ee-7a912a601935" height="32" width="32" border="0" style="border-radius: 3px; vertical-align: top"> </td>
<td id="header-text-container" valign="middle" style="padding: 0px; border-collapse: collapse; vertical-align: middle; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; mso-line-height-rule: exactly; mso-text-raise: 1px"> <a class="user-hover" rel="cantor.2@osu.edu" id="email_cantor.2@osu.edu" href="https://issues.shibboleth.net/jira/secure/ViewProfile.jspa?name=cantor.2%40osu.edu" style="color:#0052cc;; color: #3b73af; text-decoration: none">Scott Cantor</a> <strong>commented</strong> on <a href="https://issues.shibboleth.net/jira/browse/SSPCPP-926" style="color: #3b73af; text-decoration: none"><img src="cid:jira-generated-image-avatar-8119087e-1b2a-4dc9-a9cf-34765eeff939" height="16" width="16" border="0" align="absmiddle" alt="Task"> SSPCPP-926</a> </td>
</tr>
</tbody>
</table> </td>
</tr>
<tr>
<td id="email-content-container" style="padding: 0px; border-collapse: collapse; padding: 0 20px">
<table id="email-content-table" cellspacing="0" cellpadding="0" border="0" width="100%" style="border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt; border-spacing: 0; border-collapse: separate">
<tbody>
<tr>
<!-- there needs to be content in the cell for it to render in some clients -->
<td class="email-content-rounded-top mobile-expand" style="padding: 0px; border-collapse: collapse; color: #ffffff; padding: 0 15px 0 16px; height: 15px; background-color: #ffffff; border-left: 1px solid #cccccc; border-top: 1px solid #cccccc; border-right: 1px solid #cccccc; border-bottom: 0; border-top-right-radius: 5px; border-top-left-radius: 5px; height: 10px; line-height: 10px; padding: 0 15px 0 16px; mso-line-height-rule: exactly" height="10" bgcolor="#ffffff"> </td>
</tr>
<tr>
<td class="email-content-main mobile-expand " style="padding: 0px; border-collapse: collapse; border-left: 1px solid #cccccc; border-right: 1px solid #cccccc; border-top: 0; border-bottom: 0; padding: 0 15px 0 16px; background-color: #ffffff" bgcolor="#ffffff">
<table class="page-title-pattern" cellspacing="0" cellpadding="0" border="0" width="100%" style="border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt">
<tbody>
<tr>
<td style="vertical-align: top;; padding: 0px; border-collapse: collapse; padding-right: 5px; font-size: 20px; line-height: 30px; mso-line-height-rule: exactly" class="page-title-pattern-header-container"> <span class="page-title-pattern-header" style="font-family: Arial, sans-serif; padding: 0; font-size: 20px; line-height: 30px; mso-text-raise: 2px; mso-line-height-rule: exactly; vertical-align: middle"> <a href="https://issues.shibboleth.net/jira/browse/SSPCPP-926" style="color: #3b73af; text-decoration: none">Re: Replace XML serialization in remoting layer</a> </span> </td>
</tr>
</tbody>
</table> </td>
</tr>
<tr>
<td id="text-paragraph-pattern-top" class="email-content-main mobile-expand comment-top-pattern" style="padding: 0px; border-collapse: collapse; border-left: 1px solid #cccccc; border-right: 1px solid #cccccc; border-top: 0; border-bottom: 0; padding: 0 15px 0 16px; background-color: #ffffff; border-bottom: none; padding-bottom: 0" bgcolor="#ffffff">
<table class="text-paragraph-pattern" cellspacing="0" cellpadding="0" border="0" width="100%" style="border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; mso-line-height-rule: exactly; mso-text-raise: 2px">
<tbody>
<tr>
<td class="text-paragraph-pattern-container mobile-resize-text " style="padding: 0px; border-collapse: collapse; padding: 0 0 10px 0"> <p style="margin: 10px 0 0 0; margin-top: 0">As I started thinking about the character set implications of the Java side of this, I took a dive back into the history of the safe vs. unsafe string distinction in the old code to try and remember why I did what I did. The bulk of this dates back to <a href="https://issues.shibboleth.net/jira/browse/SSPCPP-195" title="Web server module can build incorrect XML messages which shibd then fails to parse" class="issue-link" data-issue-key="SSPCPP-195" style="color: #3b73af; text-decoration: none"><del>SSPCPP-195</del></a> and had to do with some arguably incorrect inputs.</p> <p style="margin: 10px 0 0 0">I believe it's flat out illegal to embed ASCII characters above x7F into URLs. Once you hit the "local code page" characters, all of those are supposed to be URL-encoded, and not just appear in the URL directly. The problem was people do it, and the system was breaking because the code actually treated all DDF string values as UTF-8, period. That allowed the values to be streamed into an XML document for sending, and then parsed back out and converted back via toUTF8 from the Xerces DOM.</p> <p style="margin: 10px 0 0 0">Any time values got passed to shibd that were NOT UTF-8, this broke because the embedded characters are in the 2-byte encoding range of UTF-8 and the toUTF8 calls failed due to invalid surrogate pairs. So my code comments refer to the unsafe_string construct as being for "non-UTF8 data" which was confusing the hell out of me, but now I remember the issue.</p> <p style="margin: 10px 0 0 0">Illegal URLs aside, it was also there to protect other data that could absolutely be non-UTF-8 such as cookies, form data, etc.</p> <p style="margin: 10px 0 0 0">So...the new serialization is not a problem. In fact it simply adopts the fully universal fix of always URL-encoding data so that the encoding is not assumed inside the DDF layer, at the cost of doing that extra encoding, but the decoding side is way cheaper than parsing XML and doing UTF-16 to 8 conversion.</p> <p style="margin: 10px 0 0 0">I probably will apply the same fix to the node name as well, just out of defensiveness but I stopped putting data into the names anyway.</p> <p style="margin: 10px 0 0 0">So...here's the problem. Java is Unicode aware. That's going to blow up very, very badly. Because I don't even know what the encoding for much of this data even is, I can't possibly tell Java what it is, which means it can't be turned into a Java String safely and we're right back where we started before I fixed -195.</p> <p style="margin: 10px 0 0 0">I don't know how to deal with that or even if I can. With web servers, you tell it to do UTF-8 everywhere you can and you pray and it mostly works until it doesn't (just search for Tomcat and UTF-8). Web servers are, I would imagine, doing Locale-aware String construction out of byte arrays to try and make things work.</p> <p style="margin: 10px 0 0 0">I think this is going to make porting shibd to Java quite hard in practice, if not in "common" cases. I can revert the change and go back to identifying "unsafe" data, but this would have to extend to actually representing the data as being unsafe in the DDF accessors, whereas I never had to do that with C++. In Java terms, it would mean probably exposing that data as a byte array rather than String.</p> <p style="margin: 10px 0 0 0">The problem is...this includes the damn URL itself, which is a necessary input to various processing steps. My guess is that it will prove to be intractable to make this work on illegal URLs and that we'll have to focus on graceful failure and not just "allow stuff to work if the web server's happy with it". That is going to cause problems in practice.</p> </td>
</tr>
</tbody>
</table> </td>
</tr>
<tr>
<td class="email-content-main mobile-expand " style="padding: 0px; border-collapse: collapse; border-left: 1px solid #cccccc; border-right: 1px solid #cccccc; border-top: 0; border-bottom: 0; padding: 0 15px 0 16px; background-color: #ffffff" bgcolor="#ffffff">
<table id="actions-pattern" cellspacing="0" cellpadding="0" border="0" width="100%" style="border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; mso-line-height-rule: exactly; mso-text-raise: 1px">
<tbody>
<tr>
<td id="actions-pattern-container" valign="middle" style="padding: 0px; border-collapse: collapse; padding: 10px 0 10px 24px; vertical-align: middle; padding-left: 0">
<table align="left" style="border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt">
<tbody>
<tr>
<td class="actions-pattern-action-icon-container" style="padding: 0px; border-collapse: collapse; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; mso-line-height-rule: exactly; mso-text-raise: 0; vertical-align: middle"> <a href="https://issues.shibboleth.net/jira/browse/SSPCPP-926#add-comment" target="_blank" title="Add Comment" style="color: #3b73af; text-decoration: none"> <img class="actions-pattern-action-icon-image" src="cid:jira-generated-image-static-comment-icon-d0cdb566-7dd2-45de-ac41-481f5d4dcfca" alt="Add Comment" title="Add Comment" height="16" width="16" border="0" style="vertical-align: middle"> </a> </td>
<td class="actions-pattern-action-text-container" style="padding: 0px; border-collapse: collapse; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; mso-line-height-rule: exactly; mso-text-raise: 4px; padding-left: 5px"> <a href="https://issues.shibboleth.net/jira/browse/SSPCPP-926#add-comment" target="_blank" title="Add Comment" style="color: #3b73af; text-decoration: none">Add Comment</a> </td>
</tr>
</tbody>
</table> </td>
</tr>
</tbody>
</table> </td>
</tr>
<!-- there needs to be content in the cell for it to render in some clients -->
<tr>
<td class="email-content-rounded-bottom mobile-expand" style="padding: 0px; border-collapse: collapse; color: #ffffff; padding: 0 15px 0 16px; height: 5px; line-height: 5px; background-color: #ffffff; border-top: 0; border-left: 1px solid #cccccc; border-bottom: 1px solid #cccccc; border-right: 1px solid #cccccc; border-bottom-right-radius: 5px; border-bottom-left-radius: 5px; mso-line-height-rule: exactly" height="5" bgcolor="#ffffff"> </td>
</tr>
</tbody>
</table> </td>
</tr>
<tr>
<td id="footer-pattern" style="padding: 0px; border-collapse: collapse; padding: 12px 20px">
<table id="footer-pattern-container" cellspacing="0" cellpadding="0" border="0" style="border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt">
<tbody>
<tr>
<td id="footer-pattern-text" class="mobile-resize-text" width="100%" style="padding: 0px; border-collapse: collapse; color: #999999; font-size: 12px; line-height: 18px; font-family: Arial, sans-serif; mso-line-height-rule: exactly; mso-text-raise: 2px"> This message was sent by Atlassian Jira <span id="footer-build-information">(v8.5.4#805004-<span title="0444eab799707f9ad7b248d69f858774aadfd250" data-commit-id="0444eab799707f9ad7b248d69f858774aadfd250}">sha1:0444eab</span>)</span> </td>
<td id="footer-pattern-logo-desktop-container" valign="top" style="padding: 0px; border-collapse: collapse; padding-left: 20px; vertical-align: top">
<table style="border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt">
<tbody>
<tr>
<td id="footer-pattern-logo-desktop-padding" style="padding: 0px; border-collapse: collapse; padding-top: 3px"> <img id="footer-pattern-logo-desktop" src="https://issues.shibboleth.net/jira/images/mail/atlassian-email-logo.png" alt="Atlassian logo" title="Atlassian logo" width="191" height="24" class="image_fix"> </td>
</tr>
</tbody>
</table> </td>
</tr>
</tbody>
</table> </td>
</tr>
</tbody>
</table>
</body>
</html>