<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1.0" /> <base href="https://issues.shibboleth.net/jira" />
<title>Message Title</title>
</head>
<body class="jira" style="color: #333; font-family: Arial, sans-serif; font-size: 14px; line-height: 1.429">
<table id="background-table" cellpadding="0" cellspacing="0" width="100%" style="border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt; background-color: #f5f5f5; border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt">
<!-- header here -->
<tr>
<td id="header-pattern-container" style="padding: 0px; border-collapse: collapse; padding: 10px 20px">
<table id="header-pattern" cellspacing="0" cellpadding="0" border="0" style="border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt">
<tr>
<td id="header-avatar-image-container" valign="top" style="padding: 0px; border-collapse: collapse; vertical-align: top; width: 32px; padding-right: 8px"> <img id="header-avatar-image" class="image_fix" src="cid:jira-generated-image-avatar-0dc57757-df0a-4229-ae8e-646295e267da" height="32" width="32" border="0" style="border-radius: 3px; vertical-align: top" />
</td>
<td id="header-text-container" valign="middle" style="padding: 0px; border-collapse: collapse; vertical-align: middle; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; mso-line-height-rule: exactly; mso-text-raise: 1px"> <a class="user-hover" rel="teimuzmd7u75fytaqjjv37uixwy=@https://signon.deakin.edu.au/idp/shibboleth" id="email_teimuzmd7u75fytaqjjv37uixwy=@https://signon.deakin.edu.au/idp/shibboleth" href="https://issues.shibboleth.net/jira/secure/ViewProfile.jspa?name=teimuzmd7u75fytaqjjv37uixwy%3D%40https%3A%2F%2Fsignon.deakin.edu.au%2Fidp%2Fshibboleth" style="color:#3b73af;; color: #3b73af; text-decoration: none">Mark Picone</a> <strong>commented</strong> on <a href="https://issues.shibboleth.net/jira/browse/CPPXT-116" style="color: #3b73af; text-decoration: none"><img src="cid:jira-generated-image-avatar-5a4b1469-7744-4dea-9e23-cc1f61e85c07" height="16" width="16" border="0" align="absmiddle" alt="Bug" /> CPPXT-116</a>
</td>
</tr>
</table>
</td>
</tr>
<tr>
<td id="email-content-container" style="padding: 0px; border-collapse: collapse; padding: 0 20px">
<table id="email-content-table" cellspacing="0" cellpadding="0" border="0" width="100%" style="border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt; border-spacing: 0; border-collapse: separate">
<tr>
<!-- there needs to be content in the cell for it to render in some clients -->
<td class="email-content-rounded-top mobile-expand" style="padding: 0px; border-collapse: collapse; color: #fff; padding: 0 15px 0 16px; height: 15px; background-color: #fff; border-left: 1px solid #ccc; border-top: 1px solid #ccc; border-right: 1px solid #ccc; border-bottom: 0; border-top-right-radius: 5px; border-top-left-radius: 5px; height: 10px; line-height: 10px; padding: 0 15px 0 16px; mso-line-height-rule: exactly">
</td>
</tr>
<tr>
<td class="email-content-main mobile-expand " style="padding: 0px; border-collapse: collapse; border-left: 1px solid #ccc; border-right: 1px solid #ccc; border-top: 0; border-bottom: 0; padding: 0 15px 0 16px; background-color: #fff">
<table class="page-title-pattern" cellspacing="0" cellpadding="0" border="0" width="100%" style="border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt">
<tr>
<td style="vertical-align: top;; padding: 0px; border-collapse: collapse; padding-right: 5px; font-size: 20px; line-height: 30px; mso-line-height-rule: exactly" class="page-title-pattern-header-container"> <span class="page-title-pattern-header" style="font-family: Arial, sans-serif; padding: 0; font-size: 20px; line-height: 30px; mso-text-raise: 2px; mso-line-height-rule: exactly; vertical-align: middle"> <a href="https://issues.shibboleth.net/jira/browse/CPPXT-116" style="color: #3b73af; text-decoration: none">Re: Apache 2.4 / Shibboleth Deadlock</a> </span>
</td>
</tr>
</table>
</td>
</tr>
<tr>
<td id="text-paragraph-pattern-top" class="email-content-main mobile-expand comment-top-pattern" style="padding: 0px; border-collapse: collapse; border-left: 1px solid #ccc; border-right: 1px solid #ccc; border-top: 0; border-bottom: 0; padding: 0 15px 0 16px; background-color: #fff; border-bottom: none; padding-bottom: 0">
<table class="text-paragraph-pattern" cellspacing="0" cellpadding="0" border="0" width="100%" style="border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; mso-line-height-rule: exactly; mso-text-raise: 2px">
<tr>
<td class="text-paragraph-pattern-container mobile-resize-text " style="padding: 0px; border-collapse: collapse; padding: 0 0 10px 0">
<p style="margin: 10px 0 0 0">Hi All,</p>
<p style="margin: 10px 0 0 0">Please see previous posts to the shibboleth mailing list: <a href="https://marc.info/?t=146189427500001&r=1&w=2" class="external-link" rel="nofollow" style="color: #3b73af; text-decoration: none">https://marc.info/?t=146189427500001&r=1&w=2</a></p>
<p style="margin: 10px 0 0 0">Robert Crossfield from our Appliations and Integrations team here at Deakin University has spent some time debugging our issue and seems to have found some locking issues in the xmltooling source code. We are after confirmation of this issue and feedback on the proposed patch.</p>
<p style="margin: 10px 0 0 0">Please see Robert's notes below and the attached patch files/scripts to trigger the bug:</p>
<p style="margin: 10px 0 0 0">*Problem: *<br /> Deadlock occurring in ReloadableXMLFile::lock (or in some cases, ReloadableXMLFile::reload_fn).</p>
<p style="margin: 10px 0 0 0">*Cause: *<br /> shibboleth2.xml timestamp changing during the course of a single loop of execution in ‘ap_process_http_*_connection’</p>
<p style="margin: 10px 0 0 0"><b>Explanation:</b><br /> Two instances of ShibTargetApache are being created on certain requests (ones requiring a redirect?), which leads to a deadlock in the event the timestamp changes between the creation of the first, and the second (as the first still holds the read-lock, while the second attempts to obtain write-lock).</p>
<p style="margin: 10px 0 0 0"><b>Traces:</b><br /> gdb ( bp on xmltooling::ReloadableXMLFile::lock )</p>
<p style="margin: 10px 0 0 0">The first lock occurs downstream from ‘ap_read_request’ here:<br /> <a href="https://github.com/apache/httpd/blob/fbc5e20ead005fd3a2bec05924f9e90dfd195406/modules/http/http_core.c#L146" class="external-link" rel="nofollow" style="color: #3b73af; text-decoration: none">https://github.com/apache/httpd/blob/fbc5e20ead005fd3a2bec05924f9e90dfd195406/modules/http/http_core.c#L146</a></p>
<p style="margin: 10px 0 0 0">Breakpoint 3, 0x00007fffe868e360 in xmltooling::ReloadableXMLFile::lock() () from /crobert_shib/lib/libxmltooling-lite.so.7<br /> (gdb) bt<br /> #0 0x00007fffe868e360 in xmltooling::ReloadableXMLFile::lock() () from /crobert_shib/lib/libxmltooling-lite.so.7<br /> #1 0x00007fffe8945990 in shibsp::AbstractSPRequest::AbstractSPRequest(char const*) () from /crobert_shib/lib/libshibsp-lite.so.7<br /> #2 0x00007fffe8c2f611 in ShibTargetApache (req=0x555555cc4010, this=0x555555d8cd40, __in_chrg=<optimized out>,<br /> __vtt_parm=<optimized out>) at mod_shib.cpp:353<br /> #3 shib_post_read (r=0x555555cc4010) at mod_shib.cpp:704<br /> #4 0x000055555557f0f0 in ap_run_post_read_request ()<br /> #5 0x000055555557ff21 in ap_read_request ()<br /> #6 0x00005555555a47ae in ap_process_http_connection ()<br /> #7 0x000055555559c890 in ap_run_process_connection ()<br /> #8 0x00007fffedc1b80f in child_main () from /etc/httpd/modules/mod_mpm_prefork.so<br /> #9 0x00007fffedc1ba0c in make_child () from /etc/httpd/modules/mod_mpm_prefork.so<br /> #10 0x00007fffedc1c791 in prefork_run () from /etc/httpd/modules/mod_mpm_prefork.so<br /> #11 0x00005555555795be in ap_run_mpm ()<br /> #12 0x0000555555572b46 in main ()</p>
<p style="margin: 10px 0 0 0">The second occurs from ‘ap_process_async_request’<br /> <a href="https://github.com/apache/httpd/blob/fbc5e20ead005fd3a2bec05924f9e90dfd195406/modules/http/http_core.c#L154" class="external-link" rel="nofollow" style="color: #3b73af; text-decoration: none">https://github.com/apache/httpd/blob/fbc5e20ead005fd3a2bec05924f9e90dfd195406/modules/http/http_core.c#L154</a></p>
<p style="margin: 10px 0 0 0">Breakpoint 3, 0x00007fffe868e360 in xmltooling::ReloadableXMLFile::lock() () from /crobert_shib/lib/libxmltooling-lite.so.7<br /> (gdb) bt<br /> #0 0x00007fffe868e360 in xmltooling::ReloadableXMLFile::lock() () from /crobert_shib/lib/libxmltooling-lite.so.7<br /> #1 0x00007fffe8945990 in shibsp::AbstractSPRequest::AbstractSPRequest(char const*) () from /crobert_shib/lib/libshibsp-lite.so.7<br /> #2 0x00007fffe8c2f611 in ShibTargetApache (req=0x555555cc9630, this=0x555556b2e800, __in_chrg=<optimized out>,<br /> __vtt_parm=<optimized out>) at mod_shib.cpp:353<br /> #3 shib_post_read (r=0x555555cc9630) at mod_shib.cpp:704<br /> #4 0x000055555557f0f0 in ap_run_post_read_request ()<br /> #5 0x00005555555a847a in internal_internal_redirect ()<br /> #6 0x00005555555a7636 in ap_internal_redirect ()<br /> #7 0x00007ffff07d2dec in handler_redirect () from /etc/httpd/modules/mod_rewrite.so<br /> #8 0x0000555555593290 in ap_run_handler ()<br /> #9 0x00005555555937d9 in ap_invoke_handler ()<br /> #10 0x00005555555a7bca in ap_process_async_request ()<br /> #11 0x00005555555a7ea4 in ap_process_request ()<br /> #12 0x00005555555a47f2 in ap_process_http_connection ()<br /> #13 0x000055555559c890 in ap_run_process_connection ()<br /> #14 0x00007fffedc1b80f in child_main () from /etc/httpd/modules/mod_mpm_prefork.so<br /> #15 0x00007fffedc1ba0c in make_child () from /etc/httpd/modules/mod_mpm_prefork.so<br /> #16 0x00007fffedc1c791 in prefork_run () from /etc/httpd/modules/mod_mpm_prefork.so<br /> #17 0x00005555555795be in ap_run_mpm ()<br /> #18 0x0000555555572b46 in main ()</p>
<p style="margin: 10px 0 0 0"><b>Possible Patch:</b></p>
<p style="margin: 10px 0 0 0">I don’t know if this behaviour is expected or not (ie. Should the same ShibTargetApache be found/used by the second call to shib_post_read but is being lost/missed?) </p>
<p style="margin: 10px 0 0 0">Without knowing the answer, we present this possible fix (patch is attached) </p>
<p style="margin: 10px 0 0 0">• Remove all write-lock and timestamp change code from ReloadableXMLFile::lock, and simply signal reload_fn.. which can update m_filestamp once the thread resumes</p>
<p style="margin: 10px 0 0 0"><b>How to cause the Deadlock:</b><br /> Two shell scripts can trigger it (~95% of the time – occasionally a second execution of curl_fetch.sh is required).<br /> With the attached patch applied we are unable the trigger the deadlock.</p>
</td>
</tr>
</table>
</td>
</tr>
<tr>
<td class="email-content-main mobile-expand " style="padding: 0px; border-collapse: collapse; border-left: 1px solid #ccc; border-right: 1px solid #ccc; border-top: 0; border-bottom: 0; padding: 0 15px 0 16px; background-color: #fff">
<table id="actions-pattern" cellspacing="0" cellpadding="0" border="0" width="100%" style="border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; mso-line-height-rule: exactly; mso-text-raise: 1px">
<tr>
<td id="actions-pattern-container" valign="middle" style="padding: 0px; border-collapse: collapse; padding: 10px 0 10px 24px; vertical-align: middle; padding-left: 0">
<table align="left" style="border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt">
<tr>
<td class="actions-pattern-action-icon-container" style="padding: 0px; border-collapse: collapse; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; mso-line-height-rule: exactly; mso-text-raise: 0px; vertical-align: middle"> <a href="https://issues.shibboleth.net/jira/browse/CPPXT-116#add-comment" target="_blank" title="Add Comment" style="color: #3b73af; text-decoration: none"> <img class="actions-pattern-action-icon-image" src="cid:jira-generated-image-static-comment-icon-4ac93799-47d1-4c90-8eab-efba2df72720" alt="Add Comment" title="Add Comment" height="16" width="16" border="0" style="vertical-align: middle" /> </a>
</td>
<td class="actions-pattern-action-text-container" style="padding: 0px; border-collapse: collapse; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; mso-line-height-rule: exactly; mso-text-raise: 4px; padding-left: 5px"> <a href="https://issues.shibboleth.net/jira/browse/CPPXT-116#add-comment" target="_blank" title="Add Comment" style="color: #3b73af; text-decoration: none">Add Comment</a>
</td>
</tr>
</table>
</td>
</tr>
</table>
</td>
</tr>
<!-- there needs to be content in the cell for it to render in some clients -->
<tr>
<td class="email-content-rounded-bottom mobile-expand" style="padding: 0px; border-collapse: collapse; color: #fff; padding: 0 15px 0 16px; height: 5px; line-height: 5px; background-color: #fff; border-top: 0; border-left: 1px solid #ccc; border-bottom: 1px solid #ccc; border-right: 1px solid #ccc; border-bottom-right-radius: 5px; border-bottom-left-radius: 5px; mso-line-height-rule: exactly">
</td>
</tr>
</table>
</td>
</tr>
<tr>
<td id="footer-pattern" style="padding: 0px; border-collapse: collapse; padding: 12px 20px">
<table id="footer-pattern-container" cellspacing="0" cellpadding="0" border="0" style="border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt">
<tr>
<td id="footer-pattern-text" class="mobile-resize-text" width="100%" style="padding: 0px; border-collapse: collapse; color: #999; font-size: 12px; line-height: 18px; font-family: Arial, sans-serif; mso-line-height-rule: exactly; mso-text-raise: 2px">
This message was sent by Atlassian JIRA <span id="footer-build-information">(v7.0.10#70120-<span title="37e3d7a6fc4d580639533e7f7c232c925e554a6a" data-commit-id="37e3d7a6fc4d580639533e7f7c232c925e554a6a}">sha1:37e3d7a</span>)</span>
</td>
<td id="footer-pattern-logo-desktop-container" valign="top" style="padding: 0px; border-collapse: collapse; padding-left: 20px; vertical-align: top">
<table style="border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt">
<tr>
<td id="footer-pattern-logo-desktop-padding" style="padding: 0px; border-collapse: collapse; padding-top: 3px"> <img id="footer-pattern-logo-desktop" src="cid:jira-generated-image-static-footer-desktop-logo-d28e2bda-a948-49fd-a419-6b2c4425b1bc" alt="Atlassian logo" title="Atlassian logo" width="169" height="36" class="image_fix" />
</td>
</tr>
</table>
</td>
</tr>
</table>
</td>
</tr>
</table>
</body>
</html>