<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1.0" /> <base href="https://issues.shibboleth.net/jira" />
<title>Message Title</title>
</head>
<body class="jira" style="color: #333; font-family: Arial, sans-serif; font-size: 14px; line-height: 1.429">
<table id="background-table" cellpadding="0" cellspacing="0" width="100%" style="border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt; background-color: #f5f5f5; border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt">
<!-- header here -->
<tr>
<td id="header-pattern-container" style="padding: 0px; border-collapse: collapse; padding: 10px 20px">
<table id="header-pattern" cellspacing="0" cellpadding="0" border="0" style="border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt">
<tr>
<td id="header-avatar-image-container" valign="top" style="padding: 0px; border-collapse: collapse; vertical-align: top; width: 32px; padding-right: 8px"> <img id="header-avatar-image" class="image_fix" src="cid:jira-generated-image-avatar-b8ace060-e580-47cf-b6b1-cd2281fe4a80" height="32" width="32" border="0" style="border-radius: 3px; vertical-align: top" />
</td>
<td id="header-text-container" valign="middle" style="padding: 0px; border-collapse: collapse; vertical-align: middle; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; mso-line-height-rule: exactly; mso-text-raise: 1px"> <a class="user-hover" rel="vtsji@unitedid.org" id="email_vtsji@unitedid.org" href="https://issues.shibboleth.net/jira/secure/ViewProfile.jspa?name=vtsji%40unitedid.org" style="color:#3b73af;; color: #3b73af; text-decoration: none">vtsji@unitedid.org</a> <strong>created</strong> an issue
</td>
</tr>
</table>
</td>
</tr>
<tr>
<td id="email-content-container" style="padding: 0px; border-collapse: collapse; padding: 0 20px">
<table id="email-content-table" cellspacing="0" cellpadding="0" border="0" width="100%" style="border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt; border-spacing: 0; border-collapse: separate">
<tr>
<!-- there needs to be content in the cell for it to render in some clients -->
<td class="email-content-rounded-top mobile-expand" style="padding: 0px; border-collapse: collapse; color: #fff; padding: 0 15px 0 16px; height: 15px; background-color: #fff; border-left: 1px solid #ccc; border-top: 1px solid #ccc; border-right: 1px solid #ccc; border-bottom: 0; border-top-right-radius: 5px; border-top-left-radius: 5px; height: 10px; line-height: 10px; padding: 0 15px 0 16px; mso-line-height-rule: exactly">
</td>
</tr>
<tr>
<td class="email-content-main mobile-expand " style="padding: 0px; border-collapse: collapse; border-left: 1px solid #ccc; border-right: 1px solid #ccc; border-top: 0; border-bottom: 0; padding: 0 15px 0 16px; background-color: #fff">
<table class="page-title-pattern" cellspacing="0" cellpadding="0" border="0" width="100%" style="border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt">
<tr>
<td class="page-title-pattern-first-line " style="padding: 0px; border-collapse: collapse; font-family: Arial, sans-serif; font-size: 14px; padding-top: 10px"> <a href="https://issues.shibboleth.net/jira/browse/SSPCPP" style="color: #3b73af; text-decoration: none">Shibboleth SP - C++</a> / <a href="https://issues.shibboleth.net/jira/browse/SSPCPP-712" style="color: #3b73af; text-decoration: none"><img src="cid:jira-generated-image-avatar-fd7556c6-2b5e-4d4d-a52d-1fa57bf72299" height="16" width="16" border="0" align="absmiddle" alt="Bug" style="vertical-align: text-bottom" /></a> <a href="https://issues.shibboleth.net/jira/browse/SSPCPP-712" style="color: #3b73af; text-decoration: none">SSPCPP-712</a>
</td>
</tr>
<tr>
<td style="vertical-align: top;; padding: 0px; border-collapse: collapse; padding-right: 5px; font-size: 20px; line-height: 30px; mso-line-height-rule: exactly" class="page-title-pattern-header-container"> <span class="page-title-pattern-header" style="font-family: Arial, sans-serif; padding: 0; font-size: 20px; line-height: 30px; mso-text-raise: 2px; mso-line-height-rule: exactly; vertical-align: middle"> <a href="https://issues.shibboleth.net/jira/browse/SSPCPP-712" style="color: #3b73af; text-decoration: none">Shibd leaks file handles after rotating logs and touching shibboleth2.xml</a> </span>
</td>
</tr>
</table>
</td>
</tr>
<tr>
<td class="email-content-main mobile-expand wrapper-special-margin" style="padding: 0px; border-collapse: collapse; border-left: 1px solid #ccc; border-right: 1px solid #ccc; border-top: 0; border-bottom: 0; padding: 0 15px 0 16px; background-color: #fff; padding-top: 10px; padding-bottom: 5px">
<table class="keyvalue-table" style="border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt">
<tr>
<th style="color: #707070; font: normal 14px/20px Arial, sans-serif; text-align: left; vertical-align: top; padding: 2px 0">
Issue Type:
</th>
<td class="has-icon" style="padding: 0px; border-collapse: collapse; font: normal 14px/20px Arial, sans-serif; padding: 2px 0 2px 5px; vertical-align: top"> <img src="cid:jira-generated-image-avatar-fd7556c6-2b5e-4d4d-a52d-1fa57bf72299" height="16" width="16" border="0" align="absmiddle" alt="Bug" style="vertical-align: text-bottom" /> Bug
</td>
</tr>
<tr>
<th style="color: #707070; font: normal 14px/20px Arial, sans-serif; text-align: left; vertical-align: top; padding: 2px 0">
Affects Versions:
</th>
<td style="padding: 0px; border-collapse: collapse; font: normal 14px/20px Arial, sans-serif; padding: 2px 0 2px 5px; vertical-align: top">
2.6.0
</td>
</tr>
<tr>
<th style="color: #707070; font: normal 14px/20px Arial, sans-serif; text-align: left; vertical-align: top; padding: 2px 0">
Assignee:
</th>
<td style="padding: 0px; border-collapse: collapse; font: normal 14px/20px Arial, sans-serif; padding: 2px 0 2px 5px; vertical-align: top"> <a class="user-hover" rel="cantor.2@osu.edu" id="email_cantor.2@osu.edu" href="https://issues.shibboleth.net/jira/secure/ViewProfile.jspa?name=cantor.2%40osu.edu" style="color:#3b73af;; color: #3b73af; text-decoration: none">Scott Cantor</a>
</td>
</tr>
<tr>
<th style="color: #707070; font: normal 14px/20px Arial, sans-serif; text-align: left; vertical-align: top; padding: 2px 0">
Components:
</th>
<td style="padding: 0px; border-collapse: collapse; font: normal 14px/20px Arial, sans-serif; padding: 2px 0 2px 5px; vertical-align: top">
Logging
</td>
</tr>
<tr>
<th style="color: #707070; font: normal 14px/20px Arial, sans-serif; text-align: left; vertical-align: top; padding: 2px 0">
Created:
</th>
<td style="padding: 0px; border-collapse: collapse; font: normal 14px/20px Arial, sans-serif; padding: 2px 0 2px 5px; vertical-align: top">
07/Oct/16 4:41 AM
</td>
</tr>
<tr>
<th style="color: #707070; font: normal 14px/20px Arial, sans-serif; text-align: left; vertical-align: top; padding: 2px 0">
Labels:
</th>
<td style="padding: 0px; border-collapse: collapse; font: normal 14px/20px Arial, sans-serif; padding: 2px 0 2px 5px; vertical-align: top">
log4shib
</td>
</tr>
<tr>
<th style="color: #707070; font: normal 14px/20px Arial, sans-serif; text-align: left; vertical-align: top; padding: 2px 0">
Priority:
</th>
<td class="has-icon" style="padding: 0px; border-collapse: collapse; font: normal 14px/20px Arial, sans-serif; padding: 2px 0 2px 5px; vertical-align: top"> <img src="cid:jira-generated-image-static-major-d53cde8e-02af-435e-845a-2846d61f7d1d" height="16" width="16" border="0" align="absmiddle" alt="Major" style="vertical-align: text-bottom" /> Major
</td>
</tr>
<tr>
<th style="color: #707070; font: normal 14px/20px Arial, sans-serif; text-align: left; vertical-align: top; padding: 2px 0">
Reporter:
</th>
<td style="padding: 0px; border-collapse: collapse; font: normal 14px/20px Arial, sans-serif; padding: 2px 0 2px 5px; vertical-align: top"> <a class="user-hover" rel="vtsji@unitedid.org" id="email_vtsji@unitedid.org" href="https://issues.shibboleth.net/jira/secure/ViewProfile.jspa?name=vtsji%40unitedid.org" style="color:#3b73af;; color: #3b73af; text-decoration: none">vtsji@unitedid.org</a>
</td>
</tr>
</table>
</td>
</tr>
<tr>
<td class="email-content-main mobile-expand issue-description-container" style="padding: 0px; border-collapse: collapse; border-left: 1px solid #ccc; border-right: 1px solid #ccc; border-top: 0; border-bottom: 0; padding: 0 15px 0 16px; background-color: #fff; padding-top: 5px; padding-bottom: 10px">
<table class="text-paragraph-pattern" cellspacing="0" cellpadding="0" border="0" width="100%" style="border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; mso-line-height-rule: exactly; mso-text-raise: 2px">
<tr>
<td class="text-paragraph-pattern-container mobile-resize-text " style="padding: 0px; border-collapse: collapse; padding: 0 0 10px 0">
<p style="margin-top:0;margin-bottom:10px;; margin: 10px 0 0 0">As outlined in the <a href="https://wiki.shibboleth.net/confluence/display/SHIB2/NativeSPLogRotation" class="external-link" rel="nofollow" style="color: #3b73af; text-decoration: none">NativeSPLogRotation</a> page, shibd can be told to reload its log files by rotating them and then touching shibboleth2.xml. On the next request that arrives, this will then cause shibd to reload shibboleth2.xml and its log files.</p>
<p style="margin-top:0;margin-bottom:10px;; margin: 10px 0 0 0">(It took me a while to figure out that the file handles will only be reloaded on the next request that comes in, maybe this could be documented?)</p>
<p style="margin-top:0;margin-bottom:10px;; margin: 10px 0 0 0">Unfortunately it looks like the issue from <a href="https://issues.shibboleth.net/jira/browse/SSPCPP-550" title="Problems with native.log file rotation" class="issue-link" data-issue-key="SSPCPP-550" style="color: #3b73af; text-decoration: none"></a></p>
<del>
SSPCPP-550
</del> is still in play here: any appender that is not attached to the root category does not get closed and leaves leftover file handles, eventually causing shibd to run into the open file handle limit. This is a problem for default Shibboleth installations since shibd.logger includes two such categories, tran_log and sig_log.
<p style="margin-top:0;margin-bottom:10px;; margin: 10px 0 0 0">Logrotate.conf sample:</p>
<div class="code panel" style="border-width: 1px;; border: 1px solid #ccc; background: #f5f5f5; font-size: 12px; line-height: 1.333; font-family: monospace; border: 1px solid #ccc; -moz-border-radius: 3px 3px 3px 3px; border-radius: 3px 3px 3px 3px; margin: 9px 0">
<div class="codeContent panelContent" style="padding: 9px 12px">
<pre class="code-java" style="margin: 10px 0 0 0; max-height: 30em; overflow: auto; white-space: pre-wrap; word-wrap: normal">
/home/vmuser/shibsp-build/install/<span class="code-keyword" style="color: #000091">var</span>/log/shibboleth/*.log {
daily
missingok
rotate 30
compress
delaycompress
ifempty
create 640 root root
sharedscripts
postrotate
touch /home/vmuser/shibsp-build/install/etc/shibboleth/shibboleth2.xml
endscript
}
</pre>
</div>
</div>
<p style="margin-top:0;margin-bottom:10px;; margin: 10px 0 0 0">Before logrotate:</p>
<div class="code panel" style="border-width: 1px;; border: 1px solid #ccc; background: #f5f5f5; font-size: 12px; line-height: 1.333; font-family: monospace; border: 1px solid #ccc; -moz-border-radius: 3px 3px 3px 3px; border-radius: 3px 3px 3px 3px; margin: 9px 0">
<div class="codeContent panelContent" style="padding: 9px 12px">
<pre class="code-java" style="margin: 10px 0 0 0; max-height: 30em; overflow: auto; white-space: pre-wrap; word-wrap: normal">
root@vm:/home/vmuser/shibsp-build/install/<span class="code-keyword" style="color: #000091">var</span>/log/shibboleth# lsof -p $(pgrep shibd) | grep '<span class="code-keyword" style="color: #000091">var</span>/log'
shibd 53436 root 5w REG 252,0 16076 917541 /home/vmuser/shibsp-build/install/<span class="code-keyword" style="color: #000091">var</span>/log/shibboleth/shibd.log
shibd 53436 root 6w REG 252,0 0 921702 /home/vmuser/shibsp-build/install/<span class="code-keyword" style="color: #000091">var</span>/log/shibboleth/signature.log
shibd 53436 root 7w REG 252,0 0 921703 /home/vmuser/shibsp-build/install/<span class="code-keyword" style="color: #000091">var</span>/log/shibboleth/transaction.log
shibd 53436 root 8w REG 252,0 307 921704 /home/vmuser/shibsp-build/install/<span class="code-keyword" style="color: #000091">var</span>/log/shibboleth/shibd_warn.log
</pre>
</div>
</div>
<p style="margin-top:0;margin-bottom:10px;; margin: 10px 0 0 0">After logrotate + at least one request to shibd:</p>
<div class="code panel" style="border-width: 1px;; border: 1px solid #ccc; background: #f5f5f5; font-size: 12px; line-height: 1.333; font-family: monospace; border: 1px solid #ccc; -moz-border-radius: 3px 3px 3px 3px; border-radius: 3px 3px 3px 3px; margin: 9px 0">
<div class="codeContent panelContent" style="padding: 9px 12px">
<pre class="code-java" style="margin: 10px 0 0 0; max-height: 30em; overflow: auto; white-space: pre-wrap; word-wrap: normal">
shibd 53564 root 6w REG 252,0 0 921702 /home/vmuser/shibsp-build/install/<span class="code-keyword" style="color: #000091">var</span>/log/shibboleth/signature.log.1
shibd 53564 root 7w REG 252,0 657 921703 /home/vmuser/shibsp-build/install/<span class="code-keyword" style="color: #000091">var</span>/log/shibboleth/transaction.log.1
shibd 53564 root 12w REG 252,0 14102 921706 /home/vmuser/shibsp-build/install/<span class="code-keyword" style="color: #000091">var</span>/log/shibboleth/shibd.log
shibd 53564 root 13w REG 252,0 0 921708 /home/vmuser/shibsp-build/install/<span class="code-keyword" style="color: #000091">var</span>/log/shibboleth/signature.log
shibd 53564 root 14w REG 252,0 657 921709 /home/vmuser/shibsp-build/install/<span class="code-keyword" style="color: #000091">var</span>/log/shibboleth/transaction.log
shibd 53564 root 15w REG 252,0 297 921707 /home/vmuser/shibsp-build/install/<span class="code-keyword" style="color: #000091">var</span>/log/shibboleth/shibd_warn.log
</pre>
</div>
</div>
<p style="margin-top:0;margin-bottom:10px;; margin: 10px 0 0 0">Note that shibd.log and shibd_warn.log were properly reloaded, but the old file handles for signature.log and transaction.log were left dangling.</p>
<p style="margin-top:0;margin-bottom:10px;; margin: 10px 0 0 0">I tested <a href="https://issues.shibboleth.net/jira/browse/SSPCPP-550" title="Problems with native.log file rotation" class="issue-link" data-issue-key="SSPCPP-550" style="color: #3b73af; text-decoration: none"></a></p>
<del>
SSPCPP-550
</del>'s suggestion of inserting a call to Category::shutdown() in XMLToolingConfig::log_config(), but that makes no difference since it's calling the same removeAllAppenders() cleanup function that was already getting called anyway when individual categories get reloaded.
<p style="margin-top:0;margin-bottom:10px;; margin: 10px 0 0 0">The root cause here seems to be that there is no way in log4cpp to actually clean up any appenders, as they are all attached as "non-owning" to their categories and therefore never get cleaned up when a category is reloaded or deleted. Presumably this is done to avoid double deletes when an appender is used by multiple categories?</p>
<p style="margin-top:0;margin-bottom:10px;; margin: 10px 0 0 0"><a href="https://issues.shibboleth.net/jira/browse/SSPCPP-550" title="Problems with native.log file rotation" class="issue-link" data-issue-key="SSPCPP-550" style="color: #3b73af; text-decoration: none"></a></p>
<del>
SSPCPP-550
</del> improves the situation for log4shib by making the root category own its appenders (and thus clean them up on reload), but the issue remains for non-root ones.
<p style="margin-top:0;margin-bottom:10px;; margin: 10px 0 0 0">The only proper fix I see is for log4cpp to fix their Appender pointer ownership issues so that Category::shutdown() can work properly. In the meantime, the workaround for logrotating shibd logs is to use copytruncate so file handles can stay the same, and not touch shibboleth2.xml in the postrotate script since there is no more need to.</p>
</td>
</tr>
</table>
</td>
</tr>
<tr>
<td class="email-content-main mobile-expand " style="padding: 0px; border-collapse: collapse; border-left: 1px solid #ccc; border-right: 1px solid #ccc; border-top: 0; border-bottom: 0; padding: 0 15px 0 16px; background-color: #fff">
<table id="actions-pattern" cellspacing="0" cellpadding="0" border="0" width="100%" style="border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; mso-line-height-rule: exactly; mso-text-raise: 1px">
<tr>
<td id="actions-pattern-container" valign="middle" style="padding: 0px; border-collapse: collapse; padding: 10px 0 10px 24px; vertical-align: middle; padding-left: 0">
<table align="left" style="border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt">
<tr>
<td class="actions-pattern-action-icon-container" style="padding: 0px; border-collapse: collapse; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; mso-line-height-rule: exactly; mso-text-raise: 0px; vertical-align: middle"> <a href="https://issues.shibboleth.net/jira/browse/SSPCPP-712#add-comment" target="_blank" title="Add Comment" style="color: #3b73af; text-decoration: none"> <img class="actions-pattern-action-icon-image" src="cid:jira-generated-image-static-comment-icon-4c70847e-11b6-4a66-8a73-3c61cbf1cb58" alt="Add Comment" title="Add Comment" height="16" width="16" border="0" style="vertical-align: middle" /> </a>
</td>
<td class="actions-pattern-action-text-container" style="padding: 0px; border-collapse: collapse; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; mso-line-height-rule: exactly; mso-text-raise: 4px; padding-left: 5px"> <a href="https://issues.shibboleth.net/jira/browse/SSPCPP-712#add-comment" target="_blank" title="Add Comment" style="color: #3b73af; text-decoration: none">Add Comment</a>
</td>
</tr>
</table>
</td>
</tr>
</table>
</td>
</tr>
<!-- there needs to be content in the cell for it to render in some clients -->
<tr>
<td class="email-content-rounded-bottom mobile-expand" style="padding: 0px; border-collapse: collapse; color: #fff; padding: 0 15px 0 16px; height: 5px; line-height: 5px; background-color: #fff; border-top: 0; border-left: 1px solid #ccc; border-bottom: 1px solid #ccc; border-right: 1px solid #ccc; border-bottom-right-radius: 5px; border-bottom-left-radius: 5px; mso-line-height-rule: exactly">
</td>
</tr>
</table>
</td>
</tr>
<tr>
<td id="footer-pattern" style="padding: 0px; border-collapse: collapse; padding: 12px 20px">
<table id="footer-pattern-container" cellspacing="0" cellpadding="0" border="0" style="border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt">
<tr>
<td id="footer-pattern-text" class="mobile-resize-text" width="100%" style="padding: 0px; border-collapse: collapse; color: #999; font-size: 12px; line-height: 18px; font-family: Arial, sans-serif; mso-line-height-rule: exactly; mso-text-raise: 2px">
This message was sent by Atlassian JIRA <span id="footer-build-information">(v7.0.10#70120-<span title="37e3d7a6fc4d580639533e7f7c232c925e554a6a" data-commit-id="37e3d7a6fc4d580639533e7f7c232c925e554a6a}">sha1:37e3d7a</span>)</span>
</td>
<td id="footer-pattern-logo-desktop-container" valign="top" style="padding: 0px; border-collapse: collapse; padding-left: 20px; vertical-align: top">
<table style="border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt">
<tr>
<td id="footer-pattern-logo-desktop-padding" style="padding: 0px; border-collapse: collapse; padding-top: 3px"> <img id="footer-pattern-logo-desktop" src="cid:jira-generated-image-static-footer-desktop-logo-53764ad2-967b-4b50-bc4d-829a81869606" alt="Atlassian logo" title="Atlassian logo" width="169" height="36" class="image_fix" />
</td>
</tr>
</table>
</td>
</tr>
</table>
</td>
</tr>
</table>
</body>
</html>