<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1.0">
<base href="https://issues.shibboleth.net/jira">
<title>Message Title</title>
</head>
<body class="jira" style="color: #333333; font-family: Arial, sans-serif; font-size: 14px; line-height: 1.429">
<table id="background-table" cellpadding="0" cellspacing="0" width="100%" style="border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt; background-color: #f5f5f5; border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt" bgcolor="#f5f5f5">
<!-- header here -->
<tbody>
<tr>
<td id="header-pattern-container" style="padding: 0px; border-collapse: collapse; padding: 10px 20px">
<table id="header-pattern" cellspacing="0" cellpadding="0" border="0" style="border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt">
<tbody>
<tr>
<td id="header-avatar-image-container" valign="top" style="padding: 0px; border-collapse: collapse; vertical-align: top; width: 32px; padding-right: 8px" width="32"> <img id="header-avatar-image" class="image_fix" src="cid:jira-generated-image-avatar-f686528d-e897-4d23-a51b-307dd328cdb1" height="32" width="32" border="0" style="border-radius: 3px; vertical-align: top"> </td>
<td id="header-text-container" valign="middle" style="padding: 0px; border-collapse: collapse; vertical-align: middle; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; mso-line-height-rule: exactly; mso-text-raise: 1px"> <a class="user-hover" rel="ian@iay.org.uk" id="email_ian@iay.org.uk" href="https://issues.shibboleth.net/jira/secure/ViewProfile.jspa?name=ian%40iay.org.uk" style="color:#0052cc;; color: #3b73af; text-decoration: none">Ian Young</a> <strong>commented</strong> on <a href="https://issues.shibboleth.net/jira/browse/JPAR-142" style="color: #3b73af; text-decoration: none"><img src="cid:jira-generated-image-avatar-cba39009-ae1c-4317-ba86-4157e45e9f02" height="16" width="16" border="0" align="absmiddle" alt="Task"> JPAR-142</a> </td>
</tr>
</tbody>
</table> </td>
</tr>
<tr>
<td id="email-content-container" style="padding: 0px; border-collapse: collapse; padding: 0 20px">
<table id="email-content-table" cellspacing="0" cellpadding="0" border="0" width="100%" style="border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt; border-spacing: 0; border-collapse: separate">
<tbody>
<tr>
<!-- there needs to be content in the cell for it to render in some clients -->
<td class="email-content-rounded-top mobile-expand" style="padding: 0px; border-collapse: collapse; color: #ffffff; padding: 0 15px 0 16px; height: 15px; background-color: #ffffff; border-left: 1px solid #cccccc; border-top: 1px solid #cccccc; border-right: 1px solid #cccccc; border-bottom: 0; border-top-right-radius: 5px; border-top-left-radius: 5px; height: 10px; line-height: 10px; padding: 0 15px 0 16px; mso-line-height-rule: exactly" height="10" bgcolor="#ffffff"> </td>
</tr>
<tr>
<td class="email-content-main mobile-expand " style="padding: 0px; border-collapse: collapse; border-left: 1px solid #cccccc; border-right: 1px solid #cccccc; border-top: 0; border-bottom: 0; padding: 0 15px 0 16px; background-color: #ffffff" bgcolor="#ffffff">
<table class="page-title-pattern" cellspacing="0" cellpadding="0" border="0" width="100%" style="border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt">
<tbody>
<tr>
<td style="vertical-align: top;; padding: 0px; border-collapse: collapse; padding-right: 5px; font-size: 20px; line-height: 30px; mso-line-height-rule: exactly" class="page-title-pattern-header-container"> <span class="page-title-pattern-header" style="font-family: Arial, sans-serif; padding: 0; font-size: 20px; line-height: 30px; mso-text-raise: 2px; mso-line-height-rule: exactly; vertical-align: middle"> <a href="https://issues.shibboleth.net/jira/browse/JPAR-142" style="color: #3b73af; text-decoration: none">Re: Smoke test Debian 10 "buster"</a> </span> </td>
</tr>
</tbody>
</table> </td>
</tr>
<tr>
<td id="text-paragraph-pattern-top" class="email-content-main mobile-expand comment-top-pattern" style="padding: 0px; border-collapse: collapse; border-left: 1px solid #cccccc; border-right: 1px solid #cccccc; border-top: 0; border-bottom: 0; padding: 0 15px 0 16px; background-color: #ffffff; border-bottom: none; padding-bottom: 0" bgcolor="#ffffff">
<table class="text-paragraph-pattern" cellspacing="0" cellpadding="0" border="0" width="100%" style="border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; mso-line-height-rule: exactly; mso-text-raise: 2px">
<tbody>
<tr>
<td class="text-paragraph-pattern-container mobile-resize-text " style="padding: 0px; border-collapse: collapse; padding: 0 0 10px 0"> <p style="margin: 10px 0 0 0; margin-top: 0">TL;DR: The problem turns out to be that the IdP instances run up for each test were blocking waiting for Java to acquire random bits from the kernel. Making more random bits available allowed the tests to run to completion without failures.</p> <p style="margin: 10px 0 0 0">Takeaways: this might turn out to be an issue for people deploying under Debian 10 regardless of the version of Java in use. It will be particularly likely to affect deployments in virtual machine environments other than KVM, for example under VMware. Some cloud providers may be affected even if they are using a KVM derivative. It may be that real deployers won't notice this issue, as in general they won't be firing up multiple JVMs each of which will be initialising random streams – but perhaps they will.</p> <p style="margin: 10px 0 0 0">Details:</p> <p style="margin: 10px 0 0 0">There's a long-running religious war about how the kernel should acquire and dispense randomness. The exact details vary between kernels and between distributions. Some systems take the view that once "enough" entropy has been (estimated to be) collected, there's no further need to block when cryptographic randomness is acquired. Others attempt to track the amount of entropy remaining in their internal pool (you can see their current estimate in <tt>/proc/sys/kernel/random/entropy_avail</tt>) and will block until "enough" entropy is (estimated to be) available again.</p> <p style="margin: 10px 0 0 0">It looks like Debian 10 has changed its policy on when to block and when not to block, relative to Debian 9 and to other Linux distributions. It's not clear to me whether they are just ahead of the curve and other distributions can be expected to go down the same path, or whether Debian will continue to be an outlier.</p> <p style="margin: 10px 0 0 0">For our integration tests, we spin up a new Java process with an IdP inside it for each test method, which is to say very often. Each such process demands cryptographic random bits from the kernel. If the kernel thinks that it has enough entropy available, that request will succeed quickly and the test will pass. If the "available" entropy has been depleted by previous test methods, however, the kernel will block the subprocess until more entropy becomes available, which may take a long time. I found that increasing the timeout from 60 seconds to 120 drastically reduced but did not eliminate failures. I was seeing about 70 to 90 seconds delay for process startup in most cases.</p> <p style="margin: 10px 0 0 0">The Debian 10 release notes talk about entropy starvation here:</p> <p style="margin: 10px 0 0 0"><a href="https://www.debian.org/releases/stable/amd64/release-notes/ch-information.en.html#entropy-starvation" class="external-link" rel="nofollow" style="color: #3b73af; text-decoration: none">https://www.debian.org/releases/stable/amd64/release-notes/ch-information.en.html#entropy-starvation</a></p> <p style="margin: 10px 0 0 0">That discussion is in the context of it taking many minutes or even hours for <tt>ssh</tt> to be available on newly booted Debian 10 systems, particularly for the first boot case where keys must be generated. The "ping the system a lot" suggestion actually does help a little bit.</p> <p style="margin: 10px 0 0 0">Other articles on the issue, and the more general religious war involved:</p>
<ul>
<li><a href="https://daniel-lange.com/archives/152-hello-buster.html" class="external-link" rel="nofollow" style="color: #3b73af; text-decoration: none">https://daniel-lange.com/archives/152-hello-buster.html</a></li>
<li><a href="https://www.2uo.de/myths-about-urandom/" class="external-link" rel="nofollow" style="color: #3b73af; text-decoration: none">https://www.2uo.de/myths-about-urandom/</a></li>
<li><a href="https://lists.debian.org/debian-devel/2019/01/msg00167.html" class="external-link" rel="nofollow" style="color: #3b73af; text-decoration: none">https://lists.debian.org/debian-devel/2019/01/msg00167.html</a></li>
</ul> <p style="margin: 10px 0 0 0">For the purpose of running tests, there's an easy workround and that is to install the <tt>haveged</tt> package.</p> <p style="margin: 10px 0 0 0"><a href="http://issihosts.com/haveged/index.html" class="external-link" rel="nofollow" style="color: #3b73af; text-decoration: none">http://issihosts.com/haveged/index.html</a></p> <p style="margin: 10px 0 0 0">This runs an additional daemon that claims to gather entropy from instruction timing jitter and inject it into the kernel if it notices that the kernel's available entropy estimate is getting low. Given the tendency for religious wars in this area, I am unsurprised to hear that there is a lot of debate about whether this solution is appropriate for production use. My personal view is that as long as the system has acquired enough <em>initial</em> entropy that it should be fine (because it's actually unnecessary to block after that).</p> <p style="margin: 10px 0 0 0">Another approach which deployers might want to look into if they run into this is to tell Java to use the urandom device rather than the random device:</p> <p style="margin: 10px 0 0 0"><a href="https://ruleoftech.com/2016/avoiding-jvm-delays-caused-by-random-number-generation" class="external-link" rel="nofollow" style="color: #3b73af; text-decoration: none">https://ruleoftech.com/2016/avoiding-jvm-delays-caused-by-random-number-generation</a></p> <p style="margin: 10px 0 0 0">Again, I'd personally be fine with that because I don't buy the whole entropy starvation line of argument, but other people vehemently fight against avoiding the problem in this way. If the system's entropy has been depleted, they would argue, then what you're getting from urandom isn't good enough. </p> </td>
</tr>
</tbody>
</table> </td>
</tr>
<tr>
<td class="email-content-main mobile-expand " style="padding: 0px; border-collapse: collapse; border-left: 1px solid #cccccc; border-right: 1px solid #cccccc; border-top: 0; border-bottom: 0; padding: 0 15px 0 16px; background-color: #ffffff" bgcolor="#ffffff">
<table id="actions-pattern" cellspacing="0" cellpadding="0" border="0" width="100%" style="border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; mso-line-height-rule: exactly; mso-text-raise: 1px">
<tbody>
<tr>
<td id="actions-pattern-container" valign="middle" style="padding: 0px; border-collapse: collapse; padding: 10px 0 10px 24px; vertical-align: middle; padding-left: 0">
<table align="left" style="border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt">
<tbody>
<tr>
<td class="actions-pattern-action-icon-container" style="padding: 0px; border-collapse: collapse; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; mso-line-height-rule: exactly; mso-text-raise: 0; vertical-align: middle"> <a href="https://issues.shibboleth.net/jira/browse/JPAR-142#add-comment" target="_blank" title="Add Comment" style="color: #3b73af; text-decoration: none"> <img class="actions-pattern-action-icon-image" src="cid:jira-generated-image-static-comment-icon-5f773ccc-6b0f-4949-8ccf-50c1a5a90065" alt="Add Comment" title="Add Comment" height="16" width="16" border="0" style="vertical-align: middle"> </a> </td>
<td class="actions-pattern-action-text-container" style="padding: 0px; border-collapse: collapse; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; mso-line-height-rule: exactly; mso-text-raise: 4px; padding-left: 5px"> <a href="https://issues.shibboleth.net/jira/browse/JPAR-142#add-comment" target="_blank" title="Add Comment" style="color: #3b73af; text-decoration: none">Add Comment</a> </td>
</tr>
</tbody>
</table> </td>
</tr>
</tbody>
</table> </td>
</tr>
<!-- there needs to be content in the cell for it to render in some clients -->
<tr>
<td class="email-content-rounded-bottom mobile-expand" style="padding: 0px; border-collapse: collapse; color: #ffffff; padding: 0 15px 0 16px; height: 5px; line-height: 5px; background-color: #ffffff; border-top: 0; border-left: 1px solid #cccccc; border-bottom: 1px solid #cccccc; border-right: 1px solid #cccccc; border-bottom-right-radius: 5px; border-bottom-left-radius: 5px; mso-line-height-rule: exactly" height="5" bgcolor="#ffffff"> </td>
</tr>
</tbody>
</table> </td>
</tr>
<tr>
<td id="footer-pattern" style="padding: 0px; border-collapse: collapse; padding: 12px 20px">
<table id="footer-pattern-container" cellspacing="0" cellpadding="0" border="0" style="border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt">
<tbody>
<tr>
<td id="footer-pattern-text" class="mobile-resize-text" width="100%" style="padding: 0px; border-collapse: collapse; color: #999999; font-size: 12px; line-height: 18px; font-family: Arial, sans-serif; mso-line-height-rule: exactly; mso-text-raise: 2px"> This message was sent by Atlassian Jira <span id="footer-build-information">(v8.2.3#802003-<span title="5986657bbb45d2d15e7e179e83f09254924a6523" data-commit-id="5986657bbb45d2d15e7e179e83f09254924a6523}">sha1:5986657</span>)</span> </td>
<td id="footer-pattern-logo-desktop-container" valign="top" style="padding: 0px; border-collapse: collapse; padding-left: 20px; vertical-align: top">
<table style="border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt">
<tbody>
<tr>
<td id="footer-pattern-logo-desktop-padding" style="padding: 0px; border-collapse: collapse; padding-top: 3px"> <img id="footer-pattern-logo-desktop" src="https://issues.shibboleth.net/jira/images/mail/atlassian-email-logo.png" alt="Atlassian logo" title="Atlassian logo" width="191" height="24" class="image_fix"> </td>
</tr>
</tbody>
</table> </td>
</tr>
</tbody>
</table> </td>
</tr>
</tbody>
</table>
</body>
</html>