View error about "can only resume paused view states" is occurring intermittently during SSO

[Koeritz, Chris (cak0l)]

Dear Shibboleth Gurus,
We are experiencing a transient problem that causes a fairly gross error to show up for people during login.
An example of the error message shown to the user is attached below, but the upshot seems to be this message:
"You can only resume paused view states, and state {…details...} is not a view state - programmer error” 
So far, we cannot easily reproduce this error, and are somewhat baffled by its cause(s).
The error has been occurring for quite a while.  We were previously running Shibboleth IdP v3.4.6, but upgraded to 4.1.6 several months ago.  The error is still occurring on v4.1.6.
The problem does not happen with great regularity, and people whom it affects are able to log in again right afterwards with no problem.
The error occurs most frequently (and perhaps exclusively) on our production IdP cluster.  We have four production nodes in the cluster, load balanced by an F5 (with sticky sessions).
I am attaching some logging (pastebin) from the most recent occurrence of the error that we know of.
Can you folks help point us in the right direction for tracking down this problem?  99.9% of SSO requests seem to be fine... but the people who encounter this error are likely to start a help desk ticket when they see this, and we’d like to shut down this problem if at all possible.
I can provide more details about our configuration as needed.
Thanks,
Chris Koeritz

idp-process.log with the activity for the user around the time of the error:
https://pastebin.com/QS3S28V3 <https://pastebin.com/QS3S28V3>

jetty access.log with activity from the same time frame:
https://pastebin.com/AyZKYsd8 <https://pastebin.com/AyZKYsd8>

the visible error message:
(Note: Netbadge is just the University of Virginia’s branded SSO, but it is Shibboleth IdP implementing the SSO.)

Netbadge Message
Uncaught Exception

A software error was encountered that prevents normal operation:
java.lang.RuntimeException: java.lang.IllegalStateException: You can only resume paused view states, and state [EndState at 39535c9b id = 'ErrorView', flow = 'SAML2/Redirect/SSO', entryActionList = list[[AnnotatedAction at 4806b626 targetAction = [EvaluateAction at 726f85c9 expression = environment, resultExpression = requestScope.environment], attributes = map[[empty]]], [AnnotatedAction at 42886a17 targetAction = [EvaluateAction at 619c5159 expression = opensamlProfileRequestContext, resultExpression = requestScope.profileRequestContext], attributes = map[[empty]]], [AnnotatedAction at 1cd67cca targetAction = [EvaluateAction at 1299f354 expression = T(net.shibboleth.utilities.java.support.codec.HTMLEncoder), resultExpression = requestScope.encoder], attributes = map[[empty]]], [AnnotatedAction at 1340fbc2 targetAction = [EvaluateAction at 1055766a expression = flowRequestContext.getExternalContext().getNativeRequest(), resultExpression = requestScope.request], attributes = map[[empty]]], [AnnotatedAction at 3fca6886 targetAction = [EvaluateAction at 62142f78 expression = flowRequestContext.getExternalContext().getNativeResponse(), resultExpression = requestScope.response], attributes = map[[empty]]], [AnnotatedAction at 769e5613 targetAction = [EvaluateAction at 4dc3988c expression = flowRequestContext.getActiveFlow().getApplicationContext().containsBean('shibboleth.CustomViewContext') ? flowRequestContext.getActiveFlow().getApplicationContext().getBean('shibboleth.CustomViewContext') : null, resultExpression = requestScope.custom], attributes = map[[empty]]]], exceptionHandlerSet = list[net.shibboleth.idp.profile.impl.RethrowingFlowExecutionExceptionHandler at 53e2d181], finalResponseAction = org.springframework.webflow.action.ViewFactoryActionAdapter at 58011f46, outputMapper = [null]] is not a view state - programmer error

Please report this problem to your Help Desk or administrative staff. It has also been logged for an administrator to review.

-- 
Chris Koeritz
Senior Linux and Storage Engineer 
University of Virginia ITS - Backup, Storage, and Archive Services
P: 434 982 4690
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://shibboleth.net/pipermail/users/attachments/20221014/943909e4/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 3359 bytes
Desc: not available
URL: <http://shibboleth.net/pipermail/users/attachments/20221014/943909e4/attachment.p7s>