IdP 3.3.1 - mac check in GCM failed

[George Stoynev]

Well, I've run some side-by-side comparisons. Basically, I have the same 
configuration for all 4 servers in the Prod pool (only two are active 
and running at a time).

Here is how it goes:
servers 1/2 sport RHEL7 and Java 8 (Shibboleth config has been modified 
to reflect that change). They had Shibboleth IdP 3.2.1 which I have put 
on a side and did clean install of 3.3.1.
servers 3/4 are on RHEL6 and Java 7, Shibboleth 3.2.1.

Comparing the logs from version 3.2.1 (since this is the common version 
on all 4 servers) - the error messages are present in RHEL7, Java 8 
servers but not in the RHEL6, Java 7 ones. All servers have JCE Policy 
installed. We use the Oracle versions of Java, never the OpenJDK. Both 
Java 7 and 8 are on their latest versions.

I have compared back and forth, can't see any differences in the way 
Java and JCE are installed.
Moreover I have run this:
$JAVA_HOME/bin/jrunscript -e 'print 
(javax.crypto.Cipher.getMaxAllowedKeyLength("RC5") >= 256);

on all servers - it resulted in "true". I suppose this is a good sign.

Any clue of what else to look at?

Thank you,

George

On 2017-06-13 09:31 AM, users-request at shibboleth.net wrote:
>     2. Re: IdP 3.3.1 - mac check in GCM failed (Cantor, Scott)
>
>
> ----------------------------------------------------------------------
> ------------------------------
>
> Message: 2
> Date: Tue, 13 Jun 2017 13:30:54 +0000
> From: "Cantor, Scott" <cantor.2 at osu.edu>
> To: Shib Users <users at shibboleth.net>
> Subject: Re: IdP 3.3.1 - mac check in GCM failed
> Message-ID: <9C434AC1-B130-427E-B60D-1FE4E46E0317 at osu.edu>
> Content-Type: text/plain; charset="utf-8"
>
> On 6/13/17, 8:30 AM, "users on behalf of George Stoynev" <users-bounces at shibboleth.net on behalf of george.stoynev at mcgill.ca> wrote:
>
>> Interesting ... I found the following entries in the idp-process.log which may be related:
> Not directly, that case is a missing key, and that's normal, people come back days later with stale data encrypted under an older key that rolled off. All of that is tuneable, but decrypting an expired session doesn't really do anything useful anyway.
>
>> The secret key has been updated on daily basis and pushed from the first server to the second one (two servers in total). Just
>> compared the md5sums on both nodes - they match. Rerun the cron job manually and copied the files over manually as well - no
>> errors, still md5sums match. The error messages persist.
> I scanned my logs, there's no sign of that error, so I have no reason to doubt my diagnosis.
>
> If the problem had something to do with the crypto policy files in the JDK it would prevent a node from ever working. I suppose if every single decryption attempt is failing, that would fit.
>
> Otherwise, my next question would be the usual, is this OpenJDK? If it is, switch.
>
> -- Scott
>