Best debug logging to enable

[Dan McLaughlin]

What's the best logging to enable to debug the following error?

2018-09-06 10:25:42 DEBUG OpenSAML.SecurityPolicyRule.XMLSigning [1]
[default]: validating signature profile
2018-09-06 10:25:42 DEBUG XMLTooling.CredentialCriteria [1] [default]:
keys didn't match
2018-09-06 10:25:42 DEBUG XMLTooling.TrustEngine.ExplicitKey [1]
[default]: unable to validate signature, no credentials available from
peer
2018-09-06 10:25:42 DEBUG XMLTooling.TrustEngine.PKIX [1] [default]:
validating signature using certificate from within the signature
2018-09-06 10:25:42 DEBUG XMLTooling.TrustEngine.PKIX [1] [default]:
Digital signature does not validate with the supplied key.
2018-09-06 10:25:42 DEBUG XMLTooling.TrustEngine.PKIX [1] [default]:
failed to verify signature with embedded certificates
2018-09-06 10:25:42 ERROR OpenSAML.SecurityPolicyRule.XMLSigning [1]
[default]: unable to verify message signature with supplied trust
engine
2018-09-06 10:25:42 WARN Shibboleth.SSO.SAML2 [1] [default]: error
processing incoming assertion: Message was signed, but signature could
not be verified.

We are running SP 3.0.2 on Apache 2.4. We have an integration with an
IDP using F5 BIG-IP APM and everything was working fine for over a
year, then they upgraded to the 12.3.6 release of the BIG-IP APM
software and now we can't get past this issue.  We've tried
everything, double-checked their certs in their metadata, and
everything looks good.   They are using a wildcard certificate signed
by DigiCert for both their signing and encryption cert.  One thing
I've noticed is that the in the SAML 2 response from their IDP there
are two certificates, the first is the Digicert CA Public cert, the
second is their wildcard public cert.  The metadata they sent us (auto
generated by the F5 device) only contains the public key for their
wildcard certificate that's signed by the DigiCert CA.  Is it possible
that their metadata should also contain the DigiCert CA public cert or
a certificate chain?   This is our only integration with someone using
a CA signed certificate, and also the only one using a wild card
cert...most use self-signed certs.   One thing to note is before they
upgraded their IDP, their metadata was the same, it only included a
signal wildcard cert that was signed by DigiCert.  Unfortunately I
don't have debug logs from before they upgraded to see if the SAML 2
response contained the CA cert before, so I'm not sure if this
behavior has anything to do with the issue.

Questions 1) Should wildcard certs coming from an IDP work for signing
and encryption? They worked before, so I'm assuming the answer is yes.
Question 2) Is it normal for the SAML 2 response to contain the CA's
public key that signed their signing/encryption certificate?

What else should we be looking at and is there any additional logging
we can enable to help get to the bottom of the issue?

--

Thanks,

Dan