Stored persistent ID and migration to 3.2

[Chris Phillips]

I think I'm close to narrowing this down but still need guidance.

Note this is a longer post and has a stack trace.
Hopefully working this out on the shib-users list publicly may help
others..

Recap of desired outcome:
Appropriate and safe population of eduPersonTargetedId from a DB
connection that will not issue WARN conditions on the database in IdP
3.2.0.



As Scott commented in this thread earlier, there WAS another datasource
property in play in attribute-filter.xml which was/is for
eduPersonTargetedId generation/storage in a mysql DB using the StoredId
connector approach.

First question: 
That connector has no ability to handle the retry conditions and there's
no way to configure it to in attribute-resolver.xml -- or is there?

I assume there isn't.
Given that, I looked at attempting to re-use the bean references in
saml-nameid.xml in attribute-resolver.xml and they were not available. I
assume it's a sequence of events thing where the beans in saml-nameid.xml
are instantiated AFTER the resolver is in place.


Since I encountered that problem, I took the approach inspired by the
StorageConfiguration[1] with the technique from
PersistentNameIDGenerationConfiguration[2] persistentID Store, and created
this in global.xml thinking I could create a global datasource connector
just like the one in saml-nameid.xml to handle the extra retryableErrors
conditions but available more globally:

In global.xml:
--begin--
 <!-- BaseGlobalDataSource -->

<bean id="BaseGlobalDataSource"
class="org.apache.commons.dbcp2.BasicDataSource"
    p:driverClassName="com.mysql.jdbc.Driver"
    p:url="jdbc:mysql://127.0.0.1:3306/shibboleth"
    p:username="shibboleth"
    p:password="SUPPRESED"
    p:maxIdle="5"
    p:maxWaitMillis="15000"
    p:testOnBorrow="true"
    p:validationQuery="select 1"
    p:validationQueryTimeout="5" />

  <!-- Parent bean for users to configure a custom ID store rather than a
data source only. -->
    <bean id="custom.JDBCPersistentParentStore" abstract="true"
        
class="net.shibboleth.idp.saml.nameid.impl.JDBCPersistentIdStoreEx" />

<!-- A "store" bean suitable for use  to avoid the JDBC errors -->
<bean id="MyGlobalDataSource" parent="custom.JDBCPersistentParentStore"
    p:dataSource-ref="BaseGlobalDataSource"
    p:queryTimeout="PT2S"
    p:retryableErrors="#{{'23000'}}" />



--end--

Which I then refer to in attribute-resolver.xml that I can then refer to
via a BeanManaged Connection:

<resolver:DataConnector id="StoredId" xsi:type="StoredId"
xmlns="urn:mace:shibboleth:2.0:resolver:dc"
            generatedAttributeID="persistentId" sourceAttributeID="uid"
salt="SUPPRESSED">
	<resolver:Dependency ref="uid" />
	<dc:BeanManagedConnection>MyGlobalDataSource</dc:BeanManagedConnection>
    </resolver:DataConnector>



However, what I get is the following stack trace saying that the the
Beantype is not the right type and yet this technique works in
saml-nameid.xml:


2015-12-09 15:23:21,002 - WARN
[net.shibboleth.ext.spring.context.FilesystemGenericApplicationContext:545]
 - Exception encountered during context initialization - cancelling
refresh attempt: org.springframework.beans.factory.BeanCreationException:
Error creating bean with name 'StoredId': Initialization of bean failed;
nested exception is
org.springframework.beans.ConversionNotSupportedException: Failed to
convert property value of type
'net.shibboleth.idp.saml.nameid.impl.JDBCPersistentIdStoreEx' to required
type 'javax.sql.DataSource' for property 'dataSource'; nested exception is
java.lang.IllegalStateException: Cannot convert value of type
[net.shibboleth.idp.saml.nameid.impl.JDBCPersistentIdStoreEx] to required
type [javax.sql.DataSource] for property 'dataSource': no matching editors
or conversion strategy found
2015-12-09 15:23:21,038 - ERROR
[net.shibboleth.utilities.java.support.service.AbstractReloadableService:18
1] - Service 'shibboleth.AttributeResolverService': Initial load failed
net.shibboleth.utilities.java.support.service.ServiceException:
org.springframework.beans.factory.BeanCreationException: Error creating
bean with name 'StoredId': Initialization of bean failed; nested exception
is org.springframework.beans.ConversionNotSupportedException: Failed to
convert property value of type
'net.shibboleth.idp.saml.nameid.impl.JDBCPersistentIdStoreEx' to required
type 'javax.sql.DataSource' for property 'dataSource'; nested exception is
java.lang.IllegalStateException: Cannot convert value of type
[net.shibboleth.idp.saml.nameid.impl.JDBCPersistentIdStoreEx] to required
type [javax.sql.DataSource] for property 'dataSource': no matching editors
or conversion strategy found
        at 
net.shibboleth.ext.spring.service.ReloadableSpringService.doReload(Reloadab
leSpringService.java:334)



So, at this point I'm wondering:

A. How StorageConfiguration[1] escapes the problem of the retry scenario?
If I can't do this bean approach to capture retry conditions is the
StorageConfiguration going to suffer the same risk as the saml-nameid.xml
connection pool, and if so, how would one deal with that? Is this a bug
yet to be encountered by the JPA/global.xml storage?

B. What is the recommended way to handle database connections in
attribute-resolver.xml in Shib 3.2.0 that need to trap the retry scenario
and avoid the WARN statement?

C. If the attribute-resolver.xml can properly route the population of
ePTId to the saml-nameid.xml configuration where it may (should?) reside
what is the recommended way to do that?

Thanks for thoughts and comments.

C


[1] 
https://wiki.shibboleth.net/confluence/display/IDP30/StorageConfiguration
[2] 
https://wiki.shibboleth.net/confluence/display/IDP30/PersistentNameIDGenera
tionConfiguration




On 2015-12-07, 5:18 PM, "users on behalf of Cantor, Scott"
<users-bounces at shibboleth.net on behalf of cantor.2 at osu.edu> wrote:

>On 12/7/15, 5:07 PM, "users on behalf of Chris Phillips"
><users-bounces at shibboleth.net on behalf of Chris.Phillips at canarie.ca>
>wrote:
>
>
>
>>In saml-nameid.properties I set idp.persistentId.store =
>>MyPersistentIdStore
>
>And the other/new dataSource property isn't set?
>
>>I've watched the mysql general_logs table to see what's flowing and it's
>>not clear how the 23000 error determination is happening.
>>I see the dummy insertion and deletion at the end.
>
>There's a second insert done and that either succeeds (which means
>there's no key defined), or it fails and it checks if the SQLState value
>coming back is configured in the retryableErrors collection. You should
>have that logged one way or the other. Not just an insert and delete.
>It's two inserts and then a delete.
>
>>It's like retryableErrors is not being trapped/observed and would like a
>>recommendation on how to chase this down further.
>
>Well, if it's not configuring that, the config isn't what you think it
>is. There's no way I know of to debug that other than an actual debugger.
>I'd also make sure there aren't any other log messages suggesting it's
>generating some kind of compatibility config because of older settings.
>
>The exception handling code's quite simple:
>
>if (!retryableErrors.contains(e.getSQLState())) {
>   log.warn("{} Duplicate insert failed as required with SQL State '{}',
>ensure this value is "
>      + "configured as a retryable error", getLogPrefix(),
>e.getSQLState());
>            }
>
>-- Scott
>
>
>-- 
>To unsubscribe from this list send an email to
>users-unsubscribe at shibboleth.net