IdPv3: race condition causes two persistentId to be generated

[Cantor, Scott]

We're starting to dig into solution analysis for this, to determine what we can do for the next release, given that we were hoping to ship it this month. After reviewing the existing code more, I wanted to note something:

On 9/23/15, 11:28 AM, "Dominique Petitpierre" <Dominique.Petitpierre at unige.ch> wrote:

>
>c) To avoid that two identical persistentId be stored (second case
>    above), may be one could rely on a database (or other storage)
>    constraint preventing duplicates, and for the second thread to
>    ignore the duplicate entry error message from the database.  (But
>    this would not work well in a load balanced asynchronously
>    replicated database cluster unless there is some kind of tolerant
>    conflict resolution).

I don't see how we can't give you a solution that allows for asynchronous replication without understanding that it will lead to problems. If two different nodes both believe they need to generate an ID (first time in, old one expired, whatever), they're both going to. That's a problem, obviously, for a lot of use cases for identifiers.

Now, we could implement some complex solution to have nodes communicate and try and prevent duplication, but I don't see it happening. Not right now anyway.

If we implement on top of the Storage API, nothing changes here, that API assumes consistent data just like the code does now. It has some more explicit ways of detecting conflict, but there's still a resolution behind the scenes that the IdP doesn't know about.

So I wanted to highlight this, I think this is just a non-starter as a consideration. I would say that in fact the unique constraint on the table is precisely what should be happening now (we don't generate the DDL, so that's always been more of a documentation thing and I never worked on that documentation).

Of course, the constraint only helps with the initial ID generation via hash, because the others would be random and wouldn't end up as duplicates, so there are still issues here, but I just wanted to call out that particular statement. I don't think it's in scope for us to solve this for multiple database nodes. Which is why databases are generally another single point of failure from the perspective of the IDP and probably not the focus of our development.

We sort of came to terms with using hashing as the method of choice for persistent IDs, and we definitely believe client-side storage is the better model for consent (yes, I think per-device prompting is better than having to run a database).

-- Scott