Too many database connections
kwessel at illinois.edu
Fri Aug 24 17:03:40 EDT 2018
I think this is precisely the problem. We're going through an SLB, and I know for a fact that our SLB has a 300-second idle timeout that does exactly as you described. It closes the connection without telling either side. The IdP is getting rid of them the next time it validates connections in the pool: netstat is reporting 10 connections to the database server. But the database server doesn't know to get rid of them.
By increase, I assume you mean you made the TCP keepalives more freuquent? Can you tell me how you changed this? I see no controls for HikariCP for doing this.
From: users <users-bounces at shibboleth.net> On Behalf Of Brian Biggs
Sent: Friday, August 24, 2018 3:30 PM
To: users at shibboleth.net
Subject: Re: Too many database connections
Do the connections between your IdP and the DB flow through another network device (FW, Router, or switch)? We had an issue a while back with a router that would time out connections and not tell either side that it had terminated. Since our network folks couldn't find the issue we increased the TCP Keepalive settings and that fixed it.
On 08/24/2018 01:09 PM, Wessel, Keith wrote:
> Hi, all,
> A strange problem started up this morning that we haven't seen since moving to HikariCP for connection pooling a couple years ago. Our IdP nodes are opening way more database connections than they should be. I've seen as many s 80 connections from mysql's show processlist command from a single IdP node.
> We're only using database storage for consent, and we've left hikaricp at all of its defaults. Fro the docs, it looks like the max pool size is 10.
> When we shut down an IdP node, its connections seem to hang around, too, slowly going down in umber. We had a node that I shut down a couple hours ago. It had 50+ connections when I shut it down, it's still got 15, but the IdP (and there's nothing else on the server that makes database connections) hasn't been running for a couple hours.
> Any suggestions for what a cause might be or where I might start troubleshooting this?
> I can always raise my MySQL max client connections, but I suspect that will just put off the problem until I have even more connections.
Lead IdM/IdP/Systems Integration
Sonoma State University
For Consortium Member technical support, see https://wiki.shibboleth.net/confluence/x/coFAAg
To unsubscribe from this list send an email to users-unsubscribe at shibboleth.net
More information about the users