Too many database connections

[Brian Biggs]

Hi,

My situation involved OpenLDAP (syncing issues between masters and 
slaves in different zones in the network) not HikariCP; However, you 
should be able to adjust the TCP keepalive settings at the network 
interface layer on the server that your software is running on.

-Brian

On 08/24/2018 02:03 PM, Wessel, Keith wrote:
> Brian,
>
> I think this is precisely the problem. We're going through an SLB, and I know for a fact that our SLB has a 300-second idle timeout that does exactly as you described. It closes the connection without telling either side. The IdP is getting rid of them the next time it validates connections in the pool: netstat is reporting 10 connections to the database server. But the database server doesn't know to get rid of them.
>
> By increase, I assume you mean you made the TCP keepalives more freuquent? Can you tell me how you changed this? I see no controls for HikariCP for doing this.
>
> Thanks,
> Keith
>
>
> -----Original Message-----
> From: users <users-bounces at shibboleth.net> On Behalf Of Brian Biggs
> Sent: Friday, August 24, 2018 3:30 PM
> To: users at shibboleth.net
> Subject: Re: Too many database connections
>
> Do the connections between your IdP and the DB flow through another network device (FW, Router, or switch)? We had an issue a while back with a router that would time out connections and not tell either side that it had terminated. Since our network folks couldn't find the issue we increased the TCP Keepalive settings and that fixed it.
>
> -Brian
>
> On 08/24/2018 01:09 PM, Wessel, Keith wrote:
>> Hi, all,
>>
>> A strange problem started up this morning that we haven't seen since moving to HikariCP for connection pooling a couple years ago. Our IdP nodes are opening way more database connections than they should be. I've seen as many s 80 connections from mysql's show processlist command from a single IdP node.
>>
>> We're only using database storage for consent, and we've left hikaricp at all of its defaults. Fro the docs, it looks like the max pool size is 10.
>>
>> When we shut down an IdP node, its connections seem to hang around, too, slowly going down in umber. We had a node that I shut down a couple hours ago. It had 50+ connections when I shut it down, it's still got 15, but the IdP (and there's nothing else on the server that makes database connections) hasn't been running for a couple hours.
>>
>> Any suggestions for what a cause might be or where I might start troubleshooting this?
>>
>> I can always raise my MySQL max client connections, but I suspect that will just put off the problem until I have even more connections.
>>
>> Thanks,
>> Keith
>>
> --
> Lead IdM/IdP/Systems Integration
> Information Technology
> Sonoma State University
>
> --
> For Consortium Member technical support, see https://wiki.shibboleth.net/confluence/x/coFAAg
> To unsubscribe from this list send an email to users-unsubscribe at shibboleth.net

-- 
Lead IdM/IdP/Systems Integration
Information Technology
Sonoma State University