Too many database connections

[Wessel, Keith]

Thanks, Brian. The OS-level change is one option. I chose to set maxLifetime for my HikariCP configuration to a lower value after reading this which I somehow missed before:

maxLifeTime:
... We strongly recommend setting this value, and it should be several seconds shorter than any database or infrastructure imposed connection time limit. A value of 0 indicates no maximum lifetime (infinite lifetime), subject of course to the idleTimeout setting. Default: 1800000 (30 minutes)

I changed this to 4 minutes to be less than our SLB's 5-minute idle timeout. We'll see if it works. I suspect it will. And I'm not sure why this never showed up before today. Granted, the students are back on campus, so...

Keith


-----Original Message-----
From: users <users-bounces at shibboleth.net> On Behalf Of Brian Biggs
Sent: Friday, August 24, 2018 4:10 PM
To: users at shibboleth.net
Subject: Re: Too many database connections

Hi,

My situation involved OpenLDAP (syncing issues between masters and 
slaves in different zones in the network) not HikariCP; However, you 
should be able to adjust the TCP keepalive settings at the network 
interface layer on the server that your software is running on.

-Brian

On 08/24/2018 02:03 PM, Wessel, Keith wrote:
> Brian,
>
> I think this is precisely the problem. We're going through an SLB, and I know for a fact that our SLB has a 300-second idle timeout that does exactly as you described. It closes the connection without telling either side. The IdP is getting rid of them the next time it validates connections in the pool: netstat is reporting 10 connections to the database server. But the database server doesn't know to get rid of them.
>
> By increase, I assume you mean you made the TCP keepalives more freuquent? Can you tell me how you changed this? I see no controls for HikariCP for doing this.
>
> Thanks,
> Keith
>
>
> -----Original Message-----
> From: users <users-bounces at shibboleth.net> On Behalf Of Brian Biggs
> Sent: Friday, August 24, 2018 3:30 PM
> To: users at shibboleth.net
> Subject: Re: Too many database connections
>
> Do the connections between your IdP and the DB flow through another network device (FW, Router, or switch)? We had an issue a while back with a router that would time out connections and not tell either side that it had terminated. Since our network folks couldn't find the issue we increased the TCP Keepalive settings and that fixed it.
>
> -Brian
>
> On 08/24/2018 01:09 PM, Wessel, Keith wrote:
>> Hi, all,
>>
>> A strange problem started up this morning that we haven't seen since moving to HikariCP for connection pooling a couple years ago. Our IdP nodes are opening way more database connections than they should be. I've seen as many s 80 connections from mysql's show processlist command from a single IdP node.
>>
>> We're only using database storage for consent, and we've left hikaricp at all of its defaults. Fro the docs, it looks like the max pool size is 10.
>>
>> When we shut down an IdP node, its connections seem to hang around, too, slowly going down in umber. We had a node that I shut down a couple hours ago. It had 50+ connections when I shut it down, it's still got 15, but the IdP (and there's nothing else on the server that makes database connections) hasn't been running for a couple hours.
>>
>> Any suggestions for what a cause might be or where I might start troubleshooting this?
>>
>> I can always raise my MySQL max client connections, but I suspect that will just put off the problem until I have even more connections.
>>
>> Thanks,
>> Keith
>>
> --
> Lead IdM/IdP/Systems Integration
> Information Technology
> Sonoma State University
>
> --
> For Consortium Member technical support, see https://wiki.shibboleth.net/confluence/x/coFAAg
> To unsubscribe from this list send an email to users-unsubscribe at shibboleth.net

-- 
Lead IdM/IdP/Systems Integration
Information Technology
Sonoma State University

-- 
For Consortium Member technical support, see https://wiki.shibboleth.net/confluence/x/coFAAg
To unsubscribe from this list send an email to users-unsubscribe at shibboleth.net