Tomcat database connection pools failing on start-up
Wessel, Keith
kwessel at illinois.edu
Wed Feb 18 12:29:21 EST 2015
Hi, Scott,
Yes, Tomcat does appear to start. The IDP, in fact, also starts. I looked more at logs and was wrong about that. But since the session factory for Hibernate couldn't be initialized because it points to my Tomcat connection pool, Hibernate throws an exception after the user submits their username and password which prevents the session from being created or anything else from happening. The user gets a blank screen in their browser.
Tomcat fails to initialize the database connection pool, then hibernate fails to associate its resource with the non-existent pool. Here are the logs. I'm no longer sure if this is something I should be configuring around in Tomcat or something that needs to be coded around in the DB-backed storage service.
Thoughts?
Keith
Feb 18, 2015 6:06:55 AM org.apache.catalina.startup.Catalina load
INFO: Initialization processed in 1069 ms
Feb 18, 2015 6:06:54 AM org.apache.catalina.core.StandardService start
INFO: Starting service Catalina
Feb 18, 2015 6:06:54 AM org.apache.catalina.core.StandardEngine start
INFO: Starting Servlet Engine: Apache Tomcat/6.0.41
Feb 18, 2015 6:06:54 AM org.apache.catalina.startup.HostConfig deployDescriptor
INFO: Deploying configuration descriptor idp.xml
Feb 18, 2015 6:06:57 AM org.apache.naming.NamingContext lookup
WARNING: Unexpected exception resolving reference
org.apache.tomcat.dbcp.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (The Network Adapter could not establish the connection)
at org.apache.tomcat.dbcp.dbcp.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:1549)
at org.apache.tomcat.dbcp.dbcp.BasicDataSource.createDataSource(BasicDataSource.java:1388)
at org.apache.tomcat.dbcp.dbcp.BasicDataSource.getLogWriter(BasicDataSource.java:1098)
at org.apache.tomcat.dbcp.dbcp.BasicDataSourceFactory.createDataSource(BasicDataSourceFactory.java:350)
at org.apache.tomcat.dbcp.dbcp.BasicDataSourceFactory.getObjectInstance(BasicDataSourceFactory.java:156)
at org.apache.naming.factory.ResourceFactory.getObjectInstance(ResourceFactory.java:140)
at javax.naming.spi.NamingManager.getObjectInstance(NamingManager.java:321)
at org.apache.naming.NamingContext.lookup(NamingContext.java:793)
at org.apache.naming.NamingContext.lookup(NamingContext.java:140)
at org.apache.naming.NamingContext.lookup(NamingContext.java:781)
at org.apache.naming.NamingContext.lookup(NamingContext.java:153)
at org.apache.catalina.core.NamingContextListener.addResource(NamingContextListener.java:1030)
at org.apache.catalina.core.NamingContextListener.createNamingContext(NamingContextListener.java:639)
at org.apache.catalina.core.NamingContextListener.lifecycleEvent(NamingContextListener.java:239)
at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:142)
at org.apache.catalina.core.StandardContext.start(StandardContext.java:4616)
at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:799)
at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:779)
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:583)
at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:675)
at org.apache.catalina.startup.HostConfig.deployDescriptors(HostConfig.java:601)
at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:502)
at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1317)
at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:324)
at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:142)
at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1065)
at org.apache.catalina.core.StandardHost.start(StandardHost.java:822)
at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1057)
at org.apache.catalina.core.StandardEngine.start(StandardEngine.java:463)
at org.apache.catalina.core.StandardService.start(StandardService.java:525)
at org.apache.catalina.core.StandardServer.start(StandardServer.java:754)
at org.apache.catalina.startup.Catalina.start(Catalina.java:595)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414)
Caused by: java.sql.SQLException: The Network Adapter could not establish the connection
at oracle.jdbc.driver.SQLStateMapping.newSQLException(SQLStateMapping.java:70)
at oracle.jdbc.driver.DatabaseError.newSQLException(DatabaseError.java:133)
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:199)
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:480)
at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:413)
at oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:508)
at oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:203)
at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:33)
at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:510)
at org.apache.tomcat.dbcp.dbcp.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:38)
at org.apache.tomcat.dbcp.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:582)
at org.apache.tomcat.dbcp.dbcp.BasicDataSource.validateConnectionFactory(BasicDataSource.java:1556)
at org.apache.tomcat.dbcp.dbcp.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:1545)
... 37 more
Caused by: oracle.net.ns.NetException: The Network Adapter could not establish the connection
at oracle.net.nt.ConnStrategy.execute(ConnStrategy.java:328)
at oracle.net.resolver.AddrResolution.resolveAndExecute(AddrResolution.java:421)
at oracle.net.ns.NSProtocol.establishConnection(NSProtocol.java:630)
at oracle.net.ns.NSProtocol.connect(NSProtocol.java:206)
at oracle.jdbc.driver.T4CConnection.connect(T4CConnection.java:966)
at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:292)
... 45 more
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at java.net.Socket.connect(Socket.java:528)
at java.net.Socket.<init>(Socket.java:425)
at java.net.Socket.<init>(Socket.java:208)
at oracle.net.nt.TcpNTAdapter.connect(TcpNTAdapter.java:127)
at oracle.net.nt.ConnOption.connect(ConnOption.java:126)
at oracle.net.nt.ConnStrategy.execute(ConnStrategy.java:306)
... 50 more
Feb 18, 2015 6:06:57 AM org.apache.catalina.core.NamingContextListener addResource
WARNING: Failed to register in JMX: javax.naming.NamingException: Cannot create PoolableConnectionFactory (The Network Adapter could not establish the connection)
Feb 18, 2015 6:07:04 AM org.apache.naming.NamingContext lookup
WARNING: Unexpected exception resolving reference
org.apache.tomcat.dbcp.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (The Network Adapter could not establish the connection)
at org.apache.tomcat.dbcp.dbcp.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:1549)
at org.apache.tomcat.dbcp.dbcp.BasicDataSource.createDataSource(BasicDataSource.java:1388)
at org.apache.tomcat.dbcp.dbcp.BasicDataSource.getLogWriter(BasicDataSource.java:1098)
at org.apache.tomcat.dbcp.dbcp.BasicDataSourceFactory.createDataSource(BasicDataSourceFactory.java:350)
at org.apache.tomcat.dbcp.dbcp.BasicDataSourceFactory.getObjectInstance(BasicDataSourceFactory.java:156)
at org.apache.naming.factory.ResourceFactory.getObjectInstance(ResourceFactory.java:140)
at javax.naming.spi.NamingManager.getObjectInstance(NamingManager.java:321)
at org.apache.naming.NamingContext.lookup(NamingContext.java:793)
at org.apache.naming.NamingContext.lookup(NamingContext.java:140)
at org.apache.naming.NamingContext.lookup(NamingContext.java:781)
at org.apache.naming.NamingContext.lookup(NamingContext.java:140)
at org.apache.naming.NamingContext.lookup(NamingContext.java:781)
at org.apache.naming.NamingContext.lookup(NamingContext.java:140)
at org.apache.naming.NamingContext.lookup(NamingContext.java:781)
at org.apache.naming.NamingContext.lookup(NamingContext.java:153)
at org.apache.naming.SelectorContext.lookup(SelectorContext.java:152)
at javax.naming.InitialContext.lookup(InitialContext.java:411)
at org.hibernate.connection.DatasourceConnectionProvider.configure(DatasourceConnectionProvider.java:52)
at org.hibernate.connection.ConnectionProviderFactory.newConnectionProvider(ConnectionProviderFactory.java:124)
at org.hibernate.connection.ConnectionProviderFactory.newConnectionProvider(ConnectionProviderFactory.java:56)
at org.hibernate.cfg.SettingsFactory.createConnectionProvider(SettingsFactory.java:414)
at org.hibernate.cfg.SettingsFactory.buildSettings(SettingsFactory.java:62)
at org.hibernate.cfg.Configuration.buildSettings(Configuration.java:2009)
at org.hibernate.cfg.Configuration.buildSessionFactory(Configuration.java:1292)
at net.clareitysecurity.shibboleth.db.SessionObjectDAOImpl$HibernateUtil.<clinit>(SessionObjectDAOImpl.java:370)
at net.clareitysecurity.shibboleth.db.SessionObjectDAOImpl.openSession(SessionObjectDAOImpl.java:341)
at net.clareitysecurity.shibboleth.db.SessionObjectDAOImpl.getExpiredSessions(SessionObjectDAOImpl.java:243)
at net.clareitysecurity.shibboleth.storage.StorageCleaner.run(StorageCleaner.java:50)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.sql.SQLException: The Network Adapter could not establish the connection
at oracle.jdbc.driver.SQLStateMapping.newSQLException(SQLStateMapping.java:70)
at oracle.jdbc.driver.DatabaseError.newSQLException(DatabaseError.java:133)
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:199)
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:480)
at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:413)
at oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:508)
at oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:203)
at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:33)
at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:510)
at org.apache.tomcat.dbcp.dbcp.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:38)
at org.apache.tomcat.dbcp.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:582)
at org.apache.tomcat.dbcp.dbcp.BasicDataSource.validateConnectionFactory(BasicDataSource.java:1556)
at org.apache.tomcat.dbcp.dbcp.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:1545)
... 28 more
Caused by: oracle.net.ns.NetException: The Network Adapter could not establish the connection
at oracle.net.nt.ConnStrategy.execute(ConnStrategy.java:328)
at oracle.net.resolver.AddrResolution.resolveAndExecute(AddrResolution.java:421)
at oracle.net.ns.NSProtocol.establishConnection(NSProtocol.java:630)
at oracle.net.ns.NSProtocol.connect(NSProtocol.java:206)
at oracle.jdbc.driver.T4CConnection.connect(T4CConnection.java:966)
at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:292)
... 36 more
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at java.net.Socket.connect(Socket.java:528)
at java.net.Socket.<init>(Socket.java:425)
at java.net.Socket.<init>(Socket.java:208)
at oracle.net.nt.TcpNTAdapter.connect(TcpNTAdapter.java:127)
at oracle.net.nt.ConnOption.connect(ConnOption.java:126)
at oracle.net.nt.ConnStrategy.execute(ConnStrategy.java:306)
... 41 more
Initial SessionFactory creation failed.org.hibernate.HibernateException: Could not find datasource
Exception in thread "Thread-4" java.lang.ExceptionInInitializerError
at net.clareitysecurity.shibboleth.db.SessionObjectDAOImpl$HibernateUtil.<clinit>(SessionObjectDAOImpl.java:375)
at net.clareitysecurity.shibboleth.db.SessionObjectDAOImpl.openSession(SessionObjectDAOImpl.java:341)
at net.clareitysecurity.shibboleth.db.SessionObjectDAOImpl.getExpiredSessions(SessionObjectDAOImpl.java:243)
at net.clareitysecurity.shibboleth.storage.StorageCleaner.run(StorageCleaner.java:50)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.hibernate.HibernateException: Could not find datasource
at org.hibernate.connection.DatasourceConnectionProvider.configure(DatasourceConnectionProvider.java:56)
at org.hibernate.connection.ConnectionProviderFactory.newConnectionProvider(ConnectionProviderFactory.java:124)
at org.hibernate.connection.ConnectionProviderFactory.newConnectionProvider(ConnectionProviderFactory.java:56)
at org.hibernate.cfg.SettingsFactory.createConnectionProvider(SettingsFactory.java:414)
at org.hibernate.cfg.SettingsFactory.buildSettings(SettingsFactory.java:62)
at org.hibernate.cfg.Configuration.buildSettings(Configuration.java:2009)
at org.hibernate.cfg.Configuration.buildSessionFactory(Configuration.java:1292)
at net.clareitysecurity.shibboleth.db.SessionObjectDAOImpl$HibernateUtil.<clinit>(SessionObjectDAOImpl.java:370)
... 4 more
Caused by: javax.naming.NamingException: Cannot create PoolableConnectionFactory (The Network Adapter could not establish the connection)
at org.apache.naming.NamingContext.lookup(NamingContext.java:805)
at org.apache.naming.NamingContext.lookup(NamingContext.java:140)
at org.apache.naming.NamingContext.lookup(NamingContext.java:781)
at org.apache.naming.NamingContext.lookup(NamingContext.java:140)
at org.apache.naming.NamingContext.lookup(NamingContext.java:781)
at org.apache.naming.NamingContext.lookup(NamingContext.java:140)
at org.apache.naming.NamingContext.lookup(NamingContext.java:781)
at org.apache.naming.NamingContext.lookup(NamingContext.java:153)
at org.apache.naming.SelectorContext.lookup(SelectorContext.java:152)
at javax.naming.InitialContext.lookup(InitialContext.java:411)
at org.hibernate.connection.DatasourceConnectionProvider.configure(DatasourceConnectionProvider.java:52)
... 11 more
Feb 18, 2015 6:07:41 AM org.apache.jk.common.ChannelSocket init
INFO: JK: ajp13 listening on /127.0.0.1:8000
Feb 18, 2015 6:07:41 AM org.apache.jk.server.JkMain start
INFO: Jk running ID=0 time=0/110 config=null
Feb 18, 2015 6:07:41 AM org.apache.catalina.startup.Catalina start
INFO: Server startup in 46448 ms
-----Original Message-----
From: users-bounces at shibboleth.net [mailto:users-bounces at shibboleth.net] On Behalf Of Cantor, Scott
Sent: Wednesday, February 18, 2015 10:17 AM
To: Shib Users
Subject: Re: Tomcat database connection pools failing on start-up
On 2/18/15, 4:10 PM, "Wessel, Keith" <kwessel at illinois.edu> wrote:
>
>We're using a Tomcat resource for the database connection pool. Does
>anyone know of any way to tell Tomcat it can continue running and start
>the IDP even when a resource like that is unavailable? Or do I just have
>to tell our sysadmins not to restart the server if the database server is
>down? :)
I'm surprised Tomcat would care, and it doesn't really know anything about
pooling, that's just a function of the data source. It's more likely to do
with the pool configuration itself, I imagine.
>If Jetty handles this situation better, BTW, this could be my excuse to
>switch. I'm planning on doing that, anyway, for my IDP V3 move later this
>year. Does anyone know if it does?
I don't really know. I'm not clear what Tomcat's doing with it. Does
Tomcat itself not start, or is it just the IdP not starting?
-- Scott
--
To unsubscribe from this list send an email to users-unsubscribe at shibboleth.net
More information about the users
mailing list