does round robin policy cause a delay in giving new connections?
karthik
Vertica Customer ✭
Hi folks,
I have a 3-node vertica cluster with vertica native connection load-balancer set to "round-robin". This is version 8.0.1.x running on vmware esxi. Our application runs on a separate node on another vmware esxi.
Our app needs to establish a bunch of connections (jdbc) during start-up and start-up is seen getting delayed. If the round-robin is disabled, this would instantly work.
Are there any known issues around this ? Our app is deployed into wildfly (10.1) application server.
Could this be because the load-balancer is doing re-directs which is somehow causing the new connections to be established with a delay (some kind of tcp state becoming stale ?)
Best Answer
-
LenoyJ
- Select Field - Employee
It is not a requirement that you need a public IP for clients.
Try setting the export addresses of the nodes (using this guide) to IPs of if3, turn on load balancing on both client and server and try again. See if that helps?
Quoting from this doc page:When both client and server enable load balancing, the following process takes place when the client attempts to open a connection to Vertica:
- The client connects to a host in the database cluster, with a connection parameter indicating that it is requesting a load-balanced connection.
- The host chooses a host from the list of currently up hosts in the cluster, according to the current load balancing scheme. Under all schemes, it is possible for a host to select itself.
- The host tells the client which host it selected to handle the client's connection.
- If the host chose another host in the database to handle the client connection, the client disconnects from the initial host. Otherwise, the client jumps to step 6.
- The client establishes a connection to the host that will handle its connection. The client sets this second connection request so that the second host does not interpret the connection as a request for load balancing.
- The client connection proceeds as usual.
So basically, your application connects to one of the IPs of if3. Since load balancing is enabled, Vertica returns one of the IPs listed in export addresses and tells the JDBC driver to reconnect to that. In your case, it returns the one of the IPs of if1 (not if3) as that's whats set in export addresses. Changing export addresses to IPs of if3 may help.
5
Answers
Hi karthik,
Do you have a private and a public interface? Have you set up network interfaces or subnets https://www.vertica.com/docs/8.1.x/HTML/index.htm#Authoring/SQLReferenceManual/Statements/CREATENETWORKINTERFACE.htm?
I would make sure the load balancing is working correctly.
I've seen cases when it's not configured correctly, Vertica tries to reach the private interface and once it fails, it connects to the initiator's public interface which results in a delay.
Hope this helps
Chaima
Hi Chaima
Thanks. We have both private and public IPs on each node of vertica as well as the node where our application is running. However, the application server is configured to use the private IPs used by vertica cluster for the internal communication. Is it mandatory that the application server (or any client outside of vertica cluster) should connect using the public IP of the vertica nodes?
"ServerService Thread Pool -- 64" #184 prio=5 os_prio=0 tid=0x00007f74800ecdb0 nid=0x2828 runnable [0x00007f7364319000]
java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
- locked <0x0000000781711ae0> (a java.net.SocksSocketImpl)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at java.net.Socket.connect(Socket.java:538)
at com.vertica.io.VStream.establishConnection(Unknown Source)
at com.vertica.io.VStream.(Unknown Source)
at com.vertica.io.ProtocolStream.loadBalance(Unknown Source)
at com.vertica.io.ProtocolStream.initSession(Unknown Source)
at com.vertica.core.VConnection.tryConnect(Unknown Source)
at com.vertica.core.VConnection.connect(Unknown Source)
at com.vertica.jdbc.common.BaseConnectionFactory.doConnect(Unknown Source)
at com.vertica.jdbc.common.AbstractDriver.connect(Unknown Source)
at org.jboss.jca.adapters.jdbc.local.LocalManagedConnectionFactory.createLocalManagedConnection(LocalManagedConnectionFactory.java:321)
at org.jboss.jca.adapters.jdbc.local.LocalManagedConnectionFactory.getLocalManagedConnection(LocalManagedConnectionFactory.java:352)
at org.jboss.jca.adapters.jdbc.local.LocalManagedConnectionFactory.createManagedConnection(LocalManagedConnectionFactory.java:287)
at org.jboss.jca.core.connectionmanager.pool.mcp.SemaphoreConcurrentLinkedDequeManagedConnectionPool.createConnectionEventListener(SemaphoreConcurrentLinkedDequeManagedConnectionPool.java:1320)
at org.jboss.jca.core.connectionmanager.pool.mcp.SemaphoreConcurrentLinkedDequeManagedConnectionPool.getConnection(SemaphoreConcurrentLinkedDequeManagedConnectionPool.java:496)
at org.jboss.jca.core.connectionmanager.pool.AbstractPool.getSimpleConnection(AbstractPool.java:617)
at org.jboss.jca.core.connectionmanager.pool.AbstractPool.getConnection(AbstractPool.java:589)
at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.getManagedConnection(AbstractConnectionManager.java:590)
at org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.getManagedConnection(TxConnectionManagerImpl.java:429)
at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:747)
at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:138)
at org.jboss.as.connector.subsystems.datasources.WildFlyDataSource.getConnection(WildFlyDataSource.java:66)
at org.hibernate.engine.jdbc.connections.internal.DatasourceConnectionProviderImpl.getConnection(DatasourceConnectionProviderImpl.java:122)
at org.hibernate.engine.jdbc.env.internal.JdbcEnvironmentInitiator$ConnectionProviderJdbcConnectionAccess.obtainConnection(JdbcEnvironmentInitiator.java:180)
at org.hibernate.engine.jdbc.env.internal.JdbcEnvironmentInitiator.initiateService(JdbcEnvironmentInitiator.java:68)
at org.hibernate.engine.jdbc.env.internal.JdbcEnvironmentInitiator.initiateService(JdbcEnvironmentInitiator.java:35)
at org.hibernate.boot.registry.internal.StandardServiceRegistryImpl.initiateService(StandardServiceRegistryImpl.java:88)
at org.hibernate.service.internal.AbstractServiceRegistryImpl.createService(AbstractServiceRegistryImpl.java:254)
at org.hibernate.service.internal.AbstractServiceRegistryImpl.initializeService(AbstractServiceRegistryImpl.java:228)
at org.hibernate.service.internal.AbstractServiceRegistryImpl.getService(AbstractServiceRegistryImpl.java:207)
- locked <0x0000000375ff6dc8> (a org.hibernate.boot.registry.internal.StandardServiceRegistryImpl)
at org.hibernate.engine.jdbc.internal.JdbcServicesImpl.configure(JdbcServicesImpl.java:51)
at org.hibernate.boot.registry.internal.StandardServiceRegistryImpl.configureService(StandardServiceRegistryImpl.java:94)
at org.hibernate.service.internal.AbstractServiceRegistryImpl.initializeService(AbstractServiceRegistryImpl.java:237)
at org.hibernate.service.internal.AbstractServiceRegistryImpl.getService(AbstractServiceRegistryImpl.java:207)
- locked <0x0000000375ff6dc8> (a org.hibernate.boot.registry.internal.StandardServiceRegistryImpl)
at org.hibernate.boot.model.process.spi.MetadataBuildingProcess.handleTypes(MetadataBuildingProcess.java:352)
at org.hibernate.boot.model.process.spi.MetadataBuildingProcess.complete(MetadataBuildingProcess.java:111)
at org.hibernate.jpa.boot.internal.EntityManagerFactoryBuilderImpl.metadata(EntityManagerFactoryBuilderImpl.java:847)
at org.hibernate.jpa.boot.internal.EntityManagerFactoryBuilderImpl.build(EntityManagerFactoryBuilderImpl.java:874)
at org.jboss.as.jpa.hibernate5.TwoPhaseBootstrapImpl.build(TwoPhaseBootstrapImpl.java:44)
at org.jboss.as.jpa.service.PersistenceUnitServiceImpl$1$1.run(PersistenceUnitServiceImpl.java:161)
at org.jboss.as.jpa.service.PersistenceUnitServiceImpl$1$1.run(PersistenceUnitServiceImpl.java:121)
at org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:667)
at org.jboss.as.jpa.service.PersistenceUnitServiceImpl$1.run(PersistenceUnitServiceImpl.java:193)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
at org.jboss.threads.JBossThread.run(JBossThread.java:320)
StackTrace:
"ServerService Thread Pool -- 64" #184 prio=5 os_prio=0 tid=0x00007f74800ecdb0 nid=0x2828 runnable [0x00007f7364319000]
java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
- locked <0x0000000781711ae0> (a java.net.SocksSocketImpl)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at java.net.Socket.connect(Socket.java:538)
at com.vertica.io.VStream.establishConnection(Unknown Source)
at com.vertica.io.VStream.(Unknown Source)
at com.vertica.io.ProtocolStream.loadBalance(Unknown Source)
at com.vertica.io.ProtocolStream.initSession(Unknown Source)
at com.vertica.core.VConnection.tryConnect(Unknown Source)
at com.vertica.core.VConnection.connect(Unknown Source)
at com.vertica.jdbc.common.BaseConnectionFactory.doConnect(Unknown Source)
at com.vertica.jdbc.common.AbstractDriver.connect(Unknown Source)
at org.jboss.jca.adapters.jdbc.local.LocalManagedConnectionFactory.createLocalManagedConnection(LocalManagedConnectionFactory.java:321)
See if what is set in your export addresses matches what you expect the application server to connect to.
The following docs talk about export addresses in the concept of exporting/importing data but it can be applied to load balancing too:
https://www.vertica.com/docs/8.1.x/HTML/index.htm#Authoring/AdministratorsGuide/CopyExportData/ChangingNodeExportAddress.htm
https://www.vertica.com/kb/Configuring-Network-to-Import-and-Export-Data/Content/BestPractices/Configuring-Network-to-Import-and-Export-Data.htm
Hi Lenoy,
I have the below interfaces on each vertica node:
if1 -> ip1(private)
if2 -> ip2 (public)
if3 -> ip3 (private)
lo -> 127.0.0.1
The export_address and node_address is set to if1 IP (as per nodes table)
whereas the client (our application) is configured to connect to IPs of if3.
This appears to be working fine for us otherwise as our application is also within the private network..
However, when native load balancing (round-robin) is enabled, this is causing the problem.
What is the reason vertica recommends having a public IP for clients to connect to vertica nodes ? One obvious reason i know is to not interfere the node-to-node communication in vertica cluster.. But is it mandatory that we should use a public IP or an interface that is not used by node-to-node cluster communication between vertica?
Thanks,
Karthik S
Thanks very much Lenoy. Will try that and get back