Please take this survey to help us learn more about how you use third party tools. Your input is greatly appreciated!

does round robin policy cause a delay in giving new connections?

karthikkarthik
edited February 2020 in General Discussion

Hi folks,

I have a 3-node vertica cluster with vertica native connection load-balancer set to "round-robin". This is version 8.0.1.x running on vmware esxi. Our application runs on a separate node on another vmware esxi.
Our app needs to establish a bunch of connections (jdbc) during start-up and start-up is seen getting delayed. If the round-robin is disabled, this would instantly work.
Are there any known issues around this ? Our app is deployed into wildfly (10.1) application server.
Could this be because the load-balancer is doing re-directs which is somehow causing the new connections to be established with a delay (some kind of tcp state becoming stale ?)

Best Answer

Answers

  • Hi karthik,
    Do you have a private and a public interface? Have you set up network interfaces or subnets https://www.vertica.com/docs/8.1.x/HTML/index.htm#Authoring/SQLReferenceManual/Statements/CREATENETWORKINTERFACE.htm?
    I would make sure the load balancing is working correctly.
    I've seen cases when it's not configured correctly, Vertica tries to reach the private interface and once it fails, it connects to the initiator's public interface which results in a delay.
    Hope this helps
    Chaima

  • Hi Chaima
    Thanks. We have both private and public IPs on each node of vertica as well as the node where our application is running. However, the application server is configured to use the private IPs used by vertica cluster for the internal communication. Is it mandatory that the application server (or any client outside of vertica cluster) should connect using the public IP of the vertica nodes?

  • "ServerService Thread Pool -- 64" #184 prio=5 os_prio=0 tid=0x00007f74800ecdb0 nid=0x2828 runnable [0x00007f7364319000]
    java.lang.Thread.State: RUNNABLE
    at java.net.PlainSocketImpl.socketConnect(Native Method)
    at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
    - locked <0x0000000781711ae0> (a java.net.SocksSocketImpl)
    at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
    at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
    at java.net.Socket.connect(Socket.java:589)
    at java.net.Socket.connect(Socket.java:538)
    at com.vertica.io.VStream.establishConnection(Unknown Source)
    at com.vertica.io.VStream.(Unknown Source)
    at com.vertica.io.ProtocolStream.loadBalance(Unknown Source)
    at com.vertica.io.ProtocolStream.initSession(Unknown Source)
    at com.vertica.core.VConnection.tryConnect(Unknown Source)
    at com.vertica.core.VConnection.connect(Unknown Source)
    at com.vertica.jdbc.common.BaseConnectionFactory.doConnect(Unknown Source)
    at com.vertica.jdbc.common.AbstractDriver.connect(Unknown Source)
    at org.jboss.jca.adapters.jdbc.local.LocalManagedConnectionFactory.createLocalManagedConnection(LocalManagedConnectionFactory.java:321)
    at org.jboss.jca.adapters.jdbc.local.LocalManagedConnectionFactory.getLocalManagedConnection(LocalManagedConnectionFactory.java:352)
    at org.jboss.jca.adapters.jdbc.local.LocalManagedConnectionFactory.createManagedConnection(LocalManagedConnectionFactory.java:287)
    at org.jboss.jca.core.connectionmanager.pool.mcp.SemaphoreConcurrentLinkedDequeManagedConnectionPool.createConnectionEventListener(SemaphoreConcurrentLinkedDequeManagedConnectionPool.java:1320)
    at org.jboss.jca.core.connectionmanager.pool.mcp.SemaphoreConcurrentLinkedDequeManagedConnectionPool.getConnection(SemaphoreConcurrentLinkedDequeManagedConnectionPool.java:496)
    at org.jboss.jca.core.connectionmanager.pool.AbstractPool.getSimpleConnection(AbstractPool.java:617)
    at org.jboss.jca.core.connectionmanager.pool.AbstractPool.getConnection(AbstractPool.java:589)
    at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.getManagedConnection(AbstractConnectionManager.java:590)
    at org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.getManagedConnection(TxConnectionManagerImpl.java:429)
    at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:747)
    at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:138)
    at org.jboss.as.connector.subsystems.datasources.WildFlyDataSource.getConnection(WildFlyDataSource.java:66)
    at org.hibernate.engine.jdbc.connections.internal.DatasourceConnectionProviderImpl.getConnection(DatasourceConnectionProviderImpl.java:122)
    at org.hibernate.engine.jdbc.env.internal.JdbcEnvironmentInitiator$ConnectionProviderJdbcConnectionAccess.obtainConnection(JdbcEnvironmentInitiator.java:180)
    at org.hibernate.engine.jdbc.env.internal.JdbcEnvironmentInitiator.initiateService(JdbcEnvironmentInitiator.java:68)
    at org.hibernate.engine.jdbc.env.internal.JdbcEnvironmentInitiator.initiateService(JdbcEnvironmentInitiator.java:35)
    at org.hibernate.boot.registry.internal.StandardServiceRegistryImpl.initiateService(StandardServiceRegistryImpl.java:88)
    at org.hibernate.service.internal.AbstractServiceRegistryImpl.createService(AbstractServiceRegistryImpl.java:254)
    at org.hibernate.service.internal.AbstractServiceRegistryImpl.initializeService(AbstractServiceRegistryImpl.java:228)
    at org.hibernate.service.internal.AbstractServiceRegistryImpl.getService(AbstractServiceRegistryImpl.java:207)
    - locked <0x0000000375ff6dc8> (a org.hibernate.boot.registry.internal.StandardServiceRegistryImpl)
    at org.hibernate.engine.jdbc.internal.JdbcServicesImpl.configure(JdbcServicesImpl.java:51)
    at org.hibernate.boot.registry.internal.StandardServiceRegistryImpl.configureService(StandardServiceRegistryImpl.java:94)
    at org.hibernate.service.internal.AbstractServiceRegistryImpl.initializeService(AbstractServiceRegistryImpl.java:237)
    at org.hibernate.service.internal.AbstractServiceRegistryImpl.getService(AbstractServiceRegistryImpl.java:207)
    - locked <0x0000000375ff6dc8> (a org.hibernate.boot.registry.internal.StandardServiceRegistryImpl)
    at org.hibernate.boot.model.process.spi.MetadataBuildingProcess.handleTypes(MetadataBuildingProcess.java:352)
    at org.hibernate.boot.model.process.spi.MetadataBuildingProcess.complete(MetadataBuildingProcess.java:111)
    at org.hibernate.jpa.boot.internal.EntityManagerFactoryBuilderImpl.metadata(EntityManagerFactoryBuilderImpl.java:847)
    at org.hibernate.jpa.boot.internal.EntityManagerFactoryBuilderImpl.build(EntityManagerFactoryBuilderImpl.java:874)
    at org.jboss.as.jpa.hibernate5.TwoPhaseBootstrapImpl.build(TwoPhaseBootstrapImpl.java:44)
    at org.jboss.as.jpa.service.PersistenceUnitServiceImpl$1$1.run(PersistenceUnitServiceImpl.java:161)
    at org.jboss.as.jpa.service.PersistenceUnitServiceImpl$1$1.run(PersistenceUnitServiceImpl.java:121)
    at org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:667)
    at org.jboss.as.jpa.service.PersistenceUnitServiceImpl$1.run(PersistenceUnitServiceImpl.java:193)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
    at org.jboss.threads.JBossThread.run(JBossThread.java:320)

  • StackTrace:
    "ServerService Thread Pool -- 64" #184 prio=5 os_prio=0 tid=0x00007f74800ecdb0 nid=0x2828 runnable [0x00007f7364319000]
    java.lang.Thread.State: RUNNABLE
    at java.net.PlainSocketImpl.socketConnect(Native Method)
    at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
    - locked <0x0000000781711ae0> (a java.net.SocksSocketImpl)
    at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
    at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
    at java.net.Socket.connect(Socket.java:589)
    at java.net.Socket.connect(Socket.java:538)
    at com.vertica.io.VStream.establishConnection(Unknown Source)
    at com.vertica.io.VStream.(Unknown Source)
    at com.vertica.io.ProtocolStream.loadBalance(Unknown Source)
    at com.vertica.io.ProtocolStream.initSession(Unknown Source)
    at com.vertica.core.VConnection.tryConnect(Unknown Source)
    at com.vertica.core.VConnection.connect(Unknown Source)
    at com.vertica.jdbc.common.BaseConnectionFactory.doConnect(Unknown Source)
    at com.vertica.jdbc.common.AbstractDriver.connect(Unknown Source)
    at org.jboss.jca.adapters.jdbc.local.LocalManagedConnectionFactory.createLocalManagedConnection(LocalManagedConnectionFactory.java:321)

  • LenoyJLenoyJ Employee
    edited February 2020

    In classic connection load balancing, each node in the cluster can only be reached via a single IP address. This address is set in the EXPORT_ADDRESS column of the NODES system table.

    See if what is set in your export addresses matches what you expect the application server to connect to.
     
    The following docs talk about export addresses in the concept of exporting/importing data but it can be applied to load balancing too:
    https://www.vertica.com/docs/8.1.x/HTML/index.htm#Authoring/AdministratorsGuide/CopyExportData/ChangingNodeExportAddress.htm
    https://www.vertica.com/kb/Configuring-Network-to-Import-and-Export-Data/Content/BestPractices/Configuring-Network-to-Import-and-Export-Data.htm

  • Hi Lenoy,
    I have the below interfaces on each vertica node:
    if1 -> ip1(private)
    if2 -> ip2 (public)
    if3 -> ip3 (private)
    lo -> 127.0.0.1
    The export_address and node_address is set to if1 IP (as per nodes table)
    whereas the client (our application) is configured to connect to IPs of if3.
    This appears to be working fine for us otherwise as our application is also within the private network..
    However, when native load balancing (round-robin) is enabled, this is causing the problem.
    What is the reason vertica recommends having a public IP for clients to connect to vertica nodes ? One obvious reason i know is to not interfere the node-to-node communication in vertica cluster.. But is it mandatory that we should use a public IP or an interface that is not used by node-to-node cluster communication between vertica?
    Thanks,
    Karthik S

  • Thanks very much Lenoy. Will try that and get back

Leave a Comment

BoldItalicStrikethroughOrdered listUnordered list
Emoji
Image
Align leftAlign centerAlign rightToggle HTML viewToggle full pageToggle lights
Drop image/file

Can't find what you're looking for? Search the Vertica Documentation, Knowledge Base, or Blog for more information.