does round robin policy cause a delay in giving new connections?

karthikkarthik Vertica Customer
edited February 2020 in General Discussion

Hi folks,

I have a 3-node vertica cluster with vertica native connection load-balancer set to "round-robin". This is version 8.0.1.x running on vmware esxi. Our application runs on a separate node on another vmware esxi.
Our app needs to establish a bunch of connections (jdbc) during start-up and start-up is seen getting delayed. If the round-robin is disabled, this would instantly work.
Are there any known issues around this ? Our app is deployed into wildfly (10.1) application server.
Could this be because the load-balancer is doing re-directs which is somehow causing the new connections to be established with a delay (some kind of tcp state becoming stale ?)

Best Answer

Answers

  • chaimachaima Vertica Employee Employee

    Hi karthik,
    Do you have a private and a public interface? Have you set up network interfaces or subnets https://www.vertica.com/docs/8.1.x/HTML/index.htm#Authoring/SQLReferenceManual/Statements/CREATENETWORKINTERFACE.htm?
    I would make sure the load balancing is working correctly.
    I've seen cases when it's not configured correctly, Vertica tries to reach the private interface and once it fails, it connects to the initiator's public interface which results in a delay.
    Hope this helps
    Chaima

  • karthikkarthik Vertica Customer

    Hi Chaima
    Thanks. We have both private and public IPs on each node of vertica as well as the node where our application is running. However, the application server is configured to use the private IPs used by vertica cluster for the internal communication. Is it mandatory that the application server (or any client outside of vertica cluster) should connect using the public IP of the vertica nodes?

  • karthikkarthik Vertica Customer

    "ServerService Thread Pool -- 64" #184 prio=5 os_prio=0 tid=0x00007f74800ecdb0 nid=0x2828 runnable [0x00007f7364319000]
    java.lang.Thread.State: RUNNABLE
    at java.net.PlainSocketImpl.socketConnect(Native Method)
    at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
    - locked <0x0000000781711ae0> (a java.net.SocksSocketImpl)
    at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
    at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
    at java.net.Socket.connect(Socket.java:589)
    at java.net.Socket.connect(Socket.java:538)
    at com.vertica.io.VStream.establishConnection(Unknown Source)
    at com.vertica.io.VStream.(Unknown Source)
    at com.vertica.io.ProtocolStream.loadBalance(Unknown Source)
    at com.vertica.io.ProtocolStream.initSession(Unknown Source)
    at com.vertica.core.VConnection.tryConnect(Unknown Source)
    at com.vertica.core.VConnection.connect(Unknown Source)
    at com.vertica.jdbc.common.BaseConnectionFactory.doConnect(Unknown Source)
    at com.vertica.jdbc.common.AbstractDriver.connect(Unknown Source)
    at org.jboss.jca.adapters.jdbc.local.LocalManagedConnectionFactory.createLocalManagedConnection(LocalManagedConnectionFactory.java:321)
    at org.jboss.jca.adapters.jdbc.local.LocalManagedConnectionFactory.getLocalManagedConnection(LocalManagedConnectionFactory.java:352)
    at org.jboss.jca.adapters.jdbc.local.LocalManagedConnectionFactory.createManagedConnection(LocalManagedConnectionFactory.java:287)
    at org.jboss.jca.core.connectionmanager.pool.mcp.SemaphoreConcurrentLinkedDequeManagedConnectionPool.createConnectionEventListener(SemaphoreConcurrentLinkedDequeManagedConnectionPool.java:1320)
    at org.jboss.jca.core.connectionmanager.pool.mcp.SemaphoreConcurrentLinkedDequeManagedConnectionPool.getConnection(SemaphoreConcurrentLinkedDequeManagedConnectionPool.java:496)
    at org.jboss.jca.core.connectionmanager.pool.AbstractPool.getSimpleConnection(AbstractPool.java:617)
    at org.jboss.jca.core.connectionmanager.pool.AbstractPool.getConnection(AbstractPool.java:589)
    at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.getManagedConnection(AbstractConnectionManager.java:590)
    at org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.getManagedConnection(TxConnectionManagerImpl.java:429)
    at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:747)
    at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:138)
    at org.jboss.as.connector.subsystems.datasources.WildFlyDataSource.getConnection(WildFlyDataSource.java:66)
    at org.hibernate.engine.jdbc.connections.internal.DatasourceConnectionProviderImpl.getConnection(DatasourceConnectionProviderImpl.java:122)
    at org.hibernate.engine.jdbc.env.internal.JdbcEnvironmentInitiator$ConnectionProviderJdbcConnectionAccess.obtainConnection(JdbcEnvironmentInitiator.java:180)
    at org.hibernate.engine.jdbc.env.internal.JdbcEnvironmentInitiator.initiateService(JdbcEnvironmentInitiator.java:68)
    at org.hibernate.engine.jdbc.env.internal.JdbcEnvironmentInitiator.initiateService(JdbcEnvironmentInitiator.java:35)
    at org.hibernate.boot.registry.internal.StandardServiceRegistryImpl.initiateService(StandardServiceRegistryImpl.java:88)
    at org.hibernate.service.internal.AbstractServiceRegistryImpl.createService(AbstractServiceRegistryImpl.java:254)
    at org.hibernate.service.internal.AbstractServiceRegistryImpl.initializeService(AbstractServiceRegistryImpl.java:228)
    at org.hibernate.service.internal.AbstractServiceRegistryImpl.getService(AbstractServiceRegistryImpl.java:207)
    - locked <0x0000000375ff6dc8> (a org.hibernate.boot.registry.internal.StandardServiceRegistryImpl)
    at org.hibernate.engine.jdbc.internal.JdbcServicesImpl.configure(JdbcServicesImpl.java:51)
    at org.hibernate.boot.registry.internal.StandardServiceRegistryImpl.configureService(StandardServiceRegistryImpl.java:94)
    at org.hibernate.service.internal.AbstractServiceRegistryImpl.initializeService(AbstractServiceRegistryImpl.java:237)
    at org.hibernate.service.internal.AbstractServiceRegistryImpl.getService(AbstractServiceRegistryImpl.java:207)
    - locked <0x0000000375ff6dc8> (a org.hibernate.boot.registry.internal.StandardServiceRegistryImpl)
    at org.hibernate.boot.model.process.spi.MetadataBuildingProcess.handleTypes(MetadataBuildingProcess.java:352)
    at org.hibernate.boot.model.process.spi.MetadataBuildingProcess.complete(MetadataBuildingProcess.java:111)
    at org.hibernate.jpa.boot.internal.EntityManagerFactoryBuilderImpl.metadata(EntityManagerFactoryBuilderImpl.java:847)
    at org.hibernate.jpa.boot.internal.EntityManagerFactoryBuilderImpl.build(EntityManagerFactoryBuilderImpl.java:874)
    at org.jboss.as.jpa.hibernate5.TwoPhaseBootstrapImpl.build(TwoPhaseBootstrapImpl.java:44)
    at org.jboss.as.jpa.service.PersistenceUnitServiceImpl$1$1.run(PersistenceUnitServiceImpl.java:161)
    at org.jboss.as.jpa.service.PersistenceUnitServiceImpl$1$1.run(PersistenceUnitServiceImpl.java:121)
    at org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:667)
    at org.jboss.as.jpa.service.PersistenceUnitServiceImpl$1.run(PersistenceUnitServiceImpl.java:193)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
    at org.jboss.threads.JBossThread.run(JBossThread.java:320)

  • karthikkarthik Vertica Customer

    StackTrace:
    "ServerService Thread Pool -- 64" #184 prio=5 os_prio=0 tid=0x00007f74800ecdb0 nid=0x2828 runnable [0x00007f7364319000]
    java.lang.Thread.State: RUNNABLE
    at java.net.PlainSocketImpl.socketConnect(Native Method)
    at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
    - locked <0x0000000781711ae0> (a java.net.SocksSocketImpl)
    at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
    at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
    at java.net.Socket.connect(Socket.java:589)
    at java.net.Socket.connect(Socket.java:538)
    at com.vertica.io.VStream.establishConnection(Unknown Source)
    at com.vertica.io.VStream.(Unknown Source)
    at com.vertica.io.ProtocolStream.loadBalance(Unknown Source)
    at com.vertica.io.ProtocolStream.initSession(Unknown Source)
    at com.vertica.core.VConnection.tryConnect(Unknown Source)
    at com.vertica.core.VConnection.connect(Unknown Source)
    at com.vertica.jdbc.common.BaseConnectionFactory.doConnect(Unknown Source)
    at com.vertica.jdbc.common.AbstractDriver.connect(Unknown Source)
    at org.jboss.jca.adapters.jdbc.local.LocalManagedConnectionFactory.createLocalManagedConnection(LocalManagedConnectionFactory.java:321)

  • LenoyJLenoyJ - Select Field - Employee
    edited February 2020

    In classic connection load balancing, each node in the cluster can only be reached via a single IP address. This address is set in the EXPORT_ADDRESS column of the NODES system table.

    See if what is set in your export addresses matches what you expect the application server to connect to.
     
    The following docs talk about export addresses in the concept of exporting/importing data but it can be applied to load balancing too:
    https://www.vertica.com/docs/8.1.x/HTML/index.htm#Authoring/AdministratorsGuide/CopyExportData/ChangingNodeExportAddress.htm
    https://www.vertica.com/kb/Configuring-Network-to-Import-and-Export-Data/Content/BestPractices/Configuring-Network-to-Import-and-Export-Data.htm

  • karthikkarthik Vertica Customer

    Hi Lenoy,
    I have the below interfaces on each vertica node:
    if1 -> ip1(private)
    if2 -> ip2 (public)
    if3 -> ip3 (private)
    lo -> 127.0.0.1
    The export_address and node_address is set to if1 IP (as per nodes table)
    whereas the client (our application) is configured to connect to IPs of if3.
    This appears to be working fine for us otherwise as our application is also within the private network..
    However, when native load balancing (round-robin) is enabled, this is causing the problem.
    What is the reason vertica recommends having a public IP for clients to connect to vertica nodes ? One obvious reason i know is to not interfere the node-to-node communication in vertica cluster.. But is it mandatory that we should use a public IP or an interface that is not used by node-to-node cluster communication between vertica?
    Thanks,
    Karthik S

  • karthikkarthik Vertica Customer

    Thanks very much Lenoy. Will try that and get back

Leave a Comment

BoldItalicStrikethroughOrdered listUnordered list
Emoji
Image
Align leftAlign centerAlign rightToggle HTML viewToggle full pageToggle lights
Drop image/file