Load balancing with bigIP. We set it up following the document. But having problems with some reports (connection closed). Anyone else having the same issues?
Thanks for letting us know the troubles you've had Yongzhi! I have passed your problem along to our support team and they will respond to you here shortly.
Hi Yongzhi. Here are a few things to check. I am assuming that you are using a VIP for your connections. It is possible that the TCP KEEPALIVE setting needs to be changed. To check the setting use cat /proc/sys/net/ipv4/tcp_keepalive_time Setting the KEEPALIVE setting to a value lower than the firewall timeout might help. In the following example, it is being set to 10 minutes (600 seconds). echo 600 > /proc/sys/net/ipv4/tcp_keepalive_time The other suggestion is to confirm that spread.pl is running Check that the spread.pl is running - spread.pl updates the check.txt file with information as to which nodes are running and which are down. Example: cat /etc/keepalived/check.txt N192168150XX6 N192168150XX4 N192168150XX3 $ ps -ef | grep spread.pl $ sudo /etc/init.d/spread.pl start The last thing to do is to check the current timeout values: sudo ipvsadm -l --timeout Timeout (tcp tcpfin udp): 900 120 300 To change the tcp setting to a longer time, run the following to increase it to 8400 seconds. Using 0 for the other 2 arguments will leave them as the defaults. Example sudo ipvsadm --set 8400 0 0 Amy
Thanks for your quick reply. Our Vertica is 6.1 running on RedHat linux. I will check the VIP as you suggested. If the app (with connection pooling) is pointing to vertica nodes (we have three) directly, there are no problems. Only when goes through the VIP, some reports got error like" Type : System.ObjectDisposedException, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 Message : Cannot access a closed Stream. Source : Orchestrator Help link : ObjectName : Data : System.Collections.ListDictionaryInternal TargetSite : System.Data.DataSet ExecuteReportQueries(Edu.UHC.CDP.CDPJobRequest) Stack Trace : at Edu.UHC.CDP.JobExecuteWorker.ExecuteReportQueries(CDPJobRequest request) at Edu.UHC.CDP.JobExecuteWorker.ProcessJob() ". The VIP administrator asked for persistance values (setting). All the values are browser based. We have app servers in between. My gut feeling is that there were mismatches in the communication (VIP routed results to the wrong connection). Is that possible? Thanks, Yongzhi
It could be possible, I haven't seen these errors before. I think the focus should be on the message: cannot connect to a closed stream. I will see if I can find anything about this message.
It could be possible, I haven't seen these errors before. I think the focus should be on the message: cannot connect to a closed stream. I will see if I can find anything about this message.
Yongzhi, Did you have any luck in finding the solution to the "Cannot access a closed Stream." message? Did not find anything internal. I did find this on Google. You can't reopen the stream. If you need to "reset" the stream, just assign it a new instance: memoryStream = new MemoryStream();
Thanks, Amy. We tried a few things with VIP. Still had random errors. So the nodes are hard coded in the application. No such errors any more. Not an ideal solution. But we have to live with it for now. Thanks very much for your help. Yongzhi
Comments