initializing infinite loop

Brian_Lau

Hello, I'm trying to restart my vertica cluster and my vertica.log shows this repeating loop. Any ideas? 2014-01-24 02:54:39.384 Init Session:0x7f3ba800f540 @v_gaikai_node0007: {SessionRun} 57V03/5785: Cluster Status Request by 10.0.28.86:36968 HINT: Cluster State: gaikai INITIALIZING: 1 of 12 (v_gaikai_node0007) ---- LOCATION: initSession, /scratch_a/release/vbuild/vertica/Session/ClientSession.cpp:429 2014-01-24 02:54:45.000 Timer Service:0x7e42490 [Util] Task 'LowDiskSpaceCheck' enabled 2014-01-24 02:54:46.214 Poll dispatch:0x80922b0 [Comms] Failed to send spmonitor request: Invalid argument 2014-01-24 02:54:55.000 Timer Service:0x7e42490 [Util] Task 'FeatureUseLogger' enabled 2014-01-24 02:54:55.000 Timer Service:0x7e42490 [Util] Task 'LicenseSizeAuditor' enabled 2014-01-24 02:54:55.000 Timer Service:0x7e42490 [Util] Task 'LowDiskSpaceCheck' enabled 2014-01-24 02:54:56.223 Poll dispatch:0x80922b0 [Comms] Failed to send spmonitor request: Invalid argument 2014-01-24 02:54:57.440 Init Session:0x7f3ba800f540 @v_gaikai_node0007: 00000/2705: Connection received: host=10.0.28.86 port=36972 (connCnt 1) 2014-01-24 02:54:57.440 Init Session:0x7f3ba800f540 @v_gaikai_node0007: 00000/4540: Received SSL negotiation startup packet 2014-01-24 02:54:57.440 Init Session:0x7f3ba800f540 @v_gaikai_node0007: 00000/4691: Sending SSL negotiation response 'N' 2014-01-24 02:54:57.440 Init Session:0x7f3ba800f540 @v_gaikai_node0007: {SessionRun} 57V03/5785: Cluster Status Request by 10.0.28.86:36972 HINT: Cluster State: gaikai INITIALIZING: 1 of 12 (v_gaikai_node0007) ---- LOCATION: initSession, /scratch_a/release/vbuild/vertica/Session/ClientSession.cpp:429 2014-01-24 02:54:57.477 Init Session:0x7f3ba800f540 @v_gaikai_node0007: 00000/2705: Connection received: host=10.0.28.86 port=36976 (connCnt 1) 2014-01-24 02:54:57.477 Init Session:0x7f3ba800f540 @v_gaikai_node0007: 00000/4540: Received SSL negotiation startup packet 2014-01-24 02:54:57.477 Init Session:0x7f3ba800f540 @v_gaikai_node0007: 00000/4691: Sending SSL negotiation response 'N' 2014-01-24 02:54:57.477 Init Session:0x7f3ba800f540 @v_gaikai_node0007: {SessionRun} 57V03/5785: Cluster Status Request by 10.0.28.86:36976 HINT: Cluster State: gaikai INITIALIZING: 1 of 12 (v_gaikai_node0007) ---- LOCATION: initSession, /scratch_a/release/vbuild/vertica/Session/ClientSession.cpp:429 2014-01-24 02:54:59.516 Init Session:0x7f3ba800f540 @v_gaikai_node0007: 00000/2705: Connection received: host=10.0.28.86 port=36980 (connCnt 1) 2014-01-24 02:54:59.516 Init Session:0x7f3ba800f540 @v_gaikai_node0007: 00000/4540: Received SSL negotiation startup packet 2014-01-24 02:54:59.516 Init Session:0x7f3ba800f540 @v_gaikai_node0007: 00000/4691: Sending SSL negotiation response 'N' 2014-01-24 02:54:59.517 Init Session:0x7f3ba800f540 @v_gaikai_node0007: {SessionRun} 57V03/5785: Cluster Status Request by 10.0.28.86:36980 HINT: Cluster State: gaikai INITIALIZING: 1 of 12 (v_gaikai_node0007) ---- LOCATION: initSession, /scratch_a/release/vbuild/vertica/Session/ClientSession.cpp:429

Matteo_Durighet

Same problem. Did you find a solution?

[Deleted User]

This is just the startup script polling to see whether the server process on the node in question has started up yet. It will keep going until the node in question reports that it is ready.

Could you post some more of "vertica.log" in the catalog directory of the node in question?

Vertica can take a long time to start up, depending on configuration/etc. How long have you waited?

Brian_Lau

Matt, Vertica support worked with me for several days with no luck, until somehow rebooting the machines helped and the problem went away. One problem was that the cluster thought that there were several other servers (which I tries and failed to add previously). With the help of support, I removed the oids for the extra servers from each machine in the cluster. We made sure spread and admintools configs were correct throughout the cluster. If reboot doesn't help and you don't have a support contract, I will be happy to go though my old emails to check if there was anything else we tried.

Matteo_Durighet

I try the reboot, and now go! 
By the way this was a test installation on AWS with a cluster and what I change during installation the spread from broadcast to point to point and now after creation of database and after a reboot, it works.

Fortunately it was a new test db for testing vertica 7.0.0-1

Daniel_Leybovic

BTW its in docs:

--point-to-point, -T

Configures spread to use direct point-to-point communication between all HP Vertica nodes. You should use this option if your nodes aren't located on the same subnet. You should also use this option for all virtual environment installations, regardless of whether the virtual servers are on the same subnet or not. Cannot be used with --broadcast, as the setting must be either --broadcast or --point-to-point

Important: When changing the configuration from --broadcast (the default) to --point-to-point or from --point-to-point to --broadcast, the --control-network parameter must also be used.

Note: Spread always runs on UDP. -T does not denote TCP.

Matteo_Durighet

Yeah Daniel, it's true, but I think the problem it's not the "virtual enviroment", but the fact that the broadcast or multicast in a AWS VPC it's not supported or not working well. In vmware the broadcast now works, and it's supported.

Daniel_Leybovic

Hi!

Did you checked a hard drives?
From my experience Amazon gives bad disks for Vertica image.
If you don't mind, can you share your results for hdd throughput and latency ?

vioperf
https://my.vertica.com/docs/7.0.x/HTML/index.htm#Authoring/InstallationGuide/scripts/vioperf.htm

MEASURE_LOCATION_PERFORMANCE
https://my.vertica.com/docs/7.0.x/HTML/index.htm#Authoring/SQLReferenceManual/Functions/VerticaFunct...

[Deleted User]

If you are running Vertica on Amazon, we recommend using our prepackaged AMI. You're correct that Amazon does not provide broadcast networking. (We say "all virtual environments" because most don't suppirt broadcast, and those that do aren't necessarily much better than our unicast mode.) There are various other issues, mostly to do with disk and networking performance, that we try to work out for you with our prepackaged AMI and instructions. If you want to build an image on your own, be aware that there's a whole lot you can do to performance-tune your cluster. Our documentation has a few pointers; there are also many sites online that talk about EC2 performance.

Matteo_Durighet

Yes, I will do some stress test to see how  it is on AWS. By the way frst I have to test some compatibility and design things, after sure I wil do some stress testing , but with better disk (optimized i/o and so on ).


Thanks to all.