Vertica does not support Linux Logical Volume Manager (LVM).
Reason behind not using LVM is file corruption. For Vertica database data and catalog disks, we do not test it and have concerns with (and have seen) file corruption.
That is unfortunate. We were evaluating this product for a large Genomics project. The fact that Vertica does not support LVM is likely to knock it out of contention as other products do not have this restriction. Is HP planning on removing this limitation or is this core to the design?
Vertica will run on LVM. You can accomplish this by running the installer, pointing it at an LVM volume, and following the displayed instructions to ignore the warning that you receive.
That said, we don't provide support for the configuration.
The reason that we don't support it isn't fundamental to Vertica. It's because, among other reasons, many older versions of LVM (shipped with Linux distributions that we still support) have known limitations in them that can lead to data corruption. For one specific technical example, LVM did not properly/fully support write barriers prior to the Linux 2.6.33 kernel. (As always, vendor kernels may vary; that's the upstream version number.) This limitation affects all applications, including databases, running the affected kernels.
Vertica will work on just about anything, more so than many other products because of our design. (I've seen Vertica run on a FUSE-backed older version of encfs, for example. It didn't run well, but it ran, which is more than I can say for many applications on that particular installation...) But we have a high standard for what we list as supported.
Given all that -- many things are possible; but if you want a commitment, why trust me :-) Talk to your sales rep.
Out of curiosity, why is LVM important to your use case? The typical use cases for LVM are partitioning of disk space among multiple services on a single machine, and fast snapshot-based backup/restore. Vertica is intended for large data sets and workloads; you wouldn't generally use Vertica on part of a machine, you use it on part of a data center. For snapshots, our backup tool is able to perform fast, efficient snapshots on regular ext3/4.
Thanks for the response Adam, Glad to hear it. Can you point me to resources that will show me how to do the install on a host with LVM. To answer your other question, we are doing product evaluations at this time on our development platform. I work for an Academic organization (University of California San Diego). Our budget here is pretty thin so our dev platform emulates a cluster by using VMs. This is why we have LVM on our dev hosts. This is not used on our production nodes. Thanks again, Bill West
Ah, sure, I see -- we see lots of crazy setups used for dev purposes; LVM should be entirely reasonable there.
Vertica stores data in any regular filesystem path. (You are asked for the path during database creation.) So just format and mount your LVM partition; then install Vertica as usual. Beyond that, no special instructions should be required.
Can we get data corruption due dynamic i-nodes allocation?
Once I did a heavy test on VM with LVM and fs was resized due lack of space and data corruption occurred. Is it possible from point of view of Vertica? Or it something else? If Vertica do usage in "hard links"/"inodes"/"raw data reading"/"random access" and so on - is it possible? Can be it reason for all non-supported file systems with dynamic i-nodes allocation?
Vertica itself should be relatively robust to these sorts of things. That said, we write files; if the filesystem corrupts those files, then we're going to lose data. (I think that's basically true of any software.)
When you resize a partition, there are two things that have to happen. The partition itself has to be enlarged or shrunk; that's LVM's job. (What if there's no space immediately after the partition, on the physical hard disk? Then LVM has to allocate a new block of space somewhere else on the physical drive. And all disk reads/writes/etc have to go through LVM to figure out if they're in the first block or the second block; it needs a lookup table, etc.) Also, the filesystem itself (ie., ext4, etc) has to be resized; given more space for things like additional inodes or other metadata structures, data moved in from the end of the disk if the FS is being shrunk, etc.
Both steps are a little bit tricky. The code (in LVM and in the filesystem, not Vertica) needs to be robust to things like the computer crashing in the middle of the operation, people concurrently accessing the system while it's in the middle of a resize, etc. So it's not trivial code to write; it's quite possible for LVM or the filesystem to have a bug that corrupts your data under the right circumstances.
In theory, both should be totally transparent to software that uses the basic POSIX filesystem operations (open(), read(), write(), close(); etc). I can't comment on exactly how file I/O works in Vertica, but we are careful to write code that is robust in this respect; so in theory we should work perfectly on LVM.
In practice, though, all software has bugs :-)
LVM adds complexity and risk. It also adds functionality. I can't comment on the "supported"/"not supported" decision. But if you don't need the functionality, I'd say, don't add the complexity.
So it's not trivial code to write; it's quite possible for LVM or the filesystem to have a bug that corrupts your data under the right circumstances.
Exactly this I mean by problem with "dynamic i-nodes", and not just LVM. XFS, LVM - has incidents with data corruption.(RaiseFS didn't heard O_o).
PS Under heavy workload can journal do a mistake? And if to disable async writing or disk caching? Its not easy to reproduce this problem Tests probably will show nothing.
Hm... It's certainly possible. It'd be a filesystem or LVM bug if so.
Disabling/enabling async writing or caching would most likely have effects that show up if your system crashes/reboots in the middle of an operation. So you could try that to reproduce.
These bugs can also be hardware-dependent -- sometimes the drive has hardware problems, sometimes the drive has unusual behavior that tickles a bug in the FS.
2014-01-02 13:29:54.257 Main:0x1fcb2db0 <WARNING> @v_dev_node0003: 01000/3938: MiniRos 54043196826332083 does not have proper SAL files 2014-01-02 13:29:54.278 Main:0x1fcb2db0 <PANIC> @v_dev_node0003: VX001/2973: Data consistency problems found; startup aborted HINT: Check that all file systems are properly mounted. Also, the --force option can be used to delete corrupted data and recover from the cluster LOCATION: mainEntryPoint, /scratch_a/release/vbuild/vertica/Basics/vertica.cpp:1082 2014-01-02 13:29:54.305 Main:0x1fcb2db0 [Main] <PANIC> Wrote backtrace to ErrorReport.txt
why it ask for "Check that all file systems are properly mounted" ? Soft links? No. Not mounted additional location? May be, but container exists its SAL not. So why? Does journal didn't create a hard link? Does it Vertica didn't give a name to pointer and it utilized? Or its hard link points to wrong memory sector? If I could to reproduce it
Oh, that HINT is because users often put Vertica's data directory onto a dedicated partition and sometimes that partition isn't mounted. Or, more insidiously, sometimes the wrong partition is mounted; maybe an old backup partition got mounted in place of the actual data, or something else weird. So it looks like a valid Vertica data directory but has the wrong data in it.
The error message indicates that the data store failed a basic sanity check; a data file that the catalog expects to be there is either not there or is corrupt/unusable. The hint points at one common (and easily-missed) cause. But it could be any number of reasons.
This can also occur if your system shut down uncleanly and lost data somehow as a result. Depends on system configuration, etc; there are many ways to configure a Linux filesystem s.t. recent writes can be lost in case of a sudden crash or power loss. If that's the case, start with "--force" as instructed; we'll roll back recent/lost changes and restore the DB to a consistent state.
Adam, Isn't there something in the installation script that checks for LVM mount points? This is where I am having a problem. Can I just edit the script and comment out this check? Thanks, bw
You said to accomplish this challenge, run the installer, which installer did you mean?
And you also said, HP will not this configuration, correct?!
Thanks!
aseering wrote: Hi Bill,
Vertica will run on LVM. You can accomplish this by running the installer, pointing it at an LVM volume, and following the displayed instructions to ignore the warning that you receive.
That said, we don't provide support for the configuration.
The reason that we don't support it isn't fundamental to Vertica. It's because, among other reasons, many older versions of LVM (shipped with Linux distributions that we still support) have known limitations in them that can lead to data corruption. For one specific technical example, LVM did not properly/fully support write barriers prior to the Linux 2.6.33 kernel. (As always, vendor kernels may vary; that's the upstream version number.) This limitation affects all applications, including databases, running the affected kernels.
Vertica will work on just about anything, more so than many other products because of our design. (I've seen Vertica run on a FUSE-backed older version of encfs, for example. It didn't run well, but it ran, which is more than I can say for many applications on that particular installation...) But we have a high standard for what we list as supported.
Given all that -- many things are possible; but if you want a commitment, why trust me :-) Talk to your sales rep.
Out of curiosity, why is LVM important to your use case? The typical use cases for LVM are partitioning of disk space among multiple services on a single machine, and fast snapshot-based backup/restore. Vertica is intended for large data sets and workloads; you wouldn't generally use Vertica on part of a machine, you use it on part of a data center. For snapshots, our backup tool is able to perform fast, efficient snapshots on regular ext3/4.
Comments
Vertica does not support Linux Logical Volume Manager (LVM).
Reason behind not using LVM is file corruption. For Vertica database data and catalog disks, we do not test it and have concerns with (and have seen) file corruption.
Regards
Rahul
Thanks,
Bill West
Vertica will run on LVM. You can accomplish this by running the installer, pointing it at an LVM volume, and following the displayed instructions to ignore the warning that you receive.
That said, we don't provide support for the configuration.
The reason that we don't support it isn't fundamental to Vertica. It's because, among other reasons, many older versions of LVM (shipped with Linux distributions that we still support) have known limitations in them that can lead to data corruption. For one specific technical example, LVM did not properly/fully support write barriers prior to the Linux 2.6.33 kernel. (As always, vendor kernels may vary; that's the upstream version number.) This limitation affects all applications, including databases, running the affected kernels.
Vertica will work on just about anything, more so than many other products because of our design. (I've seen Vertica run on a FUSE-backed older version of encfs, for example. It didn't run well, but it ran, which is more than I can say for many applications on that particular installation...) But we have a high standard for what we list as supported.
Given all that -- many things are possible; but if you want a commitment, why trust me :-) Talk to your sales rep.
Out of curiosity, why is LVM important to your use case? The typical use cases for LVM are partitioning of disk space among multiple services on a single machine, and fast snapshot-based backup/restore. Vertica is intended for large data sets and workloads; you wouldn't generally use Vertica on part of a machine, you use it on part of a data center. For snapshots, our backup tool is able to perform fast, efficient snapshots on regular ext3/4.
Adam
Ah, sure, I see -- we see lots of crazy setups used for dev purposes; LVM should be entirely reasonable there.
Vertica stores data in any regular filesystem path. (You are asked for the path during database creation.) So just format and mount your LVM partition; then install Vertica as usual. Beyond that, no special instructions should be required.
Adam
Can we get data corruption due dynamic i-nodes allocation?
Once I did a heavy test on VM with LVM and fs was resized due lack of space and data corruption occurred. Is it possible from point of view of Vertica? Or it something else? If Vertica do usage in "hard links"/"inodes"/"raw data reading"/"random access" and so on - is it possible? Can be it reason for all non-supported file systems with dynamic i-nodes allocation?
Daniel
Vertica itself should be relatively robust to these sorts of things. That said, we write files; if the filesystem corrupts those files, then we're going to lose data. (I think that's basically true of any software.)
When you resize a partition, there are two things that have to happen. The partition itself has to be enlarged or shrunk; that's LVM's job. (What if there's no space immediately after the partition, on the physical hard disk? Then LVM has to allocate a new block of space somewhere else on the physical drive. And all disk reads/writes/etc have to go through LVM to figure out if they're in the first block or the second block; it needs a lookup table, etc.) Also, the filesystem itself (ie., ext4, etc) has to be resized; given more space for things like additional inodes or other metadata structures, data moved in from the end of the disk if the FS is being shrunk, etc.
Both steps are a little bit tricky. The code (in LVM and in the filesystem, not Vertica) needs to be robust to things like the computer crashing in the middle of the operation, people concurrently accessing the system while it's in the middle of a resize, etc. So it's not trivial code to write; it's quite possible for LVM or the filesystem to have a bug that corrupts your data under the right circumstances.
In theory, both should be totally transparent to software that uses the basic POSIX filesystem operations (open(), read(), write(), close(); etc). I can't comment on exactly how file I/O works in Vertica, but we are careful to write code that is robust in this respect; so in theory we should work perfectly on LVM.
In practice, though, all software has bugs :-)
LVM adds complexity and risk. It also adds functionality. I can't comment on the "supported"/"not supported" decision. But if you don't need the functionality, I'd say, don't add the complexity.
Adam
Thanks, totally agree with you. Exactly this I mean by problem with "dynamic i-nodes", and not just LVM.
XFS, LVM - has incidents with data corruption.(RaiseFS didn't heard O_o).
PS
Under heavy workload can journal do a mistake? And if to disable async writing or disk caching? Its not easy to reproduce this problem
Disabling/enabling async writing or caching would most likely have effects that show up if your system crashes/reboots in the middle of an operation. So you could try that to reproduce.
These bugs can also be hardware-dependent -- sometimes the drive has hardware problems, sometimes the drive has unusual behavior that tickles a bug in the FS.
That is what Im trying to understand.
For example: why it ask for "Check that all file systems are properly mounted" ? Soft links? No. Not mounted additional location? May be, but container exists its SAL not. So why? Does journal didn't create a hard link? Does it Vertica didn't give a name to pointer and it utilized? Or its hard link points to wrong memory sector?
If I could to reproduce it
The error message indicates that the data store failed a basic sanity check; a data file that the catalog expects to be there is either not there or is corrupt/unusable. The hint points at one common (and easily-missed) cause. But it could be any number of reasons.
This can also occur if your system shut down uncleanly and lost data somehow as a result. Depends on system configuration, etc; there are many ways to configure a Linux filesystem s.t. recent writes can be lost in case of a sudden crash or power loss. If that's the case, start with "--force" as instructed; we'll roll back recent/lost changes and restore the DB to a consistent state.
Could you copy/paste and post the full error text that you're getting? (Also, what version of Vertica are you using?)
Adam
https://community.vertica.com/vertica/topics/error_when_creating_database_after_clean_install_of_ver...
(He had a non-LVM-related error but was able to fix it himself in a follow-up post.)
Take note of the error messages; also of the exact command line used to invoke install_vertica.
(The install scripts were re-written between version 6 and 7; that's why I asked which version you're using.)
Hi Adam,
You said to accomplish this challenge, run the installer, which installer did you mean?
And you also said, HP will not this configuration, correct?!
Thanks!
Charlize
Hi Adam,
Missed the word "support", my apolology!
HP will not support this configuration, correct?!
Thanks!
Charlize
Which version of Vertica?
For RHL 7.4 you'll need Vertica 8.1.1-7 or later.
See:
https://my.vertica.com/docs/8.1.x/HTML/index.htm#Authoring/SupportedPlatforms/MCandServer.htm?TocPath=Vertica%208.1.x%20Supported%20Platforms|_____1
Also, your LVM version must be 2.02.66 or later, and must include device-mapper version 1.02.48 or later.
See:
https://my.vertica.com/docs/8.1.x/HTML/index.htm#Authoring/SupportedPlatforms/MCandServer.htm