Loading data from non-AWS S3

gytisgytis Registered User

Hi,

is there any way to load data into Vertica from a non AWS service that has Amazon S3 compatible API? For example CEPH, or something similar?

It seems that it's almost possible, except Vertica kinda depends on AWSRegion setting to figure out where to connect. Is there any workaround for this? Has anyone had any luck setting up something similar?

(We're using Vertica 9.1)

Answers

  • Jim_KnicelyJim_Knicely Administrator, Moderator, Employee, Registered User, VerticaExpert

    Maybe create and mount a Ceph Filesystem?

    See:
    http://docs.ceph.com/docs/kraken/cephfs/

  • gytisgytis Registered User

    Would strongly prefer to keep reuse the same loading as from Amazon S3. Is the code for the AWS loader available anywhere? We could probably easily modify the UDSource to fit both.

    Re: loading from file system - doable in principle, but if we are forced to ditch the initial idea of loading data using Vertica itself - we'll probably turn to something different. A python or scala script to accept generic S3 bucket. Mounting a CEPH filesystem is maybe a bit too fragile.

Leave a Comment

BoldItalicStrikethroughOrdered listUnordered list
Emoji
Image
Align leftAlign centerAlign rightToggle HTML viewToggle full pageToggle lights
Drop image/file