Mountpoint for Amazon S3 – Usually Accessible and Prepared for Manufacturing Workloads

Mountpoint for Amazon S3 is an open supply file consumer that makes it straightforward on your file-aware Linux purposes to attach on to Amazon Easy Storage Service (Amazon S3) buckets. Introduced earlier this yr as an alpha launch, it’s now usually out there and prepared for manufacturing use in your large-scale read-heavy purposes: knowledge lakes, machine studying coaching, picture rendering, autonomous automobile simulation, ETL, and extra. It helps file-based workloads that carry out sequential and random reads, sequential (append solely) writes, and that don’t want full POSIX semantics.

Why Recordsdata?
Many AWS prospects use the S3 APIs and the AWS SDKs to construct purposes that may listing, entry, and course of the contents of an S3 bucket. Nevertheless, many purchasers have current purposes, instructions, instruments, and workflows that know tips on how to entry recordsdata in UNIX type: studying directories, opening & studying current recordsdata, and creating & writing new ones. These prospects have requested us for an official, enterprise-ready consumer that helps performant entry to S3 at scale. After talking with these prospects and asking a lot of questions, we discovered that efficiency and stability have been their main issues, and that POSIX compliance was not a necessity.

After I first wrote about Amazon S3 again in 2006 I used to be very clear that it was meant for use as an object retailer, not as a file system. Whereas you wouldn’t need use the Mountpoint / S3 combo to retailer your Git repositories or the like, utilizing it along with instruments that may learn and write recordsdata, whereas benefiting from S3’s scale and sturdiness, is smart in lots of conditions.

All About Mountpoint
Mountpoint is conceptually quite simple. You create a mount level and mount an Amazon S3 bucket (or a path inside a bucket) on the mount level, after which entry the bucket utilizing shell instructions (ls, cat, dd, discover, and so forth), library capabilities (open, shut, learn, write, creat, opendir, and so forth) or equal instructions and capabilities as supported within the instruments and languages that you just already use.

Below the covers, the Linux Digital Filesystem (VFS) interprets these operations into calls to Mountpoint, which in turns interprets them into calls to S3: LIST, GET, PUT, and so forth. Mountpoint strives to make good use of community bandwidth, growing throughput and permitting you to scale back your compute prices by getting extra work accomplished in much less time.

Mountpoint can be utilized from an Amazon Elastic Compute Cloud (Amazon EC2) occasion, or inside an Amazon Elastic Container Service (Amazon ECS) or Amazon Elastic Kubernetes Service (EKS) container. It may also be put in in your current on-premises programs, with entry to S3 both instantly or over an AWS Direct Join connection by way of AWS PrivateLink for Amazon S3.

Putting in and Utilizing Mountpoint for Amazon S3
Mountpoint is out there in RPM format and may simply be put in on an EC2 occasion operating Amazon Linux. I merely fetch the RPM and set up it utilizing yum:

$ wget https://s3.amazonaws.com/mountpoint-s3-release/newest/x86_64/mount-s3.rpm
$ sudo yum set up ./mount-s3.rpm

For the final couple of years I’ve been usually fetching photographs from a number of of the Washington State Ferry webcams and storing them in my wsdot-ferry bucket:

I accumulate these photographs with the intention to observe the comings and goings of the ferries, with a purpose of analyzing them in some unspecified time in the future to seek out one of the best occasions to experience. My purpose right this moment is to create a film that mixes a whole day’s value of photographs into a pleasant time lapse. I begin by making a mount level and mounting the bucket:

$ mkdir wsdot-ferry
$  mount-s3 wsdot-ferry wsdot-ferry

I can traverse the mount level and examine the bucket:

$ cd wsdot-ferry
$ ls -l | head -10
complete 0
drwxr-xr-x 2 jeff jeff 0 Aug  7 23:07 2020_12_30
drwxr-xr-x 2 jeff jeff 0 Aug  7 23:07 2020_12_31
drwxr-xr-x 2 jeff jeff 0 Aug  7 23:07 2021_01_01
drwxr-xr-x 2 jeff jeff 0 Aug  7 23:07 2021_01_02
drwxr-xr-x 2 jeff jeff 0 Aug  7 23:07 2021_01_03
drwxr-xr-x 2 jeff jeff 0 Aug  7 23:07 2021_01_04
drwxr-xr-x 2 jeff jeff 0 Aug  7 23:07 2021_01_05
drwxr-xr-x 2 jeff jeff 0 Aug  7 23:07 2021_01_06
drwxr-xr-x 2 jeff jeff 0 Aug  7 23:07 2021_01_07
$
$  cd 2020_12_30
$ ls -l
complete 0
drwxr-xr-x 2 jeff jeff 0 Aug  7 23:07 fauntleroy_holding
drwxr-xr-x 2 jeff jeff 0 Aug  7 23:07 fauntleroy_way
drwxr-xr-x 2 jeff jeff 0 Aug  7 23:07 lincoln
drwxr-xr-x 2 jeff jeff 0 Aug  7 23:07 trenton
drwxr-xr-x 2 jeff jeff 0 Aug  7 23:07 vashon_112_north
drwxr-xr-x 2 jeff jeff 0 Aug  7 23:07 vashon_112_south
drwxr-xr-x 2 jeff jeff 0 Aug  7 23:07 vashon_bunker_north
drwxr-xr-x 2 jeff jeff 0 Aug  7 23:07 vashon_bunker_south
drwxr-xr-x 2 jeff jeff 0 Aug  7 23:07 vashon_holding
$
$ cd fauntleroy_holding
$  ls -l | head -10
complete 2680
-rw-r--r-- 1 jeff jeff  19337 Feb 10  2021 17-12-01.jpg
-rw-r--r-- 1 jeff jeff  19380 Feb 10  2021 17-15-01.jpg
-rw-r--r-- 1 jeff jeff  19080 Feb 10  2021 17-18-01.jpg
-rw-r--r-- 1 jeff jeff  17700 Feb 10  2021 17-21-01.jpg
-rw-r--r-- 1 jeff jeff  17016 Feb 10  2021 17-24-01.jpg
-rw-r--r-- 1 jeff jeff  16638 Feb 10  2021 17-27-01.jpg
-rw-r--r-- 1 jeff jeff  16713 Feb 10  2021 17-30-01.jpg
-rw-r--r-- 1 jeff jeff  16647 Feb 10  2021 17-33-02.jpg
-rw-r--r-- 1 jeff jeff  16750 Feb 10  2021 17-36-01.jpg
$

I can create my animation with a single command:

$ ffmpeg -framerate 10 -pattern_type glob -i "*.jpg" ferry.gif

And right here’s what I get:

As you may see, I used Mountpoint to entry the present picture recordsdata and to jot down the newly created animation again to S3. Whereas this can be a pretty easy demo, it does present how you should utilize your current instruments and expertise to course of objects in an S3 bucket. On condition that I’ve collected a number of million photographs over time, with the ability to course of them with out explicitly syncing them to my native file system is a giant win.

Mountpoint for Amazon S3 Details
Listed below are a few issues to bear in mind when utilizing Mountpoint:

Pricing – There are not any new costs for using Mountpoint; you pay just for the underlying S3 operations. It’s also possible to use Mountpoint to entry requester-pays buckets.

Efficiency – Mountpoint is ready to reap the benefits of the elastic throughput supplied by S3, together with knowledge switch at as much as 100 Gb/second between every EC2 occasion and S3.

Credentials – Mountpoint accesses your S3 buckets utilizing the AWS credentials which can be in impact whenever you mount the bucket. See the CONFIGURATION doc for extra data on credentials, bucket configuration, use of requester pays, some ideas for using S3 Object Lambda, and extra.

Operations & Semantics – Mountpoint helps fundamental file operations, and may learn recordsdata as much as 5 TB in measurement. It will probably listing and skim current recordsdata, and it could possibly create new ones. It can’t modify current recordsdata or delete directories, and it doesn’t help symbolic hyperlinks or file locking (if you happen to want POSIX semantics, check out Amazon FSx for Lustre). For extra details about the supported operations and their interpretation, learn the SEMANTICS doc.

Storage Lessons – You should utilize Mountpoint to entry S3 objects in all storage lessons besides S3 Glacier Versatile Retrieval, S3 Glacier Deep Archive, S3 Clever-Tiering Archive Entry Tier, and S3 Clever-Tiering Deep Archive Entry Tier.

Open Supply – Mountpoint is open supply and has a public roadmap. Your contributions are welcome; remember to learn our Contributing Pointers and our Code of Conduct first.

Hop On
As you may see, Mountpoint is absolutely cool and I’m guessing that you’re going to discover some superior methods to place it to make use of in your purposes. Test it out and let me know what you suppose!

— Jeff;