How to mount Amazon S3 bucket with Goofys?

While experimenting with s3fs I have also found out another tool written in go programming language that named Goofys. Like s3fs, this tool can also mount Amazon S3 bucket as a folder that can later be accessed just like any other local folder making accessing files in S3 bucket, backing up data or syncing files quite easy. Detailed installation manual for Goofys covers only Mac while Linux installation guide is basically non-existing, so this will be my guide on how to install Goofys and mount Amazon S3 bucket on an Ubuntu Linux. This has been tested on DigitalOcean droplet running a Ubuntu version 16.04.3 x64. If you’re also testing this out on a new VPS like me – make sure to execute apt-get update  before you start.

But why would anyone want to use Goofys if s3fs is working fine? Well, according to benchmarks available at Goofys github main reason you would like to use this is performance! Goofys is much faster than s3fs. For some operations Goofys is ten times faster than s3fs making it the right choice for situations when you need access to S3 to be really fast (or as close as it being just a folder on the same drive).

How to install Go language on Ubuntu Linux?

As mentioned above Goofy is written in GO so we need to install GO in order to use Goofy. Installing go requires you to download the latest version from https://golang.org/dl/ unpack it:

<code># Download Go language binary
wget https://storage.googleapis.com/golang/go1.9.3.linux-amd64.tar.gz

# Unpack the downloaded fole to /usr/local
tar -C /usr/local -xvzf go1.9.3.linux-amd64.tar.gz

# Remove the archive
rm go1.9.3.linux-amd64.tar.gz

# Create the Go language work directories
mkdir -p /go/projects/goofys/{bin,pkg,src}</code>

No create this file path.sh at /etc/profile.d path

<code>vi /etc/profile.d/path.sh</code>

copy paste the line below, save the changes and exit:

<code>export PATH=$PATH:/usr/local/go/bin</code>

Now edit ~/.bash_profile file:

<code>vi .bash_profile</code>

and copy paste these two lines, save and close:

<code>export GOBIN=/go/projects/goofys/bin/
export GOPATH=/go/projects/goofys/</code>

To apply these changes to your current bash session source both scripts

<code>source /etc/profile &amp;&amp; source ~/.bash_profile</code>

How to install AWS CLI on Ubuntu Linux?

AWS CLI is required by Goofy and installation is pretty straight forward:

<code>apt-get install awscli</code>

How to install Gooofy on Ubuntu Linux?

Once Go language is installed correctly, installing Goofy is quite simple. All you need to do is simply execute these two commands:

<code>go get github.com/kahing/goofys
go install github.com/kahing/goofys</code>

At this point Goofy should be installed and in order to use it we need to save

Setting up Amazon S3 credentials

To allow AWS CLI to access your S3 buckets create store credentials at this file:

<code># Make the hidden folder that holds the AWS credentials file
mkdir ~/.aws

# Create and start editing the credentials file
vi ~/.aws/credentials</code>

In the following format:

<code>[default]
aws_access_key_id=REPLACE_THIS_WITH_YOUR_ACCESS_KEY
aws_secret_access_key=REPLACE_THIS_WITH_YOUR_SECRET_KEY</code>

Don’t forget to copy/paste your actual AWS access and secret keys above. Save and exit once you’re done.

Check if AWS S3 is working

To verify if the credentials are right and that you have entered them correctly, list all the buckets using:

<code>aws s3 ls</code>

The response should show the list of available buckets.

For this test I have created a new bucket called GeekTnT

Mounting a Amazon S3 bucket  with Gooofy

Now let’s create a folder where we will mount our bucket. For the purpose of this post we’ll create a folder /backup and we’ll mount GeekTnT bucket on it.

<code># Creates a backup folder
mkdir /backup

# Mount a GeekTNT.com bucket to a /backup folder
/go/projects/goofys/bin/goofys GeekTnT /backup</code>

And the final result is that now we have an s3 bucket mounted to our /backup folder with basically unlimited disk space (1PB) that we can use to store data or backups.

Unmounting the S3 bucket is simple. Just execute:

<code>umount /backup</code>

Checking if it actually works

Everything seems fine – but does it work. To test that out I’ll create a file in /backup folder and since this is a new DigitalOcean droplet and I have no bigger files that I can copy, I’ll download a Go language installation file directly into /backup folder.

If everything works these should automatically be available inside Amazon’s console.

… and there it goes.

It works!

Tags: 5894, backgai, goofys for S3, goofys s3, https://geektnt com/how-to-mount-amazon-s3-bucket-with-goofys html, soldgx5

Why .dev domains automatically redirect to HTTPS?

If you’re a web developer developing in local environment you most probably use .dev for projects you are working on. If you’re using Valet on Mac development is quite simple since every folder from your projects folder is instantly accessible in browser if you just enter folder name and add .dev. It works like magic without messing with any hosts files. It just works!

But then Google decided to complicate things a bit. First they have forced all .dev domains to use SSL. Since December 2017 Google Chrome version 63 started redirecting to all .dev domains to secure protocol replacing http:// with https:// in your urls without even asking you. The reason for that is maybe somehow related to fact that Google bought the .dev making it a legit gTLD. They don’t sell these (yet) and the use for this, and for more than 100 other gTLD Google now owns (including .app, .foo), is unknown but for sure it’s going to complicate lives of web developers.

Continue Reading Tags: redirect dev domains to https


How to DNS cache on Windows or macOS?

Almost all operating systems cache DNS records to improve the performance of applications and speed up internet access. Browsers do cache DNS records too and to flush these the easiest way is just to close the browser and start ti again. But sometimes there is a need to flush operating system DNS cache. Here’s how you can do that easily.

How to flush DNS cache on Windows XP / Windows Vista?

  • Step 1: Open the Command Prompt
    • Click on the Start Menu and click Run. Type in cmd and hit Enter.
  • Step 2: Flush DNS
    • Type ipconfig /flushdns and hit Enter.

 

How to flush DNS cache on Windows Vista or Windows 7?

  • Step 1: Open the Command Prompt
    • Click on the Start Menu and type cmd in the search bar and hit Enter.
  • Step 2: Flush DNS
    • Type ipconfig /flushdns and hit Enter.

Continue Reading Tags: flush dns in filezilla, nscd caching in windows


After upgrading mac from Sierra to High Sierra git no longer works

Recently I have upgraded your macOS on my MacBook Air from Sierra to High Sierra. After the update (that took some time to complete) everything seemed to be working fine… until I was about to do some more coding and have I tried to execute git. I’ve got this error:

xcrun: error: invalid active developer path (/Library/Developer/CommandLineTools), missing xcrun at: /Library/Developer/CommandLineTools/usr/bin/xcrun

Continue Reading