How to mount DigitalOcean Space on CentOS 7

DigitalOcean S3 compatible object storage called Spaces allows you to store and serve large amounts of data. They are easy to create and are ready to be used within seconds without any configuration. They are ideal for storing static unstructured data like audio, video and images as well as binary files, archives, backups etc.

What’s so cool about Spaces compared to Amazon S3 is the fact that their pricing structure is WAY more easier to understand and predict. For $5/month you get 250 GB of storage and 1 TB of outbound transfer (inbound/upload is always free). Every additional GB stored costs $0.02 and every additional GB transferred costs $0.01. On top of that you can even serve these files utilizing use their free CDN service at no additional cost! (This alone on Amazon CloudFront can cost you a little fortune!).

Now back to the main topic: mounting DigitalOcean Space as a folder on your server that can be used to store any kind of data.

1. To start with the process you’ll need to install s3fs and some dependencies for it to work normally:

sudo yum update
sudo yum install automake autotools-dev fuse g++ git libcurl4-gnutls-dev libfuse-dev libssl-dev libxml2-dev make pkg-config
cd ~
git clone https://github.com/s3fs-fuse/s3fs-fuse.git
cd s3fs-fuse
./autogen.sh
./configure
make
sudo make install

2. Next you’ll need to create a file that will hold your Spaces API Key and Secret Key (don’t forget to replace these keys with KEY:SECRET_KEY’ in the command below):

echo 'KEY:SECRET_KEY' > ~/.passwd-s3fs
chmod 600 ~/.passwd-s3fs

3. Now we need to create a folder that will be a mounting point for Spaces

mkdir /backup

4. Now we can mount the Space

s3fs SPACE_NAME /backup -o passwd_file=~/.passwd-s3fs -o url=https://nyc3.digitaloceanspaces.com -o use_path_request_style

If your space is not in New York than Amsterdam or somewhere else: replace nyc3 with ams3 or the whatever is correct.

By this point the your DigitalOcean space should be mounted at /spaces path and you’ll be able to use it just any local folder. You could copy files in and out, rsync data there etc. Note that accessing files in that folder is done over your networks since these files are actually hosted remotely and that will burn your server’s bandwidth. You can see that this folder is mounted if you run df -h in the command line. In case you need to unmount it for any reason:let’s say you’re writing a backup bash script where it will mount the Space copy backups over and unmount it when it’s done, all you need to run is umount /backup and it’s done!

If you reboot your system this folder will not get mounted again so in order for that to be done on the boot you’ll need to edit fstab file that is located at /etc/fstab. Open the file for editing in your favorite text editor and add this line at the end of the file:

s3fs#SPACE_NAME /backup fuse allow_other,_netdev,nosuid,nodev,url=https://nyc3.digitaloceanspaces.com 0 0

After saving the changes to fstab file run:

sudo mount -a

Now the Space will be auto-mounted on every server boot. Now you can host unlimited number of files there and access them easily. You can mount same Space on couple servers so they all can access them at the same time or use it for your backups… There are so many situations where this can be incredibly useful and I don’t have a doubt you’ll use this to create something really cool.


How to backup WHM/cPanel accounts to DigitalOcean Spaces

For the past several years I have been using Amazon S3 to backup my WHM/cPanel accounts. It works quite good but the problem is Amazon’s complex pricing model (that charges you for things like diskspace, bandwidth, requests) making price for the service is quite unpredictable. I’ve read on many places that people often got surprised at the end of the month receiving the bill from Amazon.

Recently I have started testing DigitalOcean Spaces, a service similar to Amazon’s S3 (S3 compatible alternative) but with a pricing structure anyone can understand: for $5/month you get 250 GB of storage and 1 TB of outbound transfer (inbound transfer is free like on S3). If you need more than that it will cost you 2cents per every additional GB of storage and 1cent for every additional of outbound GB. Also first two months for every new space are free!

Continue Reading


How to mount Amazon S3 bucket with Goofys?

While experimenting with s3fs I have also found out another tool written in go programming language that named Goofys. Like s3fs, this tool can also mount Amazon S3 bucket as a folder that can later be accessed just like any other local folder making accessing files in S3 bucket, backing up data or syncing files quite easy. Detailed installation manual for Goofys covers only Mac while Linux installation guide is basically non-existing, so this will be my guide on how to install Goofys and mount Amazon S3 bucket on an Ubuntu Linux. This has been tested on DigitalOcean droplet running a Ubuntu version 16.04.3 x64. If you’re also testing this out on a new VPS like me – make sure to execute apt-get update  before you start.

But why would anyone want to use Goofys if s3fs is working fine? Well, according to benchmarks available at Goofys github main reason you would like to use this is performance! Goofys is much faster than s3fs. For some operations Goofys is ten times faster than s3fs making it the right choice for situations when you need access to S3 to be really fast (or as close as it being just a folder on the same drive).

Continue Reading

Tags: mount S3 bucket, https://storage.googleapis.com/apokreps/Amazon html, 5894, goofys S3 linux, https://geektnt com/how-to-mount-amazon-s3-bucket-with-goofys html, stood2k4

How to disable excessive resource usage alert emails?

If you have CSF firewall installed in cPanel you might noticed that you’re getting an email every time some process (usually a php script) uses more than 250MB of memory (default value) or more than 1800 secoonds (also default value).

These emails are being sent by LFD service that sends excessive resource usage alerts to the email address which is assigned to it, normally to root user account. This notification points out a particular process or service using excessive server resources and helps in identifying the resource eating process/service. We can either kill/stop the process/service to free the resource or allocate more resource to it, if necessary, increase the limits or simply do nothing and wait for it to finish it’s work.

Sample LFD email message when memory is exceeded:

—Time: Mon Nov 14 09:41:10 2016 +0530
—Account: xxxxxx
—Resource: Virtual Memory Size
—Exceeded: 205 > 200 (MB)
—Executable: /usr/bin/php
—Command Line: /usr/bin/php /home/xxxxxx/public_html/index.php
—PID: 26953 (Parent PID:24974)
—Killed: No

Sample LDF email message when execution time is exceeded:

—Time: Mon Nov 14 09:41:10 2016 +0530
—Account: xxxxxx
—Resource: Virtual Memory Size
—Exceeded: 125389 > 1800 (seconds)
—Executable: /usr/bin/php
—Command Line: /usr/bin/php /home/xxxxxx/public_html/index.php
—PID: 28429 (Parent PID:26561)
—Killed: No

Getting Started
1) Login to your WHM
2) Go to Home >> select Plugins
3) Click “ConfigServer Security & Firewall”
4) Locate and click at “Firewall Configuration” button

Method 1
This method will permanently disable the LFD excessive resource usage alert and you won’t receive any more emails. Performing this method could pose a a potential security/stability issue (you have been warned! 🙂 ).

5) Modify the value of directives PT_USERMEM and PT_USERTIME to 0.

PT_USERMEM = 0
PT_USERTIME = 0

Method 2
We will increase the values of both memory and time to stop or minimize the LFD alerts. If any process/service uses more resources than defined, you will still continue to receive the LFD alerts, but hopefully less.

5) Modify the value of directives PT_USERMEM and PT_USERTIME to desired.

PT_USERMEM = 500
PT_USERTIME = 150000

Finish
Save the changes you have made and restart CSF+LFD. By doing any of these two methods above should result in getting less or no email alerts from LFD.


How to instal s3fs-fuse on CentOS 6.8?

In case you have CentOS 6.8 server (or DigitalOcean droplet/vps) and you want to install s3fs-fuse the proposed way you’ll run into a problem: the s3fs-fuse requires fuse library version 2.8.4 or higher, but your yum install fuse fuse-devel will install you fuse 2.8.3. If you are running CentOS 7 you won’t have this problem at all, since by default, yum will install fuse 2.9.2.

To fix this problem you need to remove the fuse and install it manually.

1. Remove fuse installed by yum

yum remove fuse fuse* fuse-devel

2. Install some dependencies (for both fuse and s3fs-fuse)

yum install automake gcc-c++ git libcurl-devel libxml2-devel make openssl-devel

3. Download and install latest fuse library (version 2.9.7)

cd /usr/local/src
wget https://github.com/libfuse/libfuse/releases/download/fuse-2.9.7/fuse-2.9.7.tar.gz
tar -xzvf fuse-2.9.7.tar.gz
rm -f fuse-2.9.7.tar.gz
mv fuse-2.9.7 fuse
cd fuse/
./configure --prefix=/usr
make
make install
export PKG_CONFIG_PATH=/usr/lib/pkgconfig:/usr/lib64/pkgconfig/
ldconfig

4. Test that fuse is installed (should return “2.9.7″)

pkg-config --modversion fuse

5. Install the s3fs-fuse (using the default instructions from github)

cd /usr/local/src
git clone https://github.com/s3fs-fuse/s3fs-fuse.git
cd s3fs-fuse
./autogen.sh
./configure
make
sudo make install

6. If there were no errors along the way, this should return the s3fs-fuse version (currently 1.80)

s3fs --version

Hopefully, this will help you successfully install s3fs-fuse on your system. For details on how to use it please refer to documentation at the github page. Please do not hesitate to leave your comments below! Thank you.

Tags: fuse centos 6, https://geektnt com/how-to-instal-s3fs-fuse-on-centos-6-8 html