RCLONE - “The Swiss Army Knife of Cloud Storage” | DTRHnet#container.show {
background: linear-gradient(200deg,#a0cfe4,#e8c37e);
}
RCLONE - “The Swiss Army Knife of Cloud Storage”2024-02-14
NOTE: For Pixeldrain support,compile the source code here. For updates on its backend being officially integrated,check this github thread here.
BACKSTORY
I’d like to take you on a little tripdown the rabbit holewith me. I decided totake my own advice, painfully going through each and every account - changing each password to a randomly generated one, unique to each account, as well as activating 2-FA where possible. This was achieved withNordPassand a lot of patience - you can readmy article which discusses account security, data breaches, and best practices here.
This of course meant applying changes to my hosting company account, cpanel, email addresses, cloudflare, VPS accounts, cloud storages etc. I noticed I didn’t have any cron jobs running regular backups of my website after switching from Wordpress to NodeJS/Hexo. Initially I was going to write a simple script, executed via the cron daemon, that would make a g-zip of the data I wanted and move it to a local folder configured with proper permissions.
Nowin the event something went terribly wrong with my hosting company/account, I wanted to also have a backup with one or all of my cloud/file hosts. I decidedpixeldrainwould do, asI have played around a bit with their api. In reviewing the http based API, I noticed just recently pixeldrain enabled theirexperimental filesystem feature(https://pixeldrain.com/filesystem). After doing some digging, I foundthis post, where Fornax, Pixeldrains author, is attempting to merge his backend with the rclone master repository. That is the series of events which led me here, writing about this tool which I see as important.
WHAT IS RCLONE ANYWAY??
At its core, rclone is across platformcommand-line tool used to manage cloud storage. It features support forover 70 cloud storage solutions, and leverages familiar command-line equivalents to those found on unix systems. These commands simplify and speed the time it takes to learn how to properly use it. It’s true power is harnessed in its ability to be implemented into scripts for automation. There is no comparison to rclones power and efficiency when working with cloud based storage solutions.
I. Rclone copying all directories starting with “U” via FTP to local USB storage
If you want toRTFM, check out therclone docs here. I’ll be keeping this text short and sweet - offering up a fewvery important tipsand some use case examples. Quite honestly, if you are familiar with commands such as**copy (cp), move (mv), mount(mount), sync(sync)**etc, you’re already done most of the work to use this tool on a basic level. Of course, when it comes to more complex tasks,such as mounting for the purpose of streaming video over the cloudyou’ll have to do some homework! Thankfully, rclone is well documented, and each cloud solution officially supported with a custom backend has its ownspecific documentation and configuration examples.
EXAMPLE RCLONE USES
CREATING REMOTE CONFIGURATIONS
Each and every cloud storage solution is going to have its own specific set of ‘rules’ to follow, differing in authentication methods and protocols they may use. Rclone attempts to simplify this process through its ‘config’ subcommand. If your cloud solution isn’t mentioned, it doesn’t necessarily mean it isn’t supported. There are a bunch ofgeneric protocol optionsyou can use, such asFTP, SSH, Local Disk, DAV, etc. In my little experience, these backend configurations address the different expectations that each provider may require and is a constant work in progress, as noted with the Pixeldrain merge mentioned earlier.
Remotesrefer to the different storages you will be working with. You start by creating a remote in a configuration profile, which sits in your~/.config/rclone/rclone.confon linux. Within this file stores the information required for rclone to access each remote you create. This includes things such asthe remote name, the type of service/protocol, username, password, port, etc.
A NOTE ON PASSWORDS
You can choose to put your password in and have it stored in the configuration file encrypted, but from a security standpoint that doesn’t sit well with me. While the password is encrypted, anyone with access to the configuration can access the service. Encrypted passwords can be brute forced as well. So while creating a config, you can choose to be prompted to enter your password instead.
The commandrclone configwill start the process of making a remote. You will be presented with various questions about the remote you would like to access. Navigating the all CLI system is easy and self explanitory - hit ‘y’ for yes, ‘n’ for no, etc. In this case, I am creating anew remote, and begin the process after entering ‘n’ as a response in the terminal. Take note of all the options when choosingstorage type.”Using the rclone subcommand config to create a remote FTP configuration”
$ rclone config
Current remotes:
Name Type
==== ====
PD http
VRPirates ftp
pixeldrain http
e) Edit existing remote
n) New remote
d) Delete remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit config
e/n/d/r/c/s/q> n
Enter namefornew remote.
name> WHATBOX
Option Storage.
Type of storage to configure.
Choose a number from below, ortypeinyour own value.
1 / 1Fichier
\ (fichier)
2 / Akamai NetStorage
\ (netstorage)
3 / Aliasforan existing remote
\ (alias)
4 / Amazon Drive
\ (amazon cloud drive)
5 / Amazon S3 Compliant Storage Providers including AWS, Alibaba, ArvanCloud, Ceph, ChinaMobile, Cloudflare, DigitalOcean, Dreamhost, GCS, HuaweiOBS, IBMCOS, IDrive, IONOS, LyveCloud, Leviia, Liara, Linode, Minio, Netease, Petabox, RackCorp, Rclone, Scaleway, SeaweedFS, StackPath, Storj, Synology, TencentCOS, Wasabi, Qiniu and others
\ (s3) …
56 / seafile
\ (seafile)
Storage> 16
Option host.
FTP host to connect to.
E.g.”ftp.example.com”.
Enter a value.
host> dawn.whatbox.ca
Option user.
FTP username.
Enter a string value. Press Enterforthe default (android).
user> rhadmin
Option port.
FTP port number.
Enter a signedinteger. Press Enterforthe default (21).
port>
Option pass.
FTP password.
Choose an alternative below. Press Enterforthe default (n).
y) Yes,typeinmy own password
g) Generate random password
n) No, leave this optional password blank (default)
y/g/n> y <- PROBABLY DONT DO THIS
Enter the password:
password:
Confirm the password:
password:
Option tls.
Use Implicit FTPS (FTP over TLS).
When using implicit FTP over TLS the client connects using TLS
right from the startwhichbreaks compatibility with
non-TLS-aware servers. This is usually served over port 990 rather
than port 21. Cannot be usedincombination with explicit FTPS.
Enter a boolean value (trueorfalse). Press Enterforthe default (false).
tls>
Option explicit_tls.
Use Explicit FTPS (FTP over TLS).
When using explicit FTP over TLS the client explicitly requests
security from the serverinorder to upgrade a plain text connection
to an encrypted one. Cannot be usedincombination with implicit FTPS.
Enter a boolean value (trueorfalse). Press Enterforthe default (false).
explicit_tls>
Edit advanced config?
y) Yes
n) No (default)
y/n> n
Configuration complete.
Options:
-type: ftp
- host: [REDACTED].whatbox.ca
- user: [REDACTED]
- pass: *** ENCRYPTED ***
Keep this”WHATBOX”remote?
y) Yes this is OK (default)
e) Edit this remote
d) Delete this remote
y/e/d> y
Current remotes:
Name Type
==== ====
PD http
VRPirates ftp
WHATBOX ftp
pixeldrain http
e) Edit existing remote
n) New remote
d) Delete remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit config
e/n/d/r/c/s/q> q
~ $mkdirwhatbox
~ $ rclonelsWHATBOX:/
881 .bash_history
127 .bash_logout
193 .bash_profile
741 .bashrc
129 .config/rclone/rclone.conf
IMPORTANT PRACTICES
While I am sure there are many more than what I’ll list here, this will serve as a good start and possibly save you some headache.
First and foremost, and as with any tool you are using towrite, overwrite, and delete data, you’ll want to make sure that the command you let run while you go to sleep for the night isn’t wreaking havoc on your system and data. It should act in a predictable manner, and if there is a method of testing this - use it.
Rclone offers the flag**–dry-run**. Use it. The command will run its course and provide any errors or output details without actually touching the data. It can give you insight into issues you might need to address.
One such instance I encountered was when I switched from rcloning a massive amount of data to my USB external disk drive. Initially I started the job on a linux box (an iMac with Kali), but switched to my Windows laptop in favor of USB3 speeds. I was given awarningabout my system using**sparse files, which would slow the operation down and not something I was dealing with on Linux. The flag to disable that is–local-no-sparse**.
A few of the flags you will likely find useful include :FlagExampleDefinition–dry-runRun through the entire process without actually writing to disk to ensure the command acts as intended–bwlimit–bwlimit=80MSet a maximum bandwidth limit at any one time. Useful if rclone is killing other applications or services by overconsumption–transfers–transfers=2Set the amount of concurrent transfers. In this example, two files are being downloaded at any one time–local-no-sparseDisable sparse files–size-onlyWhen copying from one remote to another, only test the file size differences–checkers–checkers=30Set the amount of concurrent file checkers (the processes comparing file attributes and determining which files require work)–log-level–log-level INFOSet the level of logging data to output. Other options include ** DEBUG, NOTICE, ERROR**–log-file–log-file=FILELocation of the logfile to write–check-firstCheck the files before beginning write, determining which files will be written–config–config=CONFIGLocation of an alternate configuration file to use-PDetailed verbose output
Mount to stream contentrclone mount -P “PIXELDRAIN:/Videos” “/media/dtrh/rmount/Videos”–buffer-size=32M –dir-cache-time=84h –vfs-cache-mode=minimal –vfs-cache-max-age=6h
Sync remote backup folder with local one; NOTE: Add –bidirectional to keep directories in sync both waysrclone sync -P DTRHlocal:/public_html/ DTRHweb:/web_backup/ –transfers=6
Backup/update website; UPLOADING via COPY
rclone copy ~/Documents/website/dtrh/public/ WHATBOX:/dtrhsrv1/Downloads/
rclone copy ~/Documents/website/dtrh/public/ WEBHOST:/public_html/
MY IMPLEMENTATION EXAMPLE OF RCLONE
I’ve written the following bash script for myself and to useas an example of implementing rclone in an automated way. It builds the website, updates it on my VPS before right to production. It backs it up to numerous locations in a compressed folder and rotates them out, saving 3 to 5 total backups. These compressed files have the date and time stamped into them for easy reference.
Other ways to implement automation could be via the cron daemoncrontab -e.
Incomplete, as I’ve been working on more important things, will return back and complete before uploading to github
I also intend on making a second version that can be run withhexo generateandn require absolutely no human input
#!/bin/bash
PUBTARFILE=DTRHweb$(date “+[%b.%d.%C%y]-[%H:%M:%S]”).tar.gz
PUBLICBACK=$ROOT/.backups/public
ROOTTARFILE=DTRHsrc$(date “+[%b.%d.%C%y]-[%H:%M:%S]”).tar.gz
ROOT=”$(pwd)”
WEBROOT=”$ROOT/public”
PBACKDIR=$ROOT/.backups/public
RBACKDIR=$ROOT/.backups/root
PUBTARMAX=5
ROOTTARMAX=3
hexo_exit_status=
function print_config() {
}
function webStage() {
echo “Sync public foldernwith staging server [rabbit-hole.ca]?”
select yn in “Yes” “No”; do
case $yn in
Yes ) rclone $WEBROOT VPS:/var/www/html; break;;
No ) exit;;
esac
done
}
function webLive(){
rclone $WEBROOT A2:”~/public_html”
}
function cloudCleanup() {
rclone lsf WHATBOX:/backups/pub | head -n $PUBTARMAX > to-del
rclone delete –files-from to-del WHATBOX:/backups/pub
rclone lsf WHATBOX:/backups/root | head -n $PUBTARMAX > to-del
rclone delete –files-from to-del WHATBOX:/backups/root
}
function cloudBackup() {
rclone copy “$PBACKDIR/$PUBTARFILE” PIXELDRAIN:/backups/pub
rclone copy “$PBACKDIR/$PUBTARFILE” WHATBOX:/backups/pub
rclone copy “$PBACKDIR/$PUBTARFILE” PCLOUD:/backups/pub
rclone copy “$RBACKDIR/$ROOTTARFILE” PIXELDRAIN:/backups/root
rclone copy “$RBACKDIR/$ROOTTARFILE” WHATBOX:/backups/root
rclone copy “$RBACKDIR/$ROOTTARFILE” PCLOUD:/backups/root
rclone sync “$WEBROOT” WHATBOX:/backups/live/public
rclone sync “$ROOT” WHATBOX:/backups/live/root
}
function genTars() {
npx hexo generate >/dev/null
hexo_exit_status=$?
[ $(ls -l $PBACKDIR | grep DTRHweb | wc -l) -ge 5 ] && mv $(ls -t $PBACKDIR | head -n1) /tmp
tar -zcf “$PBACKDIR/$PUBTARFILE” “$WEBROOT” >/dev/null
$(ls -l $RBACKDIR | grep DTRHroot | wc -l) -ge 3 ] && mv $(ls -t $RBACKDIR | head -n1 ) /tmp
tar -zcf “$RBACKDIR/$ROOTTARFILE” “$ROOT” >/dev/null
}
main() {
echo “DTRH.net SUB tool”
echo -e “(Stager, Updater, Backup)/t/t$date”
echo -e “\n”
echo “Display current rclone config?””
select yn in “Yes” “No”; do
case $yn in
Yes ) cat “~/.config/rclone/rclone.conf”; break;;
No ) break;;
esac
done
genTars
[ hexo_exit_status -eq 0 ] && webStage || echo “There was an error buildimg with hexo generate! Skipping webStage()”
cloudBackup
cloudCleanup
echo “Sync public folder to live server [https://dtrh.net]?”
select yn in “Yes” “No”; do
case $yn in
Yes ) webLive; break;;
No ) exit;;
esac
done
}
扫一扫，分享到微信
Reverse Tunneling SSHHardening Online Accounts - An analysis of breached data© 2024 KBSHexoThemeYiliaby Litten
tag:
WORK IN PROGRESS