How to Copy Local Files to Oracle Object Storage

Hi,

This is my first Oracle Cloud blog post. Due to some recent event, I am working on Oracle Cloud quite a bit and that “scary” cloud is not so scary anymore! I do plan to create a blog post series about how to migrate basic application/database structure to the cloud because when I started I really had some hard times to understand terminology and technology. But today I want to start with something simple and requires basic cloud knowledge.

I use a Windows Laptop which I am satisfied and I have so many documentation of mine. Whatever I learn, I document it. Every new information, a new problem I solved, sample code templates. Beside that, I have pictures of last 15 years of my life, client data, my insurance policies etc. I also use Google Drive to sync those files to a remote location in case I lost my laptop/disk. I cannot tolerate to loose this data. Few weeks ago, I have read about some news that some users has lost their files on google drive. Reason is unknown but files on google drive are gone for reason. That scared me a lot because I use a mirror technology for google drive. if a file is deleted on the google drive, it will also be deleted on my laptop. Second of all, google drive does not store data encrypted! This means, google can read my data which you must accept on their agreement.

These problems forced me to find a new “google drive” alternative. I looked for few options then all of a sudden a light ball appeared 🙂 Why don’t I use Oracle Cloud? since I have a cloud account, I can backup my files to the cloud as well and Oracle Cloud is a fantastic solution! All data in object storage is encrypted by default, no one can read your data but you. It is really fast (I uploaded some files at rate 120MB per second) and highly available. Oracle provides %99.99 availability for object storage. That means at most 2 minutes unavailability in a month! it is perfect solution.

So, I started to look for a script/software to copy my files to object storage regularly. it should have the capability to compare so that same files shouldn’t be copy over and over again. So, I found “rclone” which is designed to copy from one source to another and it provides a huge source and destination options. I am also quite new to this, hence I will not go deeper. During my google search, I found many posts about copying data to Oracle object storage but many of them are old. in old days, rclone wasn’t natively support Oracle object storage, so everyone were using it via Amazon S3 compatibility. I finally achieved to copy my files via rclone and explain how to do it.

First of all, you must already have a cloud user with enough privileges to access an object storage and you must have a “bucket”.

Step 1 Install rclone

start a powershell on windows and type

or you can simply go to https://rclone.org/ then download and install it.

Step 2 Create an API Key

Log into your Oracle Cloud account, either click “my profile” under user icon at top right or on burger menu, Identity => My Profile.

Click Add API Key, download private and public key and put them into somewhere safe. I save private key as D:\cloud\mustafakalayci@somedomain.com_2024-01-05T12_21_06.347Z.pem

After creating API key UI will show you configuration file content, save it into another text file. if you didn’t copy it at the first time, on API keys list click, three dots at the right and click “view configuration file” button. sample configuration file content is like this:

for “key_file” parameter, you must write your previously saved private key file name and full path.  So I saved this content to D:\cloud\api.conf file.

That is all what we need to do with Oracle Cloud.

Step 3 Create a Config File for rclone

rclone has a configuration file so that you can define your “resources”. by saying resources I meant source and destinations. for example you might want to copy some file from Amazon to Oracle Object Storage. you need to define both of them in this configuration file. Since I want to copy my local files to Oracle object storage, I only need to define Oracle Object Storage.

Now for this purpose, you can run “rclone config” command and follow the steps. step by step it asks you the data. I will not demonstrate all of it but instead, I will share my sample config file, you can simply copy paste this and change it according to your data. if you want to see where the default config file is, you can run “rclone config file” command. it will show the file.

sample rclone config:

in this config file, we are pointing our cloud api key configuration file (D:\cloud\api.conf) that we saved on previous step. I names this config file section (profile name) as “mybackupstorage” I will use it in rclone command while copying.

That is pretty much it! Now, you can copy your files/folders to an object storage. here is the code template:

So, if I want to copy MyDocumentation folder on D drive to a object storage bucket called “bucket-my-backup” with same folder name, I can use this:

Please remember that, remote file/folder names are mandatory. (mybackupstorage:/bucket-my-backup/MyDocumentation) if you don’t specify “MyDocumentation” after bucket name you will get “Failed to head object: value cannot be empty for field ObjectName in path” error.

on first run, it will take a while to upload the data but on second run, only changes will be uploaded and it will be much faster.

Another cons about Oracle object storage is, if I remember correctly, 20GB of object storage is free! and above that it is not much expensive. you can check https://www.oracle.com/cloud/costestimator.html for object storage prices. currently, after 20GB free, infrequent access of 100GB per month is 0.9$.

So, this is a great solution to backup your data. it is encrypted, it is very highly available, it is cheep (even free for 20GB) and it is Oracle. Also, you can “mount” object storage as a drive on your local machine and use it directly! this is also amazing. I share some links that use while setting up my environment and also I share some links for mounting object storage to your local machine. I didn’t try the mounting method but I will. you can find other useful information on those links.

I hope this post helps, thanks for reading.

https://rclone.org/oracleobjectstorage/
https://rclone.org/oracleobjectstorage/tutorial_mount/

https://blogs.oracle.com/linux/post/using-rclone-to-copy-data-in-and-out-of-oracle-cloud-object-storage

/*mounting with rclone*/

https://blogs.oracle.com/ateam/post/use-oci-object-storage-as-a-mounted-filesystem
https://medium.com/@mukundmurali/mount-oci-object-storage-using-rclone-233a71139c9a