Use Azure Blob to store files

In this post, I use Azure Storage API to upload files to azure blob container. What's more, I use FileSystemWatcher to monitor file changes in a directory, and update the blob when the file is changed. This can back up files to Azure Blob. To use it, you need to have a Azure Storage account. You can test it locally using Azure Storage Emulator.

I'm using .NET Core 3.0 on Linux. First, let's create the project named azureblob and add necessary packages

dotnet new console -n azureblob 
dotnet add package Microsoft.Azure.Storage.Blob 
dotnet add package Newtonsoft.Json

The Azure blob api lives in Microsoft.Azure.Storage.Blob and I need Newtonsoft.Json to read the setting. Speaking of setting, I create this setting class:

[JsonObject(NamingStrategyType = typeof(SnakeCaseNamingStrategy))]
public class Settings
{
    public string BlobConnectionString { get; set; }
    public string BlobContainer { get; set; }
    public string MonitoredDirectory { get; set; }
}

Correspondingly, the setting file looks like this.

{
     blob_container: "azureblobtutorial",
     blob_connection_string: "<ReplaceWithYourStorageConnectionString. You can find the one for Azure Storage Emulator from the doc.>",
     monitored_directory: "<ReplaceWithYourDirectory>"
 }

Next, I create a class to call the Azure Blob API. The key is to create the CloudBlobClient and get the blob container.

var storageAccount = CloudStorageAccount.Parse(connectionString);
var blobClient = storageAccount.CreateCloudBlobClient();
this._blobContainer = blobClient.GetContainerReference(blobContainer);
this._blobContainer.CreateIfNotExists();
this._requestOptions = new BlobRequestOptions();
this._blobContainer.CreateIfNotExists(this._requestOptions);

Before uploading or deleting a blob, we should get a reference to the blob by its name. I use the file path as the name here

var blob = await this._blobContainer.GetBlobReferenceFromServerAsync(filePath, cancellationToken);

Then we can use the blob to upload or delete a file from Azure Blob.

await blob.DeleteIfExistsAsync(cancellationToken);

Those are basic operations on an Azure Blob. Next we should monitor the file changes in the directory set in monitored_directory. We use FileSystemWatcher. I need to set up the filter to listen to the right events and the event handlers as well

this._watcher = new FileSystemWatcher(monitoredDirectory);
this._watcher.NotifyFilter = NotifyFilters.LastWrite |
                                NotifyFilters.Size |
                                 NotifyFilters.FileName |
                                NotifyFilters.DirectoryName;
this._watcher.IncludeSubdirectories = true;
this._watcher.Changed += this.OnFileChanged;
this._watcher.Created += this.OnFileChanged;
this._watcher.Renamed += this.OnFileRenamed;
this._watcher.Deleted += this.OnFileChanged;
this._watcher.Error += this.OnFileWatchError;
this._watcher.EnableRaisingEvents = true;

Whenever I receive a created/deleted/changed event, in OnFileChanged, I'll eventually trigger a upload or delete on the blob. The Renamed is treated as deletion (on the old one) and creation (on the new one).

The complete code is this commit in this github repos. It still requires some more work to be able to fully work correctly in backing up files in Azure Blob.

1. When a directory is renamed, it doesn't automatically update the names of the blobs for the files/subdirectories under it.
2. It doesn't implement differential update. A small change on the file will upload the whole file. This can cause bandwidth for a large file.
3. When there are frequent changes on the same file, it doesn't batch the changes. It'll upload the whole file that many times.

Regardless, it demonstrates how to use the Azure Blob in a program, as well as the file change.