Get Started with Scheme

Scheme is one of the three major dialects of the programming language Lisp. It has many similar syntax as Lisp but features minimalism. In this post, I'll summarize basic Scheme syntax after I read this book to get started with Scheme. That should be helpful to get started since I already have some programming experience.

Everything Meaningful Is in a List Form

The basic syntax is in the form (obj1 obj2 ...) or(expression1 expression2 ...). For example, (1 2 3) is a list that contains 1, 2, and 3. Each element can be of different types. So it can contain strings too. For example, (1 2 'hello'). If it is a procedure call, the first expression is the procedure name. For example (+ 1 2). The first element is +. It's the sum operator. It adds the numbers from the remaining of the list.

We can nest expressions. Each expression itself can be in the list form. That being said, the first one can be in the list form which returns a procedure. And then apply the remaining elements to the returned procedure.

You can have a single symbol or value too. For example "Hello" will generate "Hello". And 2 will generate 2. But if you want to have a meaningful operation, you should use the list form.

Basic Operations

These are the basic operations

  1. (quote expression)

    As I said before, the data and procedure call are in the same form. quote will force to treat the remaining as data. For example, (+ 1 2) is to calculate the sum of 1 and 2, which it generates 3. (quote (+ 1 2)) will always treat (+ 1 2) as data. + becomes the symbol instead of the operation. So it generates (+ 1 2) instead of doing the calculation. We can use ' to replace quote for abbreviation. So '(+ 1 2) is the same as (quote (+ 1 2)).

  2. (car list-expression) returns the first element in the list

    (car '(1 2 3)) => 1

  3. (cdr list-expression) returns the list that contains the elements except the first one.

    (cdr '(1 2 3)) => (2 3)

  4. (cons obj1 obj2) generates a new pair in which obj1 is the first element and the remaining are from obj2.

    It can generate a proper list or an improper list. For example,
    (cons 'a 'b) => (a . b) ; an improper list
    (cons 'a '(b c)) => (a b c) ; a proper list

Variable, Expression and Procedure

The syntax to define a lambda expression is

(lambda (var ...) body1 body2 ...)

The syntax to define a top level variable or procedure is

(define var expression)

The syntax to create a variable at the local scope is

(let ((var expression) ...) body1 body2 ...)

We can use both define and let to create a variable or a procedure.
For example, (define count 0) creates a variable count with the value 0.
(let count 0) also creates a variable count with the value 0.
To define a procedure, it's in the form

(define sum
    (lambda (x y)
        (+ x y)))

This creates a procedure sum at the top level. It sums up the two given arguments.

The difference between define and let is that define creates the variable or procedure at the top level. Any other code can reference to it. let creates it inside the let scope. That means, the variable or the procedure isn't available outside of let code.
We can nest let statements. In this case, the variable or procedure with the same name will shadow the one from outer scope. Here is the code to demonstrate it.

(define var 1)
(let ((var 2))
    (let ((var 3))
        (display var)
        (newline))
    (display var)
    (newline))
(display var)
(newline)

(display) is used to show the argument on the output. (newline) shows a new line. The output of this code is

3
2
1

The variable var in the inner scope shadows the one from outer scope.

Conditional Expression

We can do a if check in Scheme. The syntax is

(if test consequent alternative)

The test can be and, or, and not. The syntax is

(and expression ...)

(or expression ...)

(not expression)

Similarly to if, we can also do switch

(cond (test expression) ... (else expression))

There are special symbol. #t means true and #f means false.

Report errors

(assertion-violation symbol-of-scope message what-violates)

Set Assignment

(set! symbol value)

Improper List

The definition of a proper list uses recursion. A proper list is the list of which the (cdr) is a proper list. And an empty list is a proper list. If a list isn't a proper list, it's an improper list. An improper list is denoted by a '.'. For example (a . b) is an improper list and (a b) is a proper list. Below is an example in the code

(cons 'a 'b) => (a . b)
(cdr (a . b)) => b ; b is not a list
(cons 'a '(b)) => (a b)
(cdr (a b)) => (b); (b) is a list

These are the basic syntax and building block of Scheme. With that we can go on to the advanced Scheme and start writing programs.

Make It Your Habit to Fix Build Warnings

Warnings are not errors. Why do we bother to spend time fixing them? It feels so fast to get it compiled and run before looking into the warnings. It's because warnings may reveal that the code is not what you mean. It should be your habit to fix build warnings.

Errors or Warnings

Build errors are those that prevent you from getting a program or running it. Like it or not, you need to fix all the build errors. Warnings, on the other hand, don't have those restrictions. Anybody can ignore them and run the program. Luckily, the compiler usually has an option or switch to treat warnings as errors. By doing so, anybody has to address those warnings before they can continue to run the program. Having the compiler to treat warnings as errors is the only way to enforce everybody to fix the warnings too.

I'll show it how to do it in .Net Core and illustrate the importance of warnings by some examples.

Enable Warnings as Errors

There are two ways to enable warnings as errors.

The first one is to set the property TreatWarningsAsErrors in the project file. for example, in the csproj file,

<PropertyGroup>
  <TreatWarningsAsErrors>true</TreatWarningsAsErrors>
</PropertyGroup>

On the other hand, <NoWarn> is used to exclude certain warnings

<NoWarn>$(NoWarn);CS0168;CS0219</NoWarn>

<WarningLevel> is used to set the warning level. The higher the number is, the more less servere warnings it'll report. The default value is 4.

Second, use the compiler option -warnaserror. For example,

dotnet build -warnaserror example.csproj

This will turn all warnings to errors. You can set specific warning with -warnaserror like -warnaserror:642,649. -nowarn on the other hand, disables certain warnings.

-warn is used to set the warning level.

The Warning Examples

  1. warning CS4014: Because this call is not awaited, execution of the current method continues before the call is completed. Consider applying the 'await' operator to the result of the call.

    This occurs when you forget to use await on any async call. It reveals a mistake in the code even though it's not a build error. There may be a subtle bug when the program runs. It's not always happening which makes it much difficult to reproduce the bug and diagnose. For example

    using (FileStream fs = File.Create(path))
    {
         fs.WriteAsync(bytes, 0, byteSize, CancellationToken.None);
    } 

    Without await on fs.WriteAsync, it's possible that fs is being disposed while WriteAsync is still running. It all depends on the scheduling. So it's quite difficult to diagnose.

  2. warning CS1717: Assignment made to same variable; did you mean to assign something else?

    I'm sure you don't intend to assign to the same variable. It could be a typo. But you may not notice it in a quick glimpse. For example

    class Test
    {
        private int count;
        public Test(int count)
        {
            count = count;
        }
    }
    
    By treating warnings as errors, you're forced to look at the message and look at what's going on.

Exceptions

No all warnings indicate mistakes. Spend some time thinking about that. If you can justify it, you can suppress the warnings. One way of doing that is to use <NoWarn> in the project file or -nowarn in the compiler option. I don't recommend it because it applies to the whole project. I suggest to use pragma warning to disable the warning. Remember to enable the warning afterwards. For example


#pragma warning disable CS3021
//code
#pragma warning restore CS3021

Take away

Always set the flag to treat warnings as errors in your build or compiler. Always fix warnings except you can justify not to. Always explicitly surpress certain warnings before the line and restore it afterward.

Dockerize Web Service

My current web services are running on the same virtual machine. They include the WordPress (a.k.a the blog), Nextcloud, and Redmine, etc. One way to keep them secure is to keep them updated. It can be a headache when I need to find new dependencies for the newer versions. It becomes more convinient when I dockerize the web services.

The drawback of putting on the virtual machine is

  • The packages coming with the OS on the virtual machine aren't always updated. Often time I have to run a custom script to install them.
  • Installing via custom scripts may leave the system on an unsustainable state. I forget what's done and how to undo it.
  • It's possible that different web services require different versions.

I think I'm done with those struggles every time I update the services. So I decide to put them into docker.

With docker, I can

  • Easily know what I install on the docker image.
  • Re-create the image and don't worry about undoing the custom scripts.
  • Isolate different services if they have conflicts on the packages.
  • Choose the base OS image if it is needed.

I spend some time writing the Dockerfile.

  1. Figure out the dependencies. Luckily, with the right OS image, I can just use its package manager to install instead of running custom scripts. This part is straightforward since I can read the documents.
  2. Use VOLUME and save the configuration on the host instead of inside the container. This can also save me a lot of efforts when I iterate the image building.

There are two technical challenges I have to tackle and spent most time.

  1. Certbot needs to use systemd to control apache when I add SSL. After a few search, I find docker systemctl replacement. It creates a script to replace the command. The author provides some Dockerfile examples for different images. I modify that to use a custom script running a bunch of other stuffs and systemctl start httpd followed by bash. The script needs to start a command that's long running. The container will stop after the command ends.
  2. Use cron inside the container. I find a good discussion on Stack Overflow. The main steps in Dockerfile are
# Copy hello-cron file to the cron.d directory
COPY hello-cron /etc/cron.d/hello-cron
# Give execution rights on the cron job
RUN chmod 0644 /etc/cron.d/hello-cron
# Apply cron job
RUN crontab /etc/cron.d/hello-cron

There are however something I can do only when the system runs. That's done in the custom script I set to run in CMD. I use flags to make sure it only runs once for that container.

With the change, I remove the unnecessary packages and settings from my virtual machine. I can control how the environment is setup for different web services.

Use Azure Blob to store files

In this post, I use Azure Storage API to upload files to azure blob container. What's more, I use FileSystemWatcher to monitor file changes in a directory, and update the blob when the file is changed. This can back up files to Azure Blob. To use it, you need to have a Azure Storage account. You can test it locally using Azure Storage Emulator.

I'm using .NET Core 3.0 on Linux. First, let's create the project named azureblob and add necessary packages

dotnet new console -n azureblob 
dotnet add package Microsoft.Azure.Storage.Blob 
dotnet add package Newtonsoft.Json

The Azure blob api lives in Microsoft.Azure.Storage.Blob and I need Newtonsoft.Json to read the setting. Speaking of setting, I create this setting class:

[JsonObject(NamingStrategyType = typeof(SnakeCaseNamingStrategy))]
public class Settings
{
    public string BlobConnectionString { get; set; }
    public string BlobContainer { get; set; }
    public string MonitoredDirectory { get; set; }
}

Correspondingly, the setting file looks like this.

{
     blob_container: "azureblobtutorial",
     blob_connection_string: "<ReplaceWithYourStorageConnectionString. You can find the one for Azure Storage Emulator from the doc.>",
     monitored_directory: "<ReplaceWithYourDirectory>"
 }

Next, I create a class to call the Azure Blob API. The key is to create the CloudBlobClient and get the blob container.

var storageAccount = CloudStorageAccount.Parse(connectionString);
var blobClient = storageAccount.CreateCloudBlobClient();
this._blobContainer = blobClient.GetContainerReference(blobContainer);
this._blobContainer.CreateIfNotExists();
this._requestOptions = new BlobRequestOptions();
this._blobContainer.CreateIfNotExists(this._requestOptions);

Before uploading or deleting a blob, we should get a reference to the blob by its name. I use the file path as the name here

var blob = await this._blobContainer.GetBlobReferenceFromServerAsync(filePath, cancellationToken);

Then we can use the blob to upload or delete a file from Azure Blob.

await blob.DeleteIfExistsAsync(cancellationToken);

Those are basic operations on an Azure Blob. Next we should monitor the file changes in the directory set in monitored_directory. We use FileSystemWatcher. I need to set up the filter to listen to the right events and the event handlers as well

this._watcher = new FileSystemWatcher(monitoredDirectory);
this._watcher.NotifyFilter = NotifyFilters.LastWrite |
                                NotifyFilters.Size |
                                 NotifyFilters.FileName |
                                NotifyFilters.DirectoryName;
this._watcher.IncludeSubdirectories = true;
this._watcher.Changed += this.OnFileChanged;
this._watcher.Created += this.OnFileChanged;
this._watcher.Renamed += this.OnFileRenamed;
this._watcher.Deleted += this.OnFileChanged;
this._watcher.Error += this.OnFileWatchError;
this._watcher.EnableRaisingEvents = true;

Whenever I receive a created/deleted/changed event, in OnFileChanged, I'll eventually trigger a upload or delete on the blob. The Renamed is treated as deletion (on the old one) and creation (on the new one).

The complete code is this commit in this github repos. It still requires some more work to be able to fully work correctly in backing up files in Azure Blob.

  1. When a directory is renamed, it doesn't automatically update the names of the blobs for the files/subdirectories under it.
  2. It doesn't implement differential update. A small change on the file will upload the whole file. This can cause bandwidth for a large file.
  3. When there are frequent changes on the same file, it doesn't batch the changes. It'll upload the whole file that many times.

Regardless, it demonstrates how to use the Azure Blob in a program, as well as the file change.

Redirect Assembly Binding

In a large .Net project, it can be inevitable to have complex dependencies. To make it worse, multiple dependencies may have the dependency on the same assembly but different versions. There is already a way to redirect to bind a different version of assembly in your app. This document outlines how to do it to an application. Sometimes, that's not enough.

The document outlines these approaches

  1. The vendor of assemblies includes a publisher policy file with the new assembly.
  2. Specify the binding in the configuration file at the application level
  3. Specify the binding in the configuration file at the machine level.

The first approach requires the vendor to publish the publisher policy file. The file has to be in global assembly cache which will affect every application on the machine.

What if the vendor doesn't provide this file. Then we can specify the binding in the configuration file by using <bindingRedirect>. The configuration file can be applied to the specified application if it's at the application level or every application if it's at the machine leve.

What if there is not a publisher policy file, or there is no configuration file for the binding at the application level or the machine level? Is it possible to have it happening when you're writing your own application. Probably not. This issue probably happens when you write a plugin or some assembly that're run in a different aplication that you don't own. For example, you're writing a test that's run by vstest. You use a libary A which has a dependency on assembly B version 1.0, and you also use a library C which has a dependency on assembly B version 2.0. At runtime, one version of the assembly B will not be loaded. You don't own the assembly B, and you don't own the application that runs your assembly. Because of that, you cannot count on the publisher policy file or the application-level configuration file. You don't want to create a machine-level configuration file either. There is no assembly level configuration file. The assembly level configuration file is ignored at runtime. I think the best bet of solving it is to load the dependency in the program by yourself. When the runtime doesn't find the right assembly, it raises the event AppDomain.AssemblyResolve.

How do we use AppDomain.AssemblyResolve? The basic idea is:

  1. Check whether the assembly is loaded.
  2. If it's loaded, and if the loaded version satisfies your requirements, then return the loaded one.
  3. If the assembly isn't loaded, and you find one that satisfied your requirements, you can call Assembly.LoadFile to load the assembly and return it.

In a pseudo code, it is

static Assembly OnAssemblyResolve(object sender, ResolveEventArgs args)
{
    if (args.Name.Contains("AssemblyB"))
    {
        foreach (Assembly assembly in AppDomain.CurrentDomain.GetAssemblies())
        {
            if (assembly.FullName.Contains("AssemblyB"))
            {
                return assembly;
            }
        }

        return Assembly.LoadFile("PathToAssemblyB");
    }

    return null;
}

There are, however, some caveats. First, it's at the app domain level, meaning it may impact every assembly in the same app domain. AppDomain.AssemblyResolve passes an event parameter ResolveEventArgs. It has a property ResolveEventArgs.RequestingAssembly to indicate which assembly is requesting to load the one that cannot be resolved. You can use it to make sure that you're loading the assembly in the right context. Second, if you happen to use one of Assembly.Load overloads and it causes AssemblyResolve event, you'll get a stack overflow. You can check out this guidance.

Using well, I think AppDomain.AssemblyResolve can supplement configuration file in handling assembly binding issues in the application.