Tips on docker-compose

Docker is a technology to create containers. What if we have multiple contains and need them to work together? Docker compose comes to rescue. The configuration is a yaml file and it's easy to follow. I'm going to write down some tips when I learned to use it.

Keep the Docker Container Running

The docker container is supposed to run only one process. The container stops when the process exits. So if we have a process that runs in the forground, it'll most likely exit because there is no console or input. For this kind of container, usually we run the command docker run -ti <image name> to keep the container running. In docker-compose world, you need to put this in the configuration for the service.

tty: true
stdin_open: true

Expose the Network Ports

In running docker, we can use docker -P to expose all the ports specified in the Dockerfile. We can also use docker -p [] to expose the specified ports in the command line. However, docker -P doesn't guarantee to map those ports from the container to the same ports on the host. The mapping is random. We can explicitly set those mapping using docker -p []. docker-compose by default will create a bridge for the specified containers. Depending on how we write it, we can expose the port only to that bridge network or also in the host.

expose:
- "8080"

This will only expose the ports to the linked services.

ports:
- "8080:80"

This also expose the ports on the host. It maps the port 8080 on host to the port 80 on the container.

Connect to the Linked Service in a Container

We specify the service in the configuration. docker-compose will bring up one or multiple container running the same thing for the service. Even if you don't specify the networks, docker-compose still creates a default one for you. You can find all the network by docker network ls. The name of the default network for your docker-compose containers is derived from the directory of the docker-compose.yml file. For example, docker-compose.yml is in the directory 'example'. The default network for those containers is 'example_default'. This may change but this is the behavior right now.

How do you connect to other containers brought up by docker-compose? There are two ways:

  1. Use the service name. For example, in docker-compose.yml, you specify the service like
    services:
    foo:
    image: 'centos:latest'
    bar:
    image: 'ubuntu:latest'

    In the container that runs Centos, you can use bar to connect to the other container that runs Ubuntu.

  2. Use the ip address.
    When you find the network for your containers, you can use docker network inspect <network name> to inspect the details of the network. From there, you can find the ip address of the containers. For example:

    {
        "Name": "example_default",
        "Containers": {
            "4a87be92f610b839c77f0fa87c7bdd13797ca4eac3c2061a4d8e66a1c5e9c867": {
                "Name": "example_foo_1",
                "EndpointID": "0752c01405904b702459560fcc1bc90ef0e24d3e769e8bc46d5edcf06944dba1",
                "MacAddress": "02:42:ac:15:00:03",
                "IPv4Address": "172.21.0.3/16",
                "IPv6Address": ""
            },
            "6244c44cab85df31d684abb5e0b2ce12ecfbef902e13c638d1ab50981b876142": {
                "Name": "example_bar_1",
                "EndpointID": "5b0edb758b080010f8c3fca92e5cb583724c9fefcc23c5148c8c3375517d0093",
                "MacAddress": "02:42:ac:15:00:02",
                "IPv4Address": "172.21.0.2/16",
                "IPv6Address": ""
            }
        },

By far, those are the things I think it's useful when using docker-compose. I spent some time on searching online. I hope this can help you to try out the way to manage containers.

What Do We Need to Know about DNS

Not every programmer needs to know DNS. We tend to take it for granted. It's there. It's working. As we are moving to the cloud and Devops, it's inevitable that we need to deal with network configuration. It's important to understand some basics of DNS.

This is the place to add a custom domain to a web in Azure. It provides A record and CNAME record. What record type should I use?

Add a custom domain for Azure app service
Add a custom domain for Azure app service

What's DNS

If you still have some memory about basic network. IP address is what we use to locate a service on the Internet. But how many IP addresses do you remember? We rarely use the IP address directly. Those are hard-to-remember long numbers. Instead, we remember domain names. What is a domain name? When you type in the browser address bar like google.com, that google.com is the domain name. But we use the IP address. How do we find the IP address by the domain name? That comes the DNS (domain name systems). It resolves the domain name to the IP address.

DNS Hierarchy

You don't need to worry about setting up DNS for your network at home. When the modem connects, it will set up the DNS. Usually it is the ISP DNS. Does it know all the domains? No. It inquires other kinds of DNS servers. These DNS servers together resolve the domain name to the IP address.

DNS servers
dns server

At the top, it is the root server. The root server doesn't know what the domain name map to. But it knows what top level domain server (TLD) is. The top level domain is something like .com, .net etc. For example, google.com goes to the. com top level domain server. TLD server goes to authoritative servers to find the IP address.

The ISP server can cache the result. So it doesn't waste time to go to root servers and other servers.

DNS Records

DNS records are stored in the DNS servers. The record contains information about how to resolve the DNS inquiry. For example, if there is a record like [google.com 172.217.3.174], and when you inquire google.com, it returns the IP address.

There are three types I want to cover here. CNAME (Canonical Name record). It sets up a domain name to its canonical name. In another word, it creates an alias to the canonical name. For example, for this record [foo.com bar.com], when you inquire foo.com, it'll return the canonical name bar.com. The CNAME record doesn't contain the IP address.

The second type of record is A or AAA. It maps from the the domain name to the IP address. Record A maps to the IPv4 address and AAA maps to the IPv6 address.

The third type of record is TXT. It allows to add any human readable text to the domain. You can also add machine readable contents too. What is the content? It's up to you. Consider this is a note to the domain.

How to Get the IP Address

When the browser needs to get the IP address for the domain in the address bar, it'll send the inquiry to the DNS server configured for the network. The DNS server will ask for that following the DNS hierachy. If the domain in the address bar is an alias, it'll get a CNAME record. It then restart the process again with the canonical name. This time it'll receive the A record or AAA record which contains the IP address.

By now, hopefully you have a clearer answer to what to choose when you try to add a new custom donmain to the web in Azure. The DNS is the infrastracture of the Internet. We all should know what the underlying works when developing in the cloud era. Cloud computing has already taken care of a lot of stuffs. We still need to know the basics to be better working in the cloud.

Get Started with Scheme

Scheme is one of the three major dialects of the programming language Lisp. It has many similar syntax as Lisp but features minimalism. In this post, I'll summarize basic Scheme syntax after I read this book to get started with Scheme. That should be helpful to get started since I already have some programming experience.

Everything Meaningful Is in a List Form

The basic syntax is in the form (obj1 obj2 ...) or(expression1 expression2 ...). For example, (1 2 3) is a list that contains 1, 2, and 3. Each element can be of different types. So it can contain strings too. For example, (1 2 'hello'). If it is a procedure call, the first expression is the procedure name. For example (+ 1 2). The first element is +. It's the sum operator. It adds the numbers from the remaining of the list.

We can nest expressions. Each expression itself can be in the list form. That being said, the first one can be in the list form which returns a procedure. And then apply the remaining elements to the returned procedure.

You can have a single symbol or value too. For example "Hello" will generate "Hello". And 2 will generate 2. But if you want to have a meaningful operation, you should use the list form.

Basic Operations

These are the basic operations

  1. (quote expression)

    As I said before, the data and procedure call are in the same form. quote will force to treat the remaining as data. For example, (+ 1 2) is to calculate the sum of 1 and 2, which it generates 3. (quote (+ 1 2)) will always treat (+ 1 2) as data. + becomes the symbol instead of the operation. So it generates (+ 1 2) instead of doing the calculation. We can use ' to replace quote for abbreviation. So '(+ 1 2) is the same as (quote (+ 1 2)).

  2. (car list-expression) returns the first element in the list

    (car '(1 2 3)) => 1

  3. (cdr list-expression) returns the list that contains the elements except the first one.

    (cdr '(1 2 3)) => (2 3)

  4. (cons obj1 obj2) generates a new pair in which obj1 is the first element and the remaining are from obj2.

    It can generate a proper list or an improper list. For example,
    (cons 'a 'b) => (a . b) ; an improper list
    (cons 'a '(b c)) => (a b c) ; a proper list

Variable, Expression and Procedure

The syntax to define a lambda expression is

(lambda (var ...) body1 body2 ...)

The syntax to define a top level variable or procedure is

(define var expression)

The syntax to create a variable at the local scope is

(let ((var expression) ...) body1 body2 ...)

We can use both define and let to create a variable or a procedure.
For example, (define count 0) creates a variable count with the value 0.
(let count 0) also creates a variable count with the value 0.
To define a procedure, it's in the form

(define sum
    (lambda (x y)
        (+ x y)))

This creates a procedure sum at the top level. It sums up the two given arguments.

The difference between define and let is that define creates the variable or procedure at the top level. Any other code can reference to it. let creates it inside the let scope. That means, the variable or the procedure isn't available outside of let code.
We can nest let statements. In this case, the variable or procedure with the same name will shadow the one from outer scope. Here is the code to demonstrate it.

(define var 1)
(let ((var 2))
    (let ((var 3))
        (display var)
        (newline))
    (display var)
    (newline))
(display var)
(newline)

(display) is used to show the argument on the output. (newline) shows a new line. The output of this code is

3
2
1

The variable var in the inner scope shadows the one from outer scope.

Conditional Expression

We can do a if check in Scheme. The syntax is

(if test consequent alternative)

The test can be and, or, and not. The syntax is

(and expression ...)

(or expression ...)

(not expression)

Similarly to if, we can also do switch

(cond (test expression) ... (else expression))

There are special symbol. #t means true and #f means false.

Report errors

(assertion-violation symbol-of-scope message what-violates)

Set Assignment

(set! symbol value)

Improper List

The definition of a proper list uses recursion. A proper list is the list of which the (cdr) is a proper list. And an empty list is a proper list. If a list isn't a proper list, it's an improper list. An improper list is denoted by a '.'. For example (a . b) is an improper list and (a b) is a proper list. Below is an example in the code

(cons 'a 'b) => (a . b)
(cdr (a . b)) => b ; b is not a list
(cons 'a '(b)) => (a b)
(cdr (a b)) => (b); (b) is a list

These are the basic syntax and building block of Scheme. With that we can go on to the advanced Scheme and start writing programs.

Make It Your Habit to Fix Build Warnings

Warnings are not errors. Why do we bother to spend time fixing them? It feels so fast to get it compiled and run before looking into the warnings. It's because warnings may reveal that the code is not what you mean. It should be your habit to fix build warnings.

Errors or Warnings

Build errors are those that prevent you from getting a program or running it. Like it or not, you need to fix all the build errors. Warnings, on the other hand, don't have those restrictions. Anybody can ignore them and run the program. Luckily, the compiler usually has an option or switch to treat warnings as errors. By doing so, anybody has to address those warnings before they can continue to run the program. Having the compiler to treat warnings as errors is the only way to enforce everybody to fix the warnings too.

I'll show it how to do it in .Net Core and illustrate the importance of warnings by some examples.

Enable Warnings as Errors

There are two ways to enable warnings as errors.

The first one is to set the property TreatWarningsAsErrors in the project file. for example, in the csproj file,

<PropertyGroup>
  <TreatWarningsAsErrors>true</TreatWarningsAsErrors>
</PropertyGroup>

On the other hand, <NoWarn> is used to exclude certain warnings

<NoWarn>$(NoWarn);CS0168;CS0219</NoWarn>

<WarningLevel> is used to set the warning level. The higher the number is, the more less servere warnings it'll report. The default value is 4.

Second, use the compiler option -warnaserror. For example,

dotnet build -warnaserror example.csproj

This will turn all warnings to errors. You can set specific warning with -warnaserror like -warnaserror:642,649. -nowarn on the other hand, disables certain warnings.

-warn is used to set the warning level.

The Warning Examples

  1. warning CS4014: Because this call is not awaited, execution of the current method continues before the call is completed. Consider applying the 'await' operator to the result of the call.

    This occurs when you forget to use await on any async call. It reveals a mistake in the code even though it's not a build error. There may be a subtle bug when the program runs. It's not always happening which makes it much difficult to reproduce the bug and diagnose. For example

    using (FileStream fs = File.Create(path))
    {
         fs.WriteAsync(bytes, 0, byteSize, CancellationToken.None);
    } 

    Without await on fs.WriteAsync, it's possible that fs is being disposed while WriteAsync is still running. It all depends on the scheduling. So it's quite difficult to diagnose.

  2. warning CS1717: Assignment made to same variable; did you mean to assign something else?

    I'm sure you don't intend to assign to the same variable. It could be a typo. But you may not notice it in a quick glimpse. For example

    class Test
    {
        private int count;
        public Test(int count)
        {
            count = count;
        }
    }
    
    By treating warnings as errors, you're forced to look at the message and look at what's going on.

Exceptions

No all warnings indicate mistakes. Spend some time thinking about that. If you can justify it, you can suppress the warnings. One way of doing that is to use <NoWarn> in the project file or -nowarn in the compiler option. I don't recommend it because it applies to the whole project. I suggest to use pragma warning to disable the warning. Remember to enable the warning afterwards. For example


#pragma warning disable CS3021
//code
#pragma warning restore CS3021

Take away

Always set the flag to treat warnings as errors in your build or compiler. Always fix warnings except you can justify not to. Always explicitly surpress certain warnings before the line and restore it afterward.

Dockerize Web Service

My current web services are running on the same virtual machine. They include the WordPress (a.k.a the blog), Nextcloud, and Redmine, etc. One way to keep them secure is to keep them updated. It can be a headache when I need to find new dependencies for the newer versions. It becomes more convinient when I dockerize the web services.

The drawback of putting on the virtual machine is

  • The packages coming with the OS on the virtual machine aren't always updated. Often time I have to run a custom script to install them.
  • Installing via custom scripts may leave the system on an unsustainable state. I forget what's done and how to undo it.
  • It's possible that different web services require different versions.

I think I'm done with those struggles every time I update the services. So I decide to put them into docker.

With docker, I can

  • Easily know what I install on the docker image.
  • Re-create the image and don't worry about undoing the custom scripts.
  • Isolate different services if they have conflicts on the packages.
  • Choose the base OS image if it is needed.

I spend some time writing the Dockerfile.

  1. Figure out the dependencies. Luckily, with the right OS image, I can just use its package manager to install instead of running custom scripts. This part is straightforward since I can read the documents.
  2. Use VOLUME and save the configuration on the host instead of inside the container. This can also save me a lot of efforts when I iterate the image building.

There are two technical challenges I have to tackle and spent most time.

  1. Certbot needs to use systemd to control apache when I add SSL. After a few search, I find docker systemctl replacement. It creates a script to replace the command. The author provides some Dockerfile examples for different images. I modify that to use a custom script running a bunch of other stuffs and systemctl start httpd followed by bash. The script needs to start a command that's long running. The container will stop after the command ends.
  2. Use cron inside the container. I find a good discussion on Stack Overflow. The main steps in Dockerfile are
# Copy hello-cron file to the cron.d directory
COPY hello-cron /etc/cron.d/hello-cron
# Give execution rights on the cron job
RUN chmod 0644 /etc/cron.d/hello-cron
# Apply cron job
RUN crontab /etc/cron.d/hello-cron

There are however something I can do only when the system runs. That's done in the custom script I set to run in CMD. I use flags to make sure it only runs once for that container.

With the change, I remove the unnecessary packages and settings from my virtual machine. I can control how the environment is setup for different web services.