.Net Framework Native Image and Troubleshooting

As an end user, you probably don't know what a native image is, because it's transparent to the user and you don't need to worry about it. As a .Net developer, it's important to understand the native image and how to diagnose it.

A native image is a compiled process-specific machine code that's installed into native image assembly cache. Usually it's at C:\Windows\assembly\NativeImagesXXX. This path is always in CLR probing path. When CLR finds a matching and valid native image for a managed assembly, the native image will be loaded. Otherwise, the managed assembly is loaded and jit'ed. There are two major advantages of using native images. The first one is to reduce application start up time. Since the native image is already compiled, there is no need for JIT compiler to compile the managed image at start up time. Second, it reduces memory footprint. The native image can be shared by multiple processes.

How to generate native images? ngen.exe. It provides  one action to generate native images:
install <assembly name>|<assembly path>
It compiles the assembly and its dependencies. If you pass option /queue, it won't generate the native image immediately. It's put into a queue. It's beneficial to put them to queue first when it takes a long time to compile if you have a large set of assemblies. When does it compile then? There will be a task in Windows to execute compiling at idle time. Native images don't always get generated if the dependencies are missing. You can use ngen.exe action
display /verbose
to view the details of the native image. There are hard dependencies and soft dependencies. If there are any missing hard dependencies, the native image won't be loaded even if it's generated.
How to specify the locations of the dependencies? You can set /ExeConfig to an executable and the executable's app.config will be used to find the dependencies. You have to make sure the dependencies set in this config are the same ones used at runtime. Otherwise, the native images won't be loaded.

You can generate multiple native images for the same assembly for different scenarios. This can be specified in ngen.exe action install by these options: /Debug, /Profile and /NoDependencies. Note that you can specified multiple scenarios in the same action. Which native image to load then? It depends on the scenario you set when you load the assembly. The corresponding native image for that scenario is loaded. Note that you don't have to specify the scenario in install action if it's not for any of those three scenarios.

Native images can be invalid if one of these occurs. The .Net framework is updated. The dependencies are updated or the assembly itself is updated (even the timestamp is changed). In this case, you can use ngen update to re-compile the native images.

While you believe a native image is generated and it should be loaded, but it's not loaded, what can you do? Fortunately, there's a tool in Visual Studio: fuslogvw.exe. I suggest you to enable "log all binds to disk" and use custom log path. There is a bug in setting custom log path. It may complain the path is not an absolute path. Just disable custom log path and try again. Usually this will solve the problem. fuslogvw.exe will then track all the assembly load as well as native image load. It generates the log files per assembly in the specified path. You can then open the log file for the failed native image.

The log shows the assembly to load, where it tries to find the dependencies, whether the assembly or the native image is loaded successfully and why it fails. And the reason why it fails is shown as a warning in the log. It contains these information for each attempt of loading.

You need some efforts to understand what the log tries to tell you, especially why the native image fails to load. The message is simple but it's not obviously related to the cause. Here are some common warning message and how to continue the investigation.

  1. Dependency assembly was not found at ngen time, but is found at binding time. Disallow using this native image.
    This is confusing, especially when you're quite sure the dependency assembly was deployed the same time with your assembly before ngen.  Actually, this means the dependency assembly is different from the one that's used at ngen time. It's different in version, public key token, culture or create timestamp. If the dependency assembly is deployed in multiple locations, the one used in ngen should also be the same used in runtime. You can check that from the assembly path from display /verbose and fuslogvw.exe's log.
  2. Rejecting native image because it failed the security check. The assembly's permissions must have changed since the time it was ngenned, or it is running with a different security context.
    The security settings when the native image is loaded  are different from when it's generated.
  3. Native image compile options do not match request.
    Remember the scenarios you use to generate native images? Yes. That is what it means. You compile it for one scenario but load it for a different one. For example, you compile it with /Profiler, but you don't specify any when you load it.
  4. Timestamps of the IL assembly does not match record in .aux file.
    ngen stores the create timestamp of the managed assembly in the .aux file. It compares the timestamp to make sure that the managed assembly which the native image is compiled from is actually the one that should be loaded. Otherwise, the native image won't be loaded. This can happen if you update the managed assembly. Or you compile the assembly at one location but there is a copy at a different location. The other one is loaded but it has a different timestamp.

There are also a few other reasons why native images fail to load. In short, that's because the native images are invalid.  These could happen when the system is updated, the assembly is updated, the dependencies are updated, or simply the timestamps are different.

That's it about native images. It's good to use native images to improve performance. The drawback is that it can become invalid due to a few factors. It's difficult to control all of the factors. You should measure the performance versus the effort.

Adding z-wave to OpenHAB

After setting up OpenHAB, we are ready to run it. It's exciting to see that it works. I imagine to control outlets, light switches and light bulbs from my phone.

OpenHAB isn't tied to any specific devices or protocols. It talks to various devices by binding which is extensible. There's a z-wave binding to communicate z-wave devices. So first thing to do is to add z-wave binding. After you add z-wave binding to your OpenHAB, you can use it to find your z-wave devices. There are two steps. First is to add a z-wave controller. Then include various z-wave devices.

First, install a z-wave controller. A z-wave controller acts like a bridge between your other z-wave devices and OpenHAB. The controller understands z-wave network and protocol. OpenHAB sends commands to and receives states from z-wave devices via the controller. Thing of it like a Bluetooth module. Without it, your computer won't find any Bluetooth devices or use them. A z-wave controller is usually a stick or a module attached to your computer. I bought Aeotec Z-Stick Gen5. It's straightforward to add it to OpenHAB. You can do it from Paper UI or HABmin. Go to "Extensions" and add "Z-Wave Binding". The controller will then appear in Inbox and you can add it. You need to at least configure the serial port for your controller. This is to tell what device on your computer is the z-wave controller. You can list all /dev/tty* and find the one that's added when you plug in your stick. Or you can find it from "dmesg". Its name is /dev/ttyACM0 on my machine. Then you configure your controller and set the serial port.

Then you can include your other z-wave devices. I found that each device I bought came with the instruction about how to include it in a z-wave network. But there're some tricky steps. I'll only point out the tricky steps since otherwise it's straightforward from the instruction.
1. Turn your z-wave controller into inclusion mode. The instruction usually says to press a button on the z-wave controller. You don't need to do that. You turn it into inclusion mode from OpenHAB UI. Add a new thing and select "Z-Wave Binding". OpenHAB will make your z-wave controller into inclusion mode for 30 seconds.You should have time to proceed to next steps.
2. OpenHAB may not actually initialize your devices even though it shows up in your thing list on UI, especially for battery devices. I still try to understand how it works. My guess is that OpenHAB needs to wake up the devices a couple times and get the information from the device. This is done at OpenHAB side. Even if you press wake up button on your devices, it may not help. Before it's fully initialized, you may see "Unknown Device" for your device. Sometimes it's showing the right type and name, but channels' values are not available. One way to check whether it's fully initialized is to check /openhab/userdata/zwave/. There's should be an nodeXXX.xml file for your device. XXX is a number assigned to your device. If you turn on debug log when you add your z-wave device, you'll find the node number.
3. Even if the nodeXXX.xml file is created, OpenHAB may still not be able to get device states or send commands to it yet. You may need to wait one day or two. I guess OpenHAB needs to wake up the devices a few times to be able to communicate with it.
4. Paper UI is created for Eclipse SmartHome. It's not tailored for OpenHAB 2. You may find sometimes it's not working as you expect. I suggest to use HABmin.
5. Turn on debug logging for z-wave binding. You can run /openhab/runtime/karaf/bin/client to get to karaf console. And run the command log:set debug org.openhab.binding.zwave.

I spent a couple days trying to figure out why I couldn't add my device. It's always showing "Unknown Device" or its state wasn't updated. Then I first discovered that I didn't need to press the button on the z-wave controller to be in inclusion mode. Then I learned that I may need to try to include/exclude the devices a few time to have the device added correctly. Last, the state may not update immediately, but it will in a day or two. The OpenHAB community is a great resource. There are friendly users and developers help to troubleshoot. It's frustrating at first when things don't work. Then it's exciting when you see you can control from your phone. I encourage you not to be afraid of trying out and obstacles. It's worth the efforts to learn and to improve.

Set up OpenHAB and Development in Docker

OpenHAB is an open source home automation system. It's vendor and technology agnostic and extensible. Every device is supported and communicated through corresponding addons and plugins. In theory, it supports most smart devices no matter whether its protocol is z-wave or ZigBee, etc. It's now in 2.0 beta 4 which is based on Eclipse SmartHome which makes it easier to create new plugin and add new devices.

I like to use Docker too. It's easier to manage production and development environment. As long as I create the Dockerfile, I can rapidly set up the exact same environment to run my OpenHAB. I don't need to worry about differences on the OS, and the configuration is the same.

The official docker image for OpenHAB is based on Ubuntu and it always gets the latest successfully built OpenHAB snapshot which means it may change nightly. Whenever you run the command to create such a container, the OpenHAB version may be different. Besides, I prefer CentOS to Ubuntu. So I decide to create a new Dockerfile for myself. First, it's based on CentOS. second, it always installs the same OpenHAB version unless you change that. It's straightforward to create a new Dockerfile for OpenHAB since I could copy most from the current official docker image. But I did find some nuances that broke the whole things. Below are the changes I made and the workarounds to the problems.

  1. Download OpenHAB 2.0 beta 4.
    • You can find the download link for the version from here.
  2. Install Oracle Java
      • I don't find Oracle Java in CentOS repository and I cannot use yum to install it. Fortunately, I found this Stack Overflow question and used wget to download the rpm from Oracle Java website. Here's the command to download 8u102:
        wget --no-cookies --no-check-certificate --header "Cookie: gpw_e24=http%3A%2F%2Fwww.oracle.com%2F; oraclelicense=accept-securebackup-cookie" "http://download.oracle.com/otn-pub/java/jdk/8u102-b14/jre-8u102-linux-x64.rpm"
  3. Problems in setting up the user and permissions.
    • The official docker image creates a few groups and the user openhab. It's running the daemon as the user openhab. First, some gids of the new groups in the official docker image conflicts the built-in groups in CentOS. Second, For some reason, it seems to me running as the user openhab doesn't have access to /dev/ttyACM0 which is the z-wave device. Even though I set the right groups and permissions in the docker and the permissions in the host. I ended up running everything as root in docker. It's a potential security issue. The process running on docker is running on the host as root too.
  4. Setting up Eclipse in Docker. This is to allow you to develop on the docker image.
    • Eclipse is a GUI. How to work with the GUI? One way of doing it is to pass $DISPLAY to the container and run "xhost +" on the host. This will allow the docker to use the host x server. But doing that is a security concern since you open up your host.
  5. Install maven and ant.
    • Don't install them via yum. That'll pull in OpenJDK. Remember we need Oracle Java to run. So I download the tar files from the websites and set up the $PATH.
  6. Run it.
      • If you don't need to do development, you can keep the command from the original Dockerfile. That is,
        ENTRYPOINT ["/openhab/entrypoint.sh"]
        CMD ["server"]

    Otherwise, you can just comment out ENTRYPOINT and CMD and start shell from the docker image. Remember, to expose all the ports from your docker container via the option "-P" when you start your container.

That's how I set up the docker image. It took a few trial and error. Most were spent on setting up the user and permissions. It's mysterious to me. I think I already set up the permissions right but I still ended up using root. But it works fine in that approach. Development is fine. I hope you can explore OpenHAB world and enjoy it.

OneWay WPF binding Trap

I'm sure you've used WPF binding. There are five BindingMode: Default, OneTime, OneWay,  OneWayToSource, and TwoWay.

OneWay is best for read-only property. But if it's not done correctly, it'll break the binding and the target won't be updated.

Here's how it's broken: call SetValue on the control to set the target directly. What happens here is that WPF will set the value of the target to the one passed in SetValue. That will overwrite the binding. Since the binding is OneWay, WPF cannot set the source, so it need to remove the binding to set the value. After that, even if the source is updated, the target sill stay the same. With that in mind, we must not throw exceptions in the setter of the source and the converter if that exists.

This is something we need to take into account when we set up OneWay binding. Always update the source, instead setting the target. If you're creating a library and set a dependency property on the control directly, you're setting the trap to your library users. This is rather difficult to debug.

Ionic Framework Select on Windows 10 Mobile

There's a problem in touching a select element on Microsoft Edge or on Windows 10 Mobile app.
When you touch the select element, it won't open the options or the options are open twice.
It seems to be interfered with Ionic Framework's tapping (http://ionicframework.com/docs/api/page/tap/).
The solution is simple: disabling the tab system.
For select element, I use

<select data-tap-disabled="true" ...></select>

That solves the problem. Tapping on the select element will open the options correctly.