Notes on booting an Android kernel from scratch

Many people reading this will have experience building and booting a kernel, but bringing up a kernel on a new device is a different beast. It's not just about compiling the kernel and flashing it; it's about understanding the hardware, the boot process, and how the kernel interacts with the device. It's about reverse engineering, debugging, and sometimes even writing your own drivers. Without access to the original source code, you are essentially flying blind. You have to figure out how the hardware works, what the kernel expects, and how to get it all working together. It's a process of trial and error, and it can be very frustrating at times. Add to that the complexity of the Android ecosystem, with its layers of abstraction and proprietary components, and you have a recipe for a challenging but rewarding experience.

Most of the pain isn't the kernel itself; it's the device tree. In case you are unfamiliar, the device tree is a data structure that describes the hardware to the kernel. It tells the kernel what devices are present, how they are connected, and how to initialize them. The kernel uses this information to set up the hardware and provide drivers for it. If the device tree is wrong, the kernel won't be able to use the hardware correctly, and you might end up with a non-functional system.

I had the opportunity to work with the HTC U12 Life (codename iml_dugl), which is a strange piece of hardware from 2018. From my research, Longcheer (a Chinese OEM) designed the board, and HTC rebranded it. On top of that, the phone has no publicly available kernel source or even software packages which made it astronomically difficult to get the device tree right. I had to reverse engineer it from extracting the boot image, decompiling the bootloader, and decompiling the device tree blob (DTB) from the boot image which is just a fancy term for a binary file that contains the device tree. The DTB is compiled from a source file called a device tree source (DTS) file, which is a human-readable text file that describes the hardware. This job however was not made easy by the fact that the dtb not only has multiple overlapping definitions but also a lot of dead code that is not used by the kernel.

Even if you know which parts of the DTB are actually used, decompiling it is not as simple as it sounds as it produces a lot of artifacts, broken nodes, and missing properties. I had to manually clean up the decompiled DTS file and add the missing properties by looking at the bootloader code and the kernel source code. This process was very time-consuming and error-prone, but it was necessary to get the kernel to boot.

Another challenge is that you can't just compile the device tree source you modify without a kernel in the traditional way. The stock kernel needed to be modified to include the new device tree binary and had to be re-flashed to the device every time I made a change to the device tree. I had already unlocked the bootloader so this process was a lot of fun but it was also very slow. I had to wait for the device to boot up every time I made a change, and if the kernel panicked, I had to wait for it to reboot and try again. This process was very frustrating, but it was also very rewarding when I finally got the kernel to boot.

After I had a booting device tree, I decided to use the kernel source from the Code Aurora Forum (CAF) which is a repository of kernel sources for Qualcomm devices, as a starting point. This was a good decision as it had better support for the hardware and it was easier to work with. I knew from talking to other developers and community members that jumping to a modern kernel would be a nightmare so I decided to stick to a kernel source from the same release, albeit a different subversion. The CAF sources were not perfect, but they were a lot better than the mainline sources and they had a lot of the drivers and features that I needed.

The initial boot was a huge milestone, but it was not too hard after getting the device tree working. To speed up the debug cycle, I decided to build TWRP (a custom recovery) instead of the full Android system. This was a good decision as it allowed me to test the kernel without having to worry about the complexities of the Android system. I could just flash the recovery image and boot into it, and if it worked, I knew that the kernel was working. If it didn't work, I could just re-flash the recovery image and try again. This process was much faster than trying to build and flash the full Android system every time.

The last and arguably the most important thing to tackle before switching over to building Lineage OS was to get the display working. This was a huge challenge as the display driver was not well documented and the driver sources for it were not readily available. Searching through github I found a lot of hits for the display driver but they were for a Mediatek based chipset. At this point, I could have taken that code and wrote my own driver for the display, but after scanning through a few search pages, I hit one of the sole repositories that had the display driver for this specific driver. With this the kernel was at a stage where we could start working on creating the trees to build Lineage OS. That is still ongoing and hopefully, I will have a blog post about that process in the near future.

PS: I could've tried hard to find the exact Longcheer sources for the device, but I think that would have been a disservice to the community. The point of this project was to show that it's possible to bring up a kernel on a device without access to the original sources, and I think that I achieved that. The process was difficult, but it was also very rewarding, and I hope that it inspires others to try it out for themselves.

More to come as the bringup continues. Check out the GitHub repository in the meantime.