Porting Candle to Linux

My experience of porting Candle from Windows to Linux is really like riding roller coaster, with up and downs, surprises and shocks, screamings and desperations all along the way. With over 100,000 lines of Candle source code, and with no previous experience of C development on Linux, the journey was really not for faint-hearted. The most difficult part was actually getting used to the C/C++ tool chain on Linux. I'm probably too much spoiled by Visual C/C++.

Setting Up the Linux OS

For the Linux OS, I used Ubuntu. Initially I ran it within VirutalBox, but later as I really started cranking and compiling the source code, I found the Eclipse CDT compiling to be too sluggish. However my experience with Eclipse Java on direct Windows machine was much better than that, so I suspected the performance issue was related to the VM. As I switched to dual booting and ran Ubuntu on the hardware directly. The response of Eclipse CDT, especially the compiling performance is indeed much better. Regarding compiling, the GCC turn around time is still not as good as VC, but acceptable.

I would say the Ubuntu desktop environment is quite intuitive to use. Once I mounted my Windows hard drives and with FireFox running, I felt quite at home. I didn't have to install anything to play video files (including FLV files), watch DVD, license to music and view PDF files. However the file manager under Ubuntu, Nautilus, spoiled the whole good mood. It is really not as user friendly as Windows File Explorer; as of today, I still don't know how to use it to recursively search for a file under a directory.

There were things that still have to resort to command line; for example, mounting Windows hard drives. And I found that the most convenient and common way of installing software on Ubuntu is using "sudo apt-get". Couldn't there be a UI for apt-get? May I'll build one with Candle when I have time.

So far all my problems could be solved by Ubuntu forum pages found by Google. So thanks to Google and the helpful Ubuntu forum.

Once had the OS conquered, the next thing was to setup the C/C++ tool chain. I would say, setting up the C/C++ development tool chain is much more troublesome than porting the Candle source code.

 

Setting Up the C++ Tool Chain

First thing in the tool chain is of course the IDE. I don't think the suggestions given by some of gurus in the StackOverflow page, saying that Linux command line is the best IDE, is a valid argument. This really old school thinking. I settled with Eclipse CDT primarily because I'm familiar with Eclipse Java, and it gave me almost all I wanted from an IDE. I did give Code::Block a try, but it crashed right in front of me when I was still trying it out, so I thought it would not be suitable for serious work.

The next thing in the tool chain is the compiler and the build tool. On Linux, the compiler has to be GCC, I don't think there's other decent choice.

The build process on Linux is really a nightmare. And I think GCC has to be the one to be blamed. If VC is anywhere better than GCC, the build process has too be the first on the list. Building C/C++ code on Linux is still a barbarian work on Linux.

The compilation requirements for Candle source code is not complicated. Eclipse CDT's generated makefile is already sufficient to compile Candle source code. However, this is not good enough; I cannot expect everyone else to use Eclipse CDT. The build process has to be commonly accepted on Linux. That's where the problem comes in.

Is there a standard or common C/C++ build process on Linux? Yes and No. Yes, if you accept the least denominator - makefile. No, because manually writeing a makefile is not an option for me. So I needed a tool to help us generate make file. The tool offered from GCC family is the auto-tools; I think there are enough criticisms on the auto-tools in the community and I am not going to add more here. As I searched, I found CMake and SCON and there are big projects using them. I indeed gave both of them a try, but none of them were good enough for me to 'fall in love with them on the first glimpse'. CMake feels bulky and did not even work on my Windows platform. The problem was that CMake expected MS CRT header files, but that's something I had spent great effort to get rid of on Windows (that's a separate story anyway), so this made me dislike the approach of CMake. It tries to be smart but tool smart for me. I just wanted a simple tool to generate a makefile for me from a bunch of C/C++ files, I didn't need it to detect the platform or libraries for me, I could figure out those myself. I did not choose SCON, because it deviates from makefiles and I don't think that's commonly accepted on Linux, and its own build process does not integrate with Eclipse CDT well - I don't want to switch to command line every time I need to compile the code. I also tried to find if there's any decent tool that can convert VC project files into make files, tried a few and none really work. I then desperately refreshed my knowledge with makefile (which was more than a decade ago) and tried to manually write a make file myself, but I soon gave it up, it is so barbarian.

But as I was driven crazy, mad, angry, and was almost about to give up, I hit upon a tool called Premake and fall in love with it immediately. A Chinese poem best describes that  - 众里寻她千百度，蓦然回首，那人确在灯火阑珊处 (I searched for my beloved for a thousand times and failed; but as I turned around, she is just around the corner! Please pardon my bad translation)。Premake's design philosophy rhythms with what I have in mind - "the VC build model is so simple and intuitive, why can't we have something similar on Linux"? Premake models after VC, which it a very low learning curve for me. It's binary executable file premake4.exe is only 234K; how many utilities can you find today that are still of such a small size? Premake uses Lua as its configuration language, which is also a smart and right choice, comparing to other possibilities like XML (which would be too verbose) or another DSL. I managed to get Premake to work on my first attempt and easily integrated it with Eclipse CDT. Because of all these beauties of Premake, I even made donation to the project.

With the hurdle of the build process gone, I then moved onto version control tool. On Windows I've been using SVN and TortoiseSVN for years. Unfortunately, TortoiseSVN is not available on Linux. The closest is RabitVCS, but there's one major flaw of it - it may freeze Nautilus sometimes. I tried other SVN client tools but as I'm so used to the shell-integrated approach of TortoiseSVN that I have no choice, but to bear with RabitVCS for the time being.

And for the version control process to work, I gave myself another tough requirement. I want the Windows and Linux versions to not only share the same source code repository, but also the same working directory. The reason for that is that sometimes, I would forgot to check in my changes on the other platform, than I would have to specifically boot into to the other OS to check in the changes and switch back. I initially used local repository file://c:/... on Windows, and that was not accessible to Linux. So I had to switch to a remote repository like http://... for it to be cross platform. However, I don't want the source code to be hosted on some server, just for my personal development work. The response also won't be good. So how to have repository with http://... kind of address and still resides on local file system? I praise myself to figure that our myself, as I tried but did not find any answer from Google. The solution is to to use a local SVN server. It's not too difficult to setup Apache and mod_webdav on Windows and Linux. Once that done, there's no more major inconvenience in the tool chain of cross platform development, except the dual boot itself, which we cannot get rid of.

The final few secret weapons in the Candle tool chain are Gradle and Doxygen.

The reason I added Gradle in the tool chain is that I need an automated file processing tool, so as to pick and choose files to be released to the public. Initially, I wanted to used Premake, my beloved build tool for the job. Premake does allow some kind of extension mechanism for custom actions, but this capability was not fully developed and thus could not fulfill my requirement. I knew there are shell scripts for this kind of work, but they are not portable (don't tell me to use Cygwin). I also knew Ant quite well, as I had used it for a project before. But its XML syntax is too verbose. So I searched for Ant alternative and found Gradle before long. Gradle uses Groovy as the scripting language. However, because it depends on JVM, there's an obviously lag for the script to start up, comparing to Premake. Just hope one day, Premake can be extended to do the same work. (Gradle is a tool I used for my personal workspace management, it is nor required for Candle development.)

Doxygen is probably the best documentation generation tool. There's JavaDoc, but it is only for Java. The part I like Doxygen most it that it ran right out of box without any initial configuration. There is a simple GUI, and documents generated with the default settings are already of good quality. With some further twisting of the flags and settings, the documentation was already presentable. Its syntax and tags are quite simple to master. I really did not expect Doxygen to be so simple to use. If there's anything more I want out of this is probably to write some transformation templates (using Candle) to convert the XML documents into something more stylish.

For static analysis, I tried Cppcheck. But it ran very slowly and did not find any significant error in Candle source code. So I ditched it.

That's all for the tool chain.

There might be other tools added to the tool chain in future, especially in the automated and integrated testing area. I did write some code to do the regression testing in Candle. And I found my own code to be much more useful than existing C/C++ test framework, like CppTest. I don't see much value-add from those test frameworks. And there's plumbing code required for a testcase in such framework. I expect some kind of UI, like a table, for me just to key in the input values and expected output of the test cases, each case in a row. May be Candle, can be used to do that one day.

 

Porting Candle Source Code

Porting Candle source code itself is actually easier than setting up the tool chain.

As for me, VC is much more productive than the tools on Linux, I'd rather write as little code as possible on Linux.  So before I really started coding and testing on Linux, I refactored Candle source code to isolate platform specific code, and even calling to the standard C libraries, to two low level modules (Native and Gui). The refactoring took quite some days to finish, but was really worth the effort. It helped substantially smooth my subsequent porting to Linux. Later when I actually compiled Candle source code on Linux, the high level modules like Query and Scene almost compiled and ran without any modification (except some small syntax compatible issues with GCC). This refactoring exercise also made Candle source code more structured and portable.

The biggest surprise, or I would say shock, I got when porting Candle source code was that the wchar_t in GCC is 4-byte wide by default.  I really took it for granted and assumed that wchar_t was 2-byte wide as on Window. It took me a few days to find out this bug. This bug was tough to find, because one of my debugging weapons is extensive logging, but this weapon was disabled as the bug was exactly within the logging and string processing. The debugger was able to capture the exception and showed me that there was segmentation fault. But the call stack and lines around the exception point all looked innocent. I really pulled all my hair out. How come there's no article on the web that warned people about this.

After fixed this wchar_t bug, other porting was not too difficult, some rewriting and debugging here and there were all expected and reasonable. The porting of the GUI from Windows to GTK was easier than I thought. Both GTK and Cairo have quite clean APIs and reasonably good documentation in general. I did stumbled on Pango and Cairo integration for a while. There's good documentation on how to use Pango and Cairo individually, but not sufficient on how to get the two to work together. One thing I don't like about GTK is the signal or even handling functions. I found it strange that GTK used string to designate the signal to be registered, instead of using enumeration. String is prone to typo errors, which cannot be detected at compile time, and there's no clear and exhaustive listing of the exact string values to be used in the documentation.

After all these storms and turbulences, I finally crossed the deep and wide ocean between the platforms, and had my Candle ship landed on the new continent of Linux. There were difficulties that almost sank my whole ship, but thank God, that He made me persevere to the end. The journey was rough but rewarding in the end. Much has been learned and can't be learned in any another way.

There are many benefits of writing portable C/C++ code, comparing to stick to one platform:

• You application reach wider range of users;
• You enjoy the best of both worlds, e.g:
• Different compilers help detect different static errors in your code;
• You can use VC to develop and debug source code that can be reused on Linux (as I think VC is still the best IDE for C/C++);
• Your design is more general and does not tie to a platform; you'll know how to wrap and manage your dependencies;
• You'll be required to use portable libraries; personally I think, portable libraries and softwares tend to be cleaner and more open;
• You'll be required to be open and stick to cross-platform standards;

In the end, you may ask whether it is worthwhile to develop using portable C/C++ code? Why not just use portable languages like Java and Python in the first place. Yes, I won't advise anyone to use C/C++ to develop high level applications. But the Candle engine itself, is like another JVM; while it might be possible to take Scala's approach to target and reuse the JVM, the performance of Candle won't be good. Enough has been written for today, and I'll leave the discussion on that to another day.