Short Story of a Long Struggle - Part 1: the announcement that never came...

Back in September I was writing about some HUGE announcement to come related to KDE on Windows. I got everybody holding their breath, even a bit of speculation going on, but then nothing happened. The announcement never came...

So what happened ? What was that announcement about ? And why didn't it come ?

Well, this is the short story of a long struggle...

Our announcement was going to be about the porting of the KDE stack on Windows CE, including parts of kdelibs and kdepim, as well as all their dependencies: dbus, Akonadi, OpenSSL, SASL...

However, just as we were getting the announcement ready after seeing the first kde-based apps running (the first ever was kde4-config.exe) we started hitting wall after wall...

First, we got hit by the drastic process and memory limits of Windows CE5 (on top of which Windows Mobile is based), which were so tight that  even starting the simplest apps would be problematic. Just to name a few: maximum 32 processes can be running system-wide, and each process has a limit of 32MB of RAM, including loaded DLLs. So first we had to go through a complete overhaul of the build process to try and work around those limitations, for example by switching from DLLs to static libraries. This was so that the unused parts of the libs would be automatically discarded by the linker: bringing 20MB of DLL code for only a few used symbols is hardly a good idea when you're short on RAM. We also had to get Qt to go through a severe diet, by removing every single unused feature we could find, and of course having to go through the KDE code to #ifdef out parts that were relying on these features.

Once we were done, our next wall was the Visual Studio emulator, which is a great tool for testing when you don't have an actual Windows Mobile device, or even for faster debugging. But for some reason, the emulator couldn't handle more than 20 file handles being open or so. We looked for alternatives and it took us some time to work around it too, but as far as I know we never really found the cause for that one.

But then the memory issues hit us again: even with a slimmed Qt and static libraries, our executables were now quite large, and taking most of the allowed 32MB of process memory, which is heap+stack+program image+DLLs from all processes (yes, you read it right, but please don't even ask). This this left only very little heap space for the actual program, and that is not good news when you're running a QML app, with lots of images that have to be stored in heap memory. So we had to find a remedy. And it came from our fellow developer Marcus Brinkmann, from g10code, which was our partner on the project. The idea was very simple, and at the same time completely crazy : write a custom program loader which loads the program in high-memory space instead of the process space. This is based on the fact that Windows CE has a huge portion of its virtual memory space that's unreserved, so it can be allocated e.g. for shared memory segments. By managing to get the program loaded in this area (called hi-mem), all the per-process limits are lifted. This seemed insane but it was our only option at that point, so we gave it a try. And guess what : in less than 3 weeks, Marcus got a a fully functional and working hi-mem program loader: himemce.exe ! There are few things that really impress me in programming, but this was definitely one of them ! If you're interested, the code is available as part of gnupg's wincetools repository and should be usable for any program on WindowsCE, not only KDE.

But the dude didn't leave it here : since we could load the program in the hi-mem space, why not allocate our heap data there as well ? Well, no guts no glory ! So Marcus did it again, and faster than you can say "shallow grave" he brought us his infamous dlmalloc, which is a malloc() replacement that allocates memory blocks in the hi-mem space. The last step was just to integrate that into our program as the underlying malloc() used by the operator new. And even better, by reimplementing qMalloc() we could even get all of Qt's heap allocations to be done into high memory ! Success at last !

So, with our applications now running without hitting the platform limits, you may think this is where the story ends... Happy ever after ? Well, yes, if you're in a Disney movie. But this is the real world, and ponies don't stay pink for long...