Robert Schusters Blog

Comments on latest post on OpenJDK

Hi,

there have been interesting questions in the comments sections of my last post. I take the time to answer them here:

Mark Wielaard asked: Finally, do you have any speed comparisons of the hotspot/zero and cacao on arm ?

Unfortunately not. Subjective it feels as if the SwingSet2 demo runs better with Cacao. The Freerunner and its software are also not in a good shape to make performance tests. Even native C applications show unacceptable slow behavior.

Xerxes Rånby said: Your combined effort have provided the key to unlock the possibility to run OpenJDK on all embedded Linux NAS storage devices, home routers and more

Indeed and I very much welcome people to try compiling the recipes for their MIPS devices as well. I also wonder if OpenJDK can be compiled against uClibc and uClibc++. If that is the case then we can also have OpenJDK+Hotspot/Zero on AVR32 (though a few more patches to the build system will be needed for this).

Andrew John Hughes wrote: What version of IcedTea did you use? I think some issues have changed/been fixed. Certainly –with-openjdk-src-dir isn’t broken any more

I used the latest from the stable release: 1.3.1. Good to hear that some issues are already fixed. I am looking forward to integrate some patches after Xmas.

Eric Herman wrote: My sense is that Java is not truly Free if I can not bootstrap it from C, and I was of the impression that I still needed a working Java in order to build Java. I look forward to learning more about the excellent work you’ve been doing.

Actually with OpenEmbedded this is the case: OpenJDK is bootstrapped without the need for a pre-installed Java runtime. What I left out in my explanation is that all of this builds on top of my previous work on a completely self-hosting Java^H^H^H^HJava-like toolchain based on GNU Classpath, JamVM/Cacao, ECJ and eventually Jikes. I have written down the gory details of this in the OE wiki to make it possible that anyone can pick up my work enhance it or re-use it in another build environment.

Finally Wladimir Mutul asked: Why not use Scratchbox(.org) for your cross-compilation ?

Before starting with OpenEmbedded Jalimo was only targeting the Nokia Internet tablets and as such we build our stuff with Scratchbox. First of all: I really like the Debian-way of packaging things (all of the Java related recipes in OE resemble Debian conventions for instance). However three points were most important when deciding to leave the Scratchbox path:

• At the time we worked with SB it was not possible to either get JamVM or Cacao working reliably inside SB. So we had to use Jikes as a Java compiler. GNU Classpath adopting Java 1.5 syntax would have made it neccessary to use ECJ instead which needs a Java runtime.
• OpenMoko appeared on the scene and we wanted to provide binaries for this platform. Additionally we had an Irex Iliad (an Ebook reader) and wanted to support it, too. OpenEmbedded built for those out of the box and it turned out that with little work OE could be modified to support Maemo as well.
• Last but not least with SB a lot of manual work was involved when integrating a new patch or updating to a new version. With OE it is a matter of renaming a recipe and checking which patches are still needed. Gentoo packagers do it in the same way.

It should go without saying that I miss things that would be available by basing everything on Debian. E.g. source packages, direct support for Ant builds, many many tried and tested packages and of course GCJ. Furthermore it isn’t all bright and shiny with OE either. For instance the project is an extremelyfast moving target. People add new stuff to it everyday. Some things have effects on your recipes and may cause build problems. However this is why projects like Poky exists which make stable release on top of OE snapshots.

I hope my explanations cleared things up a bit.

Happy holidays!

Serving cross-compiled OpenJDK with IcedTea

As Ken and Sebastian have already announced it OpenJDK integration into Jalimo was finished. However there was a bit of work to do to not only compile OpenJDK but to also package it nicely. This work is now also completed.

Additionally – and this was not told anywhere else yet – we can now also offer Cacao+OpenJDK. So anyone who is in need of a decent JIT for its target platform can now build this combination, too.

Those who do not know OpenEmbedded may wonder what is so special about the work I have done in the last weeks. Well, the special thing is that we are cross-compiling the OpenJDK. That means the machine on which the JDK is built is of a different kind than the one on which we want to run it later on. The difficulty stems from the fact that the OpenJDK build system is not designed for this (in contrast to the one used by PhoneME btw).

Before I will tell you about the guts of cross-compiling OpenJDK lets enjoy some screenshots:

People may have seen these apps on their desktops and as such they are not very exciting. However for me they have a special meaning: As a contributor to GNU Classpath it was always my wish to be able to run any Java (=J2SE) program on any device running a Free operating system. I was contributing to the Free Swing implementation and together with the enormous work done by Roman, Thomas, Lillian and many others we were able to run a few Swing and AWT programs. Still performance, completeness and correctness of our implementation where limited in many places and would require more years of dedicated effort. Thanks to Sun releasing Java as free software and the important work done by the IcedTea team we can now have the real thing on our devices and suddenly get 100% compatibility.

Now on to the cross-compilation guts:

First of all compared to the work that would have been required with plain OpenJDK IcedTea made the effort of cross-compiling the thing a breeze. The best thing that IcedTea provides is the ability to use a GNU Classpath-based toolchain to build OpenJDK. On the major distros GCJ is used as a runtime and ECJ as the bootstrap Java compiler. In OpenEmbedded we have JamVM, Cacao and plain GNU Classpath as runtime options which work equally well. JamVM does not understand some of the “-X”options so I had to patch their use away.

Although IcedTea is a nice environment I had to patch a few things to get things started. E.g. IcedTea requires you to point to a GCJ home directory. It will then create symlinks to libraries and headers files contained within them. The problems with IcedTea are: When cross-compiling the libraries provided by your system’s GCJ do not make sense for your target (e.g. your system GCJ has AMD64 binaries while your target requires those for ARM). Secondly in a cross-compilation environment like OpenEmbedded you do not want to rely on stuff outside the environment. As such I had to modify this behavior to link to header files (jni.h and likes) which are provided by the OE-built GNU Classpath.

There where other problems like IcedTea not respecting the –with-openjdk-src-dir option. It will still try to download the OpenJDK sources itself (which is not allowed in OE, because this is done through the environment).

All in all I collected my changes in patches which I called ‘build hacks’. I consider these hacks fixable and will work together with the IcedTeam team to resolve those for future releases in order to make cross-compiling OpenJDK even easier in the future.

Besides IcedTea I had to patch OpenJDK itself to get things compiled properly. The first thing that causes trouble is that you cannot chose the compiler being used. OpenJDK contains some complicated makefiles that check your system environment and then decide which compiler to use. I used good old sed to replace the respective part of the makefile. I believe this behavior can be added to IcedTea without causing any harm to the other IcedTea users.

The next problem is the sanitizing step. This one will compile a few binaries (for the target platform!), run them (impossible) and decide on their output whether the build can continue. These checks are done for CUPS and Alsa. The fix is to patch them away. I think an optional –disable-sanity-checks would be OK for IcedTea.

The final big problem with OpenJDK’s build system is that it uses the result of ‘uname -m’ for its decision. This is troublesome because ‘uname -m’ will only tell you the architecture of your build machine not the target one. I solved this by replacing all these calls with a variable and allowed IcedTea’s makefile to provide a value for this.

The remaining issues where minor: Some unsuitable paths here, a sizer.32 binary that I had to take from a previous non-cross build there.

Apropos non-cross build. This was the really ambitious undertaking. But before I explain this here are some cross-compilation basics that I learned in the years contributing to OpenEmbedded:

Non-cross compilation aware projects are troublesome when they create binaries whose output is going to be used directly in the build. The general approach to this problem is that you first compile this particular binary for your build system and use it in the cross-build. In OpenEmbedded this will often result in a separate build recipe. If your project is cross-compilation aware it will allow you to specify the location of such binaries. E.g. Cacao 0.99.x has the –with-cacaoh option to point to the location of the non-cross-compiled header generator. In non-cross-compilation aware projects you need to patch the makefiles accordingly.

The IcedTea build normally consists of two builds of the JDK. The first one creates a bootstrap JDK. This one is heavily patched, stripped down and can be build using GNU Classpath-based software. In the second step the bootstrap JDK’s java, javac and a few other binaries are run to build the final JDK. For our cross-compilation effort this means we need to get this bootstrap JDK built for the build machine and as said above this is where things get troublesome.

The reason for this is that OpenJDK depends on a few libraries like CUPS, Alsa, libjpeg and giflib which I either had to provide in their native (= for the build system) form or patch their use away. Remember that I cannot just take e.g. Debian’s libcupsys-dev because that would be outside the OpenEmbedded build environment (and would make people unhappy who use OE on a different distro).

I decided to go the way of patching the bootstrap JDK build since printing and sound would not be needed for bootstrap purposes anyway. Originally I also disabled most of the AWT (and as such the libxt, xproto dependencies). However it turned out that at one point the OpenJDK build converts a bunch of GIF pictures into Java byte array source code using javax.imageio which requires the headless variant of the AWT.

A few lines above I told you that the bootstrap JDK is compiled separately for the build machine. In a simple project we could now skip this part from the cross-compilation build. However the bootstrap JDK build also creates libraries (for the target machine) to which the final JDK links against. That is why we cannot skip this part.

So cross-compiling OpenJDK consists of the major steps:

• build a GNU Classpath-based toolchain (gjar, gjavah), ECJ and Ant
• build the bootstrap JDK for the build machine (‘make icedtea-against-ecj’)
• build the bootstrap JDK for the target machine
• replace the binaries in bootstrap/icedtea/bin with those of your native boostrap JDK
• build the final JDK

and as I have written in the first part of this posting: You need to patch everything heavily to get some problems out of the way which stem from the fact that OpenJDK is not cross-compilation aware.

Finally a screenshot showing Cacao+OpenJDK:

The next step will be integrating Hotspot-Shark into OpenEmbedded and of course getting as many of my patches upstream as possible.

Compared to GNU Classpath + Cacao/JamVM the OpenJDK packages are ~6 times larger. I am curious to see how the modularization efforts of Java7 works out. With the work done now it will be much easier to follow those developments.

Edit: Fixed images links.