Windows build system: halfway status

Time files and we're already halfway through 2011's GSoC. There were quite a lotof reactions to the initial announcement of my project and to my previous postabout it, from people who were all excited to see Mapnik become easier to buildand use on Windows. It was very encouraging and my apologies for not gettingback to everybody yet; I thought it would be more be productive to sollicitfeedback on something working.

The implementation of everything that was planned was quite vague when theproject began at the end of May, but I'm really excited about how things areturning out. While I didn't quite get as far as I had liked, the genericness ofwhat I'm building to support Visual Studio for Mapnik is imo very cool and Ihope that other projects besides Mapnik can also take advantage of it. Of courseI'm focusing on the needs of the Mapnik project for now, and in this post I'dlike to explain how I see it work for future users trying to build Mapnik withVisual Studio, how it works under the hood, and what that means for the future.

How the new windows build system should work

I used my own experiences building not only Mapnik but quite a bit of other opensource products that are primarily developed on Linux to lay out a list ofrequirements that a system for supporting Visual Studio should fulfill. Orderedby importance, those were:

• 'one-click' deployment for the user. No long lists of steps to follow, or settings to change etc.
• Only rudimentary Visual Studio experience required. It can be quite
• Easy to use by the developers. For every action the library developers need to take to support another environment as the one that is mainly used for development, the chances that it will actually be maintained and work well go down. So ideally, support for Visual Studio should be transparant (require no extra work).
• No extra dependencies or additional software to install. Everything should work with the tools that come with a standard Windows and Visual Studio installation; that means no installing Perl or Python, Cygwin, other build tools or utilities.
• Transparency wrt the library dependecies - change as little as possible, and preferbly nothing, to any code to be sure that everything works as intended and tested by all respective developes. Unfortunately, in some cases where certain code isn't well-tested to build or function using Visual Studio there is no option but to make changes.
• Not keeping a copy of library dependencies in the Mapnik source tree. With copies of code it's error-prone and laborous to keep up to date with new releases and security fixes; it's better to always use the canonical source.

Basic workflow

With these in mind, I have the following workflow in mind for anyone wanting tobuild Mapnik on windows:

• Download and extract Mapnik release zip file from Mapnik website.
• Download and extract all library dependencies automatically by running a batch file. This makes it 100% clear to the user that the code used is the original release as intended by the library developers, and which version of the code is used.
• Configure and where needed patch the library code. By 'configure' I mean copying configuration files like the ubiquitous 'config.h' for which many libraries provide a 'config.h.vc' or similar skeleton; this mechanism isn't standardized though so to do these things manually, users would have to follow specific procedures that are slightly (or very) different for each library. By putting all these command in a simple batch file, it can all be automated but still easy to trace what is being done. By 'patching' I mean where needed make changes to source code get it to compile with Visual Studio. As mentioned before, unfortunately there are cases where there is no other option.
• Open the Mapnik solution and click 'build', or alternatively build everything from the command line, again with a transparent batch file.

For the user that isn't interested in the behind-the-scenes, these steps canthemselves be put into a batch file so that after downloading and unzipping theMapnik release zip file all further steps are run automatically while the usergets coffee or browse's reddit.com/r/funny.

What this solution is not

Now, the naive way to go about this is write a bunch of batch files, prepare theVisual Studio solutions/project files manually and put them in the MapnikSubversion to be included in each release. This has a number of obviousshortcomings:

• Requires developers to update project files manually, using Visual Studio, when new files are added to Mapnik.
• Requires manual tracking of batch file contents, downloads, new releases of libraries etc.
• Cannot be re-used - all one-off, only for Mapnik project.

Python to the rescue

So clearly this is not a viable strategy. Instead I wrote a Python program that,given a description of the source files that make up a program or library andsome settings like precompiler flags, will generate Visual Studio project filesand solutions, complete with the dependencies and all the rest required to buildthe program. The program, dubbed 'vs4os' for 'Visual Studio For Open Source',also generates the batch files mentioned above and WiX files, which are inputfiles for the open source WiX installer builder - a system to create MSIinstallers from an XML input description. Right now only Visual Studio 2008files are generated, but this abstraction allows easy generation of variousversions of files, too.

How VS2OS will be used

How do I see this being used in practice, then? The vs4os program is run at themoment the Mapnik (or any other program or library that uses vs4os) release ismake; for example as part of the 'make release' make rule. The project files andgenerated batch files are then distributed with the release. This way theend-user doesn't need to install or download anything besides the zip file. Ofcourse, developers could also choose to keep track of the generated files in thesource tree of a project; every time files are added to the project, the fileswould need to be re-generated, but that could easily be done as part of a cronjob or post-commit hook.

Next Steps

At the moment, the list of files that need to be in the project files need to bespecified manually. That means that the goal stated above about the system beingtransparent to the developers isn't fulfilled 100% yet. A possible extension tothe vs4os operations is that it would read the list of files from the main buildsystem used, for example the autotools chain or Scons. Often somewhere in such abuild system, there is a list somewhere of what files are included; by havingthe program read directly from there, there would be one canonical source offiles to build and nobody would have to even know that there is a parallel buildsystem being generated from the primary sources.

So, what is the working status of all this? As much as I would have liked tohave a complete Mapnik build system to show along with this post, there arestill a few project that I need to further describe in the vs4os input languageto get everything working from start to finish as described above. At the momentthere are about 20 libraries that are used when building Mapnik (directly orindirectly), and each of them has ideosyncracies in their build system that Ineed to delve into. Some, like ICU and Boost, have such complex system but alsofirst-class support for building with Visual Studio that I've decided not tore-invent the wheel there and use those as-is.

Following along

You can track the current stateof affairs in my branch in Mapnik SVN at svn.mapnik.org/branches/gsoc2011/roel;this branch does not contain the generated project files yet, so you need tohave Python installed and run windows\\vs4os_files\\build_all.bat to try it out.Once I have my first full working set of files done, I will put them upsomewhere and I hope that there are people who are willing to give buildingMapnik with Visual Studio a shot and send me their feedback.

I'm afraid this post has gotten way too long already, so I'll cut it short here.Of course I'm very interested in hearing from everybody who has contacted me orthe list with Visual Studio questions hwo you think about this all - will itserve your needs? Do you have any special requirements in your build environmentthat need to customized in a specific way?

As I mentioned in the beginning, I'm really excited about how this is turningout, and I hope that it will prove useful in the future both for the Mapnikproject and beyond.