Anyone who has written a non-trivial project using Autotools has realized that (and wondered why) it requires you to be aware of 5 different languages. Once you spend enough time with the innards of the system, you begin to realize that it is nothing short of an astonishing feat of engineering. Engineering that belongs in a museum. Not as part of critical infrastructure.
Autotools was created in the 1980s and caters to the needs of an entirely different world of software from what we have at present. Worse yet, it carries over accumulated cruft from the past 40 years — ostensibly for better “cross-platform support” but that “support” is mostly for extinct platforms that five people in the whole world remember.
We've learned how to make it work for most cases that concern FOSS developers on Linux, and it can be made to limp along on other platforms that the majority of people use, but it does not inspire confidence or really anything except frustration. People will not like your project or contribute to it if the build system takes 10x longer to compile on their platform of choice, does not integrate with the preferred IDE, and requires knowledge arcane enough to be indistinguishable from cargo-cult programming.
As a result there have been several (terrible) efforts at replacing it and each has been either incomplete, short-sighted, slow, or just plain ugly. During my time as a Gentoo developer in another life, I came in close contact with and developed a keen hatred for each of these alternative build systems. And so I mutely went back to Autotools and learned that I hated it the least of them all.
Sometime last year, Tim heard about this new build system called ‘Meson’ whose author had created an experimental port of GStreamer that built it in record time.
Intrigued, he tried it out and found that it finished suspiciously quickly. His first instinct was that it was broken and hadn’t actually built everything! Turns out this build system written in Python 3 with Ninja as the backend actually was that fast. About 2.5x faster on Linux and 10x faster on Windows for building the core GStreamer repository.
Upon further investigation, Tim and I found that Meson also has really clean generic cross-compilation support (including iOS and Android), runs natively (and just as quickly) on OS X and Windows, supports GNU, Clang, and MSVC toolchains, and can even (configure and) generate XCode and Visual Studio project files!
But the critical thing that convinced me was that the creator Jussi Pakkanen was genuinely interested in the use-cases of widely-used software such as Qt, GNOME, and GStreamer and had already added support for several tools and idioms that we use — pkg-config, gtk-doc, gobject-introspection, gdbus-codegen, and so on. The project places strong emphasis on both speed and ease of use and is quite friendly to contributions.
Over the past few months, Tim and I at Centricular have been working on creating Meson ports for most of the GStreamer repositories and the fundamental dependencies (libffi, glib, orc) and improving the MSVC toolchain support in Meson.
We are proud to report that you can now build GStreamer on Linux using the GNU toolchain and on Windows with either MinGW or MSVC 2015 using Meson build files that ship with the source (building upon Jussi's initial ports).
Other toolchain/platform combinations haven't been tested yet, but they should work in theory (minus bugs!), and we intend to test and bugfix all the configurations supported by GStreamer (Linux, OS X, Windows, iOS, Android) before proposing it for inclusion as an alternative build system for the GStreamer project.
You can either grab the source yourself and build everything, or use our (with luck, temporary) fork of GStreamer's cross-platform build aggregator Cerbero.
Update: I wrote a new post with detailed steps on how to build using Cerbero and generate Visual Studio project files.
Second update: All this is now upstream, see the upstream Cerbero repository's README
Personally, I really hope that Meson gains widespread adoption. Calling Autotools the Xorg of build systems is flattery. It really is just a terrible system. We really need to invest in something that works for us rather than against us.
PS: If you just want a quick look at what the build system syntax looks like, take a look at this or the basic tutorial.
Autotools was created in the 1980s and caters to the needs of an entirely different world of software from what we have at present. Worse yet, it carries over accumulated cruft from the past 40 years — ostensibly for better “cross-platform support” but that “support” is mostly for extinct platforms that five people in the whole world remember.
We've learned how to make it work for most cases that concern FOSS developers on Linux, and it can be made to limp along on other platforms that the majority of people use, but it does not inspire confidence or really anything except frustration. People will not like your project or contribute to it if the build system takes 10x longer to compile on their platform of choice, does not integrate with the preferred IDE, and requires knowledge arcane enough to be indistinguishable from cargo-cult programming.
As a result there have been several (terrible) efforts at replacing it and each has been either incomplete, short-sighted, slow, or just plain ugly. During my time as a Gentoo developer in another life, I came in close contact with and developed a keen hatred for each of these alternative build systems. And so I mutely went back to Autotools and learned that I hated it the least of them all.
Sometime last year, Tim heard about this new build system called ‘Meson’ whose author had created an experimental port of GStreamer that built it in record time.
Intrigued, he tried it out and found that it finished suspiciously quickly. His first instinct was that it was broken and hadn’t actually built everything! Turns out this build system written in Python 3 with Ninja as the backend actually was that fast. About 2.5x faster on Linux and 10x faster on Windows for building the core GStreamer repository.
Upon further investigation, Tim and I found that Meson also has really clean generic cross-compilation support (including iOS and Android), runs natively (and just as quickly) on OS X and Windows, supports GNU, Clang, and MSVC toolchains, and can even (configure and) generate XCode and Visual Studio project files!
But the critical thing that convinced me was that the creator Jussi Pakkanen was genuinely interested in the use-cases of widely-used software such as Qt, GNOME, and GStreamer and had already added support for several tools and idioms that we use — pkg-config, gtk-doc, gobject-introspection, gdbus-codegen, and so on. The project places strong emphasis on both speed and ease of use and is quite friendly to contributions.
Over the past few months, Tim and I at Centricular have been working on creating Meson ports for most of the GStreamer repositories and the fundamental dependencies (libffi, glib, orc) and improving the MSVC toolchain support in Meson.
We are proud to report that you can now build GStreamer on Linux using the GNU toolchain and on Windows with either MinGW or MSVC 2015 using Meson build files that ship with the source (building upon Jussi's initial ports).
Other toolchain/platform combinations haven't been tested yet, but they should work in theory (minus bugs!), and we intend to test and bugfix all the configurations supported by GStreamer (Linux, OS X, Windows, iOS, Android) before proposing it for inclusion as an alternative build system for the GStreamer project.
You can either grab the source yourself and build everything, or use our (with luck, temporary) fork of GStreamer's cross-platform build aggregator Cerbero.
Update: I wrote a new post with detailed steps on how to build using Cerbero and generate Visual Studio project files.
Second update: All this is now upstream, see the upstream Cerbero repository's README
Personally, I really hope that Meson gains widespread adoption. Calling Autotools the Xorg of build systems is flattery. It really is just a terrible system. We really need to invest in something that works for us rather than against us.
PS: If you just want a quick look at what the build system syntax looks like, take a look at this or the basic tutorial.
This is great news! I've not had huge successes with Meson yet but I think the build description language is dead on and we'd all be much better off if everyone described their builds using that, instead of Autoconf and Automake. I think its still early days for the project but I have high hopes, and having GStreamer fully support it (both in the technical and social sense) is a big step!
ReplyDeleteGreat article! But would you say that Meson is ready to use yet, i.e. if you started a new project, would you use it then?
ReplyDeleteI still don't understand why not using CMake. As far as I see it, its the de facto standard right now for building cross platform applications with C/C++.
ReplyDeleteAs an incomplete list, CMake has hacky cross-compilation support (no exe wrappers for tests), an ugly and hard-to-use syntax (multiple people have complained about this), difficult to use and does not come with tools that make it easy for GNOME, GStreamer, and related projects to use it.
ReplyDeleteI'm sure others can chime in with more reasons. Personally, I found the project rather lacking compared to even Autotools unless you're doing something very specific that it's very good at.
CMake works great for many people, but it doesn't work for everyone. Meson attempts to be a replacement that would work for everyone.
CMake...
ReplyDelete... is a Turing-complete programming language, but was not originally meant to be one, and as a result is a very clunky one. Look at how you return values from a function, for example, or how you do arithmetic!
... has some really confusing behaviours about dependencies between files in different directories... see: https://samthursfield.wordpress.com/2015/11/21/cmake-dependencies-between-targets-and-files-and-custom-commands/
... gives you multiple ways to do certain things. See target_link_libraries(), for example: do I use an imported target (yes, but probably there isn't one available for the library you want), or pass in the FOO_LIBRARIES flag returned from the FindXxx module (hope it's in a standard directory, or that might break: https://cmake.org/Bug/view.php?id=15804), or
... has an enourmous number of properties, making it hard to find how to do what you want just using the documentation (and this is the only documentation). See the list of all custom properties, for example:
... treats pkg-config as a second-class citizen, and ships a bunch of hand-maintained and usually obsolete custom FindXxx.cmake modules for various libraries
... trades whitespace-safety for semicolon-safety: ok, it's not common to put semicolons in filenames, but if you do you'll have a nightmake in CMake
... seems to be quite short-staffed: there are some paid maintainers who do a good job, but I've had very little luck actually getting feedback on patches and getting them merged
... is pretty unfriendly to write, because small mistakes tend to be silently ignored rather than detected, and things break somewhere much further down the line (the --warn-uninitialized and --warn-unused-vars flags help a lot, but you can also get a lot of noise from the modules you pull in)
This isn't a complete list at all. That said, I think CMake is well engineered and, in some cases, very well thought through. I certainly don't find it any worse than GNU Autotools (although that's hardly a complement). But I also think Meson steals many of the good ideas from CMake, while avoiding all of the bad things I mention above.
I missed out a link from that comment to the list of all CMake's custom properties: https://cmake.org/cmake/help/v3.5/manual/cmake-properties.7.html
ReplyDeleteHave a look through that & see if you can find which property I'd need to use to specify what libraries an imported target depends on, for example ...
The meson port of glib doesn't build for me on arch linux. There is no libintl.so on arch, and /usr/include/libintl.h is owned by glibc
ReplyDelete@Ben, that was my bad. I forgot to test that properly after making changes for Windows. Should be fixed now. Please try again!
ReplyDeleteCan you write up some description how to build gstreamer with meson?
ReplyDelete1353 lines to compile glib is still a lot. Compared to 3473 lines it sure is an improvement. But surely, a lot of the stuff in the meson.build for GLib could be reduced by having whatever is going on there in reusable modules?
ReplyDeleteIs there a Meson module type for JHbuild yet? I suspect that would go a long way towards getting people used to it and easing adoption.
ReplyDeletePhilip, you can't compare line numbers like that, you'd have to count the meson.build in the various sub-directories as well, and on the autotools side also add the various .m4 files plus all the Makefile.am and foo.mak that get included by Makefile.am. If you do that count, you'll end up with ca. 8-20k lines vs. 3k lines.
ReplyDeleteBut line count isn't everything, clarity is just as important. The meson build description language syntax is much nicer, and the fact that it combines the configure.ac and Makefile.am part in one makes for much clearer build files too. Also, in meson you can move checks into the subdirectories where they are needed if you want to, which is nice as well.
There are certainly some things in the meson.build which could be written shorter/cleaner now using features from meson git.
There might be some things to split out into modules, yes. But glib is a non-trivial thing to build. I've found more things that I felt should be consolidated in a module elsewhere (things like intrinsics/neon/sse checks and such).
In addition to what Tim said, I'd like to point out that while porting glib's Autotools files to Meson, we tried to do a 1-1 translation of the checks. However a large number of the checks seem to be cruft from old platform-specific code that is either gone, is obsolete, or has been untested/unused for a long time.
ReplyDeleteThis is a natural consequence of glib being "drive-by-maintained" for many platforms, and it's not clear at all which checks (and platform-specific code) are still useful or even work. So we can probably reduce that quite a bit.
As a result of this, I think comparing gstreamer's m4 macros + configure.ac + Makefile.am with gstreamer's meson.build files is more useful since the above is less of a problem there.
Just to clarify: that's 3k lines (meson) vs. 8-20k lines (autotools).
ReplyDelete