<brainstorm>

I've been watching through some of the videos from the Going Native 2010 conference hosted at Microsoft this year and it's been getting me excited about some of the new features offered by C++11. In light of this, I though I'd review what C++11 features we already use and which ones we might like to use in the future. There are three reasons that we might want to use new features:

1. Simpler/cleaner internal code to reduce bugs and maintenance easier
2. Simpler/cleaner API to make users of the library have an easier time
3. Add awesome new features to the library which wouldn't be possible otherwise -- less convincing than the other two

Current usage

The only C++11 features we use at the moment are standard library features, i.e. not language features. These are generally quite well supported in compilers and where support is lacking (i.e. VS2005 (VC++ 8.0), VS2008 (VC++ 9.0)) we currently fill in with Boost replacements. Currently we use:

• std::shared_ptr from <memory>
• std::function and std::bind from <functional>
• std::hash
• int8_t etc. <cstdint>

All these are defined in include/PolyVoxImpl/TypeDef.h. They are maintained as #ifdefs to things like polyvox_shared_ptr etc. We also use <type_traits> but this has been implemented in all compilers for years now (GCC 4.3, VC++ 8.0).

Potential language features for PolyVox

There are a few language features which it would be nice to be able to use for efficiency, error handling or syntax niceties. The following mostly excludes any features which are not going to be present in VS any time soon (even if they are already in GCC):

constexpr - GCC only but is optional improvement
There are a few places where we use semi-global constants such as the marching cubes table and randomUnitVectors for the ambient occlusion. By switching these to constexpr we could perhaps see some speedup due to some calculations being performed at compile-time. It will require some benchmarking and investigation into where improvements can be made.
Optional interface in TypeDef.h would be like:where constexpr would be set by CMake using a try_compile().

static_assert - GCC is fine and in VC++ since 10.0 (VS2010)
static_assert is a version of assert() which is evaluated at compile-time. It only takes compile-time constants (e.g. constexpr and literals) but this does include type_traits and sizeof(). This would allow us to implement some improved compile-time error handling for templates. Replacement functionality is included in Boost.StaticAssert but we would have to investigate how complete it is.
For example, including:into the Vector(x,y,z,w) constructor means that trying to do:will fail with:
Vector.inl:65:9: error: static assertion failed: "3 element vector cannot be created with 4 arguments"
Maybe not the best example but is an early start. It gives no run-time overhead at all.

trailing return type - GCC is fine, VC++ since 10.0
decltype - GCC fine, VC++ since 10.0 (improved in 11.0)
Haven't had time to do a review of PolyVox for places where this would be useful but could improve the API, especially in template-heavy functions where the return-type is indirectly dependent on the argument types.

lambdas - GCC fine, VC++ since 10.0 (improved in 11.0)
There are a few places we allow the user to specify callbacks (e.g. A* pathfinder). If these callbacks could be provided using C++11 lambdas then this would improve the usability of the API. We would have to ensure that lambdas are accepted through the API even though we wouldn't necessarily be using them ourselves within PolyVox.compiles but I haven't tested for AStarPathfinder although it should work.

C++11/STL API consistency - No compiler dependency
This is something that Ker brought up when discussing 'range-based for loops'. It would be nice if the PolyVox API was similar to the STL where it makes sense. For example, we could support the STL iteration API on our volume sampler (and elsewhere if it makes sense).
Also, In C++11 (was also in TR1 and so is already available in many compilers) there's a new fixed-size templated array (since VC++ 2012 and GCC 4.3 but tr1::array since VC++ 2008) called std::array which you create asThis clashes with, for example out PolyVox::Vector which is used likeIt's a small inconsistency but it makes sense to align on these things. Also, PolyVox::Array is a very different beast to std::array where the former is a multi-dimensional array but the latter is more like std::vector or PolyVox::Vector.

enum classes - GCC is fine, VC++ from 11.0
They could be used for material type instead of integers to improve type-safety (discussed here). There's likely other places we could use them too. With some macro/#IFDEF magic we might be able to only enable this usage on supporting compilers.

Language features for users of PolyVox

There are other things we can allow the use of in user code such as initializer lists and auto (in user code):

auto - GCC is fine and in VC++ since 10.0 (v0.9 only but that should be fine)
Of course any PolyVox user can use these if they have a supported compiler without us having to change a single things. There are perhaps places where we could make use of them inside PolyVox but it would require compiler support.

initializer lists - GCC only
We should make sure that all appropriate classes (e.g. PolyVox::Vectors) accept initializer lists as arguments. An API review will be needed to explore where they make the most sense - even if Bjarne's recommendation is to use them almost everywhere (except e.g. "int i {42}").
They can replace the need for the ArraySizes class.Since on the whole, they will just work, this will mostly be a documentation thing to make people aware of where they will/won't work.

Compiler support

Obviously if we want to use many of these these features it would mean upgrading the minimum compiler requirement in Windows to VS2010 which as I understand is difficult for some people. It would perhaps be useful to know how many people use compilers earlier than VS2010 and whether that is due to real restrictions or just due to not upgrading so please fill out the poll attached to this post.

If we cannot increase our requirement then we will simply have to be conservative about what C++11 features we can use and fill in gaps with typedefs and conditional compilation Even if we did decide to increase the minimum requirement then it would probably not be until after our next official release (which I think Dave was planning for a few months-time).

Appendix

To the best of my knowledge, these are the currently supported compilers:
* GCC 4.4+
* VS2005 (VC++ 8.0) with Boost, VS2010 otherwise - correct me if I'm wrong
* Clang support is in progress (by me)
* Intel C++ possibly worth a look?

and finally, here are a couple of links which lay out the current state of compiler support for various C++11 language features:

• C++11 support for all compilers
• GCC's own table
• VS2010 report
• VS2011 report

_________________
Matt Williams
Linux/CMake guy

My personal opinion (the below is copied from an email to Matt) is that it would be nice to make use of some new features but not at the expense of cutting out users. Although there are C++11 compilers available for free on both Windows and Linux, not everyone is in a position to use them. Problems I can imagine are:

• Large companies lag behind the state of the art. At my previous games company we only started using VS2008 a few months ago, and at the moment I'm working with a medical imaging company who are still using VS2005. There's no need for them to upgrade, it's expensive, and it carries some risk.
  
• Universites have the same problem - here at Groningen the standard machines have VS2008.
  
• People may be using other libraries which are dependant on an earlier version of compilers. For example, the Qt SDK for VS2010 has only recently been released (with 4.8.0) and that's two years after the compiler came out. I think is is also why the games company was slow to update - we had to wait for XBox and PS3 SDKs to become available.
  
• I'm not sure what the situation is on mobile platforms, and whether the Android/iPhone SDKs support C++11. I also don't know whether anyone is using PolyVox with them but it's a potentially massive market.

For these reasons, I would be reluctant to require a C++11 compiler. However, it doesn't need to be an all-or-nothing situation. In particular we should consider:

• Use of #ifdef to mark certain parts of the code as C++11 only. For example, we could have a ThreadSafeVolume which builds on C++11 threading features but which is only compiled if the appropriate #define is active. If this define is not active then the ThreadSafeVolume is never seen by the compiler and users have to use a differnt volume class instead. This principle can also be applied to the algorithms - a threaded surface extractor for example.
  
• There is a diffrence between language features and library features. Which ones would we take advantage of? The library features can often be emulated via Boost of course, as we are doing already.
  
• Would the use of C++11 benefit us or the users or both? For example, we can make the library more C++11 friendly with iterator compatibility, as discussed before. This brings benefits to users who have C++11 without cutting anyone out. On the other hand, if we wanted to use the auto keyword to make PolyVox easier to write then this would only benefit us (and I wouldn't be so keen on this).

So, I do think we can consider the careful use of some features. I'm not all that familier with C++11, but below are my opinions on the features Matt has identified:

constexpr
I'm not sure if it will actally be beneficial. We do have a number of constants in PolyVox (such as the Marching Cubes tables) but are these really expressions?

static_assert
Yep, I'd like to make some use of this. Working with templatised code is complex and error prone so being able to catch common mistakes at compile time would be useful.

trailing return type

This could be interesting, for example the material type in the vertices returned by the surface extractor should have the same type as the material stored in the volume. Curretnly however the material in the vertices is just returned as a float. I've been meaning to look at this but I don't think we actually need 'trailing return type' to do it as it is a direct dependany rather than an indirect one. It also seems this feature has no easy workaround for older systems.

lambdas
So lamdas can be passed to something expecting an std::function? Only yesterday I added a new test for the Raycast class which I will use to experiment with the performance of different callback approaches. Callbacks are becoming more important now that we are removing the getMaterial()/getDensity() functions as discussed here.

C++11/STL API consistency
Yep, as discussed it would be nice to be STL compatible where it makes sense.

enum classes
As you indicated this was discussed previously. It does seem to make sense to allow user code to use these, and I believe the current voxel type refactoring will allow this. I don't think we need to use enum classes internally though, so there should be no new dependancy.

auto
This is handy but not worth cutting out old compilers for. People can and should use it in their own code though. Not sure about the examples - we should probably keep them working on older compilers. Perhaps a single C++11 example could be useful to demonstrate this feature.

initializer lists
Again it seems like this is just a case of making sure PolyVox is compatible with these constructs, so there is no new dependancy?

Well 'expression' can be slightly misleading in this context. One of the usual reasons given for constexpr is so that you can create c-style arrays with what looks like a dynamic variable but is actually knowable at compile-timemaking fibonacci() a compile-time expression rather than a dynamic function call but it's much more than that. Firstly, variables you pass to a static_cast must be constexpr so it gives us more flexibility there. What's coolest though is that you can define class constructors to be constexpr which, when called, create a literal on a par with "foo" or 6 or 4.7. For example, we could define the PolyVox::Vector constructor to be constexpr so that Vector<3,int>(4,5,6) will be fully defined at compile-time and known to the compiler. Furthermore, we can define functions+methods to be constexpr which, when passed literals (including e.g. Vectors) can calculate a value at compile-time. For example isPowerOf2() or Vector::length().

Yes, I agree this one would be nicest end easiest to implement. It should be usable with Boost.StaticAssert and otherwise can easily be #ifdefed out without any problems. I'll add the TypeDef.h/CMake trickery needed soon so we can experiment. It's making me look forward to 'static if' (which is likely to be in C++1x) already!

Yes, if it's a direct dependency on the class template type then it's possible that you can get away with decltype() (mostly available since VS2010). Perhaps we can selectively provide either function declaration depending on compiler (unless that just makes thing difficult)?

I believe it is the case that lambdas are accepted as std::function arguments simply because that can be casted to them. This unfortunately makes them a little slower so we should investigate the correct API for providing lambdas to functions - I believe that the usual way is simply using a templated function parameter (which would probably accept std::functions too).

I agree. We shouldn't need to use them internally but we should make sure that they work well for users' code.

I agree. When we're established some of this a bit more I'll probably write a page for the manual describing how to interface with PolyVox in an idiomatic C++11 way.

Indeed, mostly they should just work and are only dependent on the user's usage. However, there might be a few places where we want to use the specifically like Qt has with its new QList constructors which are only available with C++11 compilers. I can't immediately think on any uses in PolyVox though so it shouldn't be a problem.

_________________
Matt Williams
Linux/CMake guy

Ok, that does sound quite interesting, and from what you said previously it is easy to work around on older compilers?



While this might be interesting, I need to actually make the changes to PolyVox to determine whether a C++11 feature is really needed. It's hard to visualize the code in my head at the moment but I will keep the idea in mind.



By 'a templated function parameter' do you mean something like this? If so, that's one of the callback options I am considering in my profiling of the raycasting code.

It seems like we have general agreement on the other features (staic_assert, auto, enum classes, initializer lists).

Yes, since constexpr is sometimes a 'strong const' and other times it's own thing we need a more complex typedef. We should define a 'polyvox_constexpr_const' (which is a const in old compilers and constexpr in new ones) and 'polyvox_constexpr' (which is also a constexpr in new compilers but will be nothing in old compilers). I can add these definitions and compiler test soon and we can start experimenting.

Yeah, like in the first answer on that page I mean something likeI'm not sure the second function call will work but I think it should. This means you can pass anything you want, std::function, lambda or functor. Maybe I should create some C++11-only unit tests.

In other news I added the typedefs for the static_assert last night. You should now be able to use both ofAnd they will use Boost.StaticAssert in older MSVC compilers. This may take some testing.

Also, I managed to get PolyVox building on Clang 3.0 last night (almost no changes but check it still compiles in Windows) so that should be another way to keep our code in check.

_________________
Matt Williams
Linux/CMake guy

This seems like a good opportunity to bring up something I've been thinking about for a while... does this stuff have applications beyond PolyVox? Is there an open source library which implements it already? Perhaps we should include all these #defines in a seperate header and make it available outside of PolyVox?

We're talking here about compiler features which are only available in C++11 vs C++98, but there are also a number of compiler extensions/features/keywords which differ between VS, GCC, Clang, Intel, etc. Example would include 'force inline' and 'branch prediction'. Could we unify these in a similar way?

I'm sure a lot of projects have the kind of defines we are talking about here but are they made available anywhere?



I'll see what I can find from the analysis of the raycasting code. I will be a few days though.



Great, I'll add some of these to the code.



I'm a big fan of Clang as I looked at it a couple of months ago as a potential basis for the work we are doing here in Groningen. Unfortunatly we're not going with it but I was very impressed by what I saw.

Yes, I guess there's a lot of 'papering over' which could be done here. In KDE we have a file kdemacros.h.cmake (which is converted to kdemacros.h by CMake) which defines some of these sorts of things. I think it's important to differentiate between things like compiler extensions and things for C++11 compatibility, especially when the latter requires Boost for full compatibility. Some sort of categorisation into different headers I think would help here. I think it could be useful to others to provide this as a separate download as another 'Volumes of Fun project'.


Yeah, there was a talk on Clang at the Going Native conference which reminded me to look into it. There's actually a bug in Clang 3.0 which causes it to fail to compile The OpenGLExample. Basically Clang is missing the copy constructor for std::shared_ptr so it falls over but apart from that one file, it compiles fine.

_________________
Matt Williams
Linux/CMake guy

Yep, agreed. It can be part of PolyVox for now as PolyVox will provide a useful testing ground. But in the future we can consider splitting off into a seperate project.

I had a look at this while I was in Italy. It's good in principle, but unfortunatly I couldn't think of any places to add the static_asserts.

This is partly because of the refactoring which is taking place. Under the old system we could, for example, say that a certain surface extractor would only work with certain voxel types (e.g. it might require the voxel type to have a material). But because the new system is more flexible and allows primitive types to be used a lot of these restrictions no longer apply.

I'm sure we'll encounter some more places where it can be used, and I'll just have to keep it in my mind as the code gets refactored.

I've been thinking one this whole topic again recently (inspired by watching a very interesting video by Scott Meyers on 'Universal References') and I have two things to report.

First, I've found a very interesting CMake module for checking for the availability of various C++11 features. You can read about the module a little at this blog post and the code is now largely available on KDE's git server. As far as I can tell, Eike is planning on submitting this code to CMake core so it will be integrated for a future release. If we want to use the feature detection code we can either wait until it's in the official package (could be a while or never) or include the code in our repo ourselves. We don't have C++11 features planned for the upcoming release but I'm planning on investigating some for the release after.

Secondly, I started looking into adding move constructors to places in the code where it makes sense (and where they wont be added automatically). I've got a local branch set up with move constructors working for Region and RawVolume but we would first need to investigate where we actually want them and where they make sense.

_________________
Matt Williams
Linux/CMake guy