I'm working with voxels that store several different informations like "pressure", "light levels", "stress load", etc. I was wondering which one was the best approach:
-different polyvox volumes for different kind of informations
-"bigger" voxels storing all the data they need (56 bits more or less, perhaps even 48 but I'm not confident about being able to make everything fit in such a little space).

Fast access to the voxels is extremely important for me, so I expect a major boost in performance if I call "getVoxelAt()" as little as I can. Perhaps the second approach is better for this reason.

Those voxels will eat up some big space in memory, so compression is important for me, too. With RLE I would probably get bad compression, because the values varies smoothly but constantly around the volume.

I found some compression algorithms that are capable of speeds above 200/250 MB/s both for compression and decompression on a modern consumer hardware. For blocks around 32kB in size (1 byte x 32^3), that would mean about 0.2 milliseconds for decompression or compression. They do a decent job at compressing, surely a better one than plain RLE.
There's one from Google under a BSD-like license that is really interesting, called snappy: http://code.google.com/p/snappy/
Would you be interested in something like this, or you don't really care about compressing "smooth" data like density?

I've been keeping my eye on the available compressors for a while now - mostly because I'm interested in using them to compress the data when it is stored on disk. But as you say, it is also possible to use them to store data compressed in memory - I agree the RLE is unlikely to be optimal is many circumstances (particularly with smoothly changing data).

I saw Snappy mentioened in a recent blog post here: http://altdevblogaday.org/2011/04/22/survey-of-fast-compression-algorithms-part-1/. Another one which looked interesting was http://code.google.com/p/lzham/

Now, I'm relucant to add and dependancies to PolyVoxCore, but it is possible that code in PolyVoxUtil could make use of these libraries. Either a new volume type could be created by copying the existing Volume and VolumeSampler and renaming them, or alternatively The Volume class could be modified so that custom compressors can be provided by the user. Then the Volume could exist in PolyVoxCore with a default RLE compressor, and PolyVoxUtil could hold an improved compressor based on an external library.

Thinking about it, this code should probably also be used to compress voxels on disk. Maybe a PolyVox::Compressor class is needed, where subclasses can be based on RLE or external libraries. Then the user can provide their compressor of choice when creating a volume or serialising to disk.

Ok, there's some interesting ideas here. But actually myself and Matt are currently focused on trying to stabalise things and get an SDK into shape. So we won't be working on (or integrating into the trunk) anything like this in the near future. But you can always play round in your own local copy - we'd be interested to know how it works out.

Edit: So it doesn't get forgotten, I added 'Investigate integration of external compression libraries' to the todo list

I don't see problems with dependencies, to be honest. We can include the code (it's usually just 2 or 3 files) directly into polyvox, can't we? Ofc the library should be compatible license-wise (so no GPL -.-')
After all, compression IS part of polyvox right now. The fact that you are not the one that wrote 2 or 3 functions doesn't mean you have to put it as an external library or dependency.
That idea about implementing the compressor class is good but looks like it's a lots of work. But I don't know, for things like this, I see them as being those core components you are never gonna change (as a normal user)... Like when you are given the opportunity of using your own memory allocator with STL containers... Most of users don't use that I think.
A faster way is probably just replacing RLE with one of the algorythms proposed in the article you posted. It _does_ make sense to me. Again, it's 2 functions, it's not something I see as an "external library" or dependency.

Ok, I can probably be convinced here. I still like the idea of allowing the user to provide their own compression algorithm (probably still defaulting to our own), but as a short term solution we could hard-code it to one particular implementation.

I suggest you give this a try and see how beneficial it is. You should be able to implement this by only modifying the Block class, though you'll also need to update the CMake scripts and make sure it works on both Windows and Linux.

If you can create a personal repository on Gitorious then other people can access and test it. We can also then pull it into the master repository if it works out.

Be aware though that I'll be moving and renaming some classes over the next few days, so you might want to wait until the weekend to avoid conflicts.

A couple more points:

• It would be desirable if the library was header only. People have said it would be nice if PolyVox was header only, and while I have no plans for this it's nice not to rule it out.
• It would be good if the library allowed control over memory allocation. PolyVox doesn't do this yet, but eventually it will need to allow the user to provide custom allocators and these should also be honoured by the compression library.