Ah, yes, I like the term paging. I was just having a mental block I'll probably go with paging if no one has any other ideas.

Yes, but how much of a performance hit would one get if I try to copy a third dimension array of volume data in the main thread? I don't want a player to suddenly halt cause I am saving/loading, or even copying data for save/load

This is actually my main reason for allowing the paging and compression to be disabled (working on that at the moment). Personally my volumes are very small, so I don't need these features and by turning them off I can make things more thread safe. Not properly thred safe I should point out, but enough that I can probably work around threading issues. Maybe

At the moment it will unload single blocks at a time, so it will depend how big your blocks are. The default 32x32x32 block will take 32kb of memory, so that's what you will have to copy.

That said, I'm thinking the current default size is too small so I might increase it in the future.

Also, it doesn't just drop the data at some arbitraty time - it will only if you are writing new data into the volume that it has to make space for.

This system is also not widely tested, ker is the only person using it. So we might have to make adjustments in the future, but only once we really establish there is a problem.

are you trying to find a way to disable/enable RLE/paging by making sure it won't even be compiled in depending on a template setting? or by allowing changes during runtime?

Just at runtime. There is now one constructor for a fixed sized volume, this has paging disabled by default. Another constructor takes no size but instead takes std::functions for the load/unload - this one has paging enabled by default. I'll commit this today and you can tell me if it makes sense

Ok, I've committed the changes to make the streaming optional. You will have to change your code to make use of the new constructors. I also changed some other function names:

setBlockCacheSize() becomes setMaxNumberOfUncompressedBlocks()
setMaxBlocksLoaded() becomes setMaxNumberOfBlocksInMemory()

I also reduced the default number of blocks which are uncompressed and which are loaded. This has a very negative impact on performance so you might want to call the above functions to set them to something appropriate to your setup.

We might need to addres these performance issues in the future - let's see how it plays out.

I'm setting those anyway... for me RLE is not useful at all currently... so I always set it reasonably high.
using this in my code was no problem at all.
a small patch for linux is still needed, i used those "typename *::iterator" for a reason, g++ seams to have problems with template types inside of templates of templates inside template classes (if I got the wording wrong somewhere, sorry ^^ it's a somewhat odd bug)

I'll look into performance once I actually end up having performance problems.

Ah, sorry. I only took it out because I didn't fully understand it, and Visual Studio seemed fine without it. Except it did complain in the case of the eraseBlock() function for some reason so I left that one in. Anyway, I've applied the patch so it's back in.



Sure thing - once this is back in the trunk and being used by more people I'm sure we'll get feedback about what needs improving. As you already said, we can try a different internal data structure (map vs hash) and also the code I wrote for finding the oldest uncompressed block is pretty slow. But these are all implementatrion details - the important thing for now is the interface.

Anyway, I'm nearly done with my changes. Hope it doesn't seem like I'm picking your work apart too much - you did a really great job with this.

doesn't seem like that at all, the current interface is much better.
did you also create a way to disable rle?

I think we need one more thing for streaming thou. and that is a function to force a load of a region, and force an unload of a region. (obviously force-unload only unloads full blocks inside the given region, and force-load loads full blocks even if they only touch the given region with 1 voxel)
these functions might even return some value (like a region) of what was actually loaded/unloaded

I do not require the force-load at all, but the force-unload is something that can't even be triggered from the outside.