Ok, I've committed some interface-breaking changes. The public functions 'm_LoadCallback' and 'm_UnloadCallback' have been renamed to 'dataRequiredHandler' and 'dataOverflowHandler' respectively. I've also added a ConstVolumeProxy to work around the const awkwardness in the setVoxelAt function described previously. The signiture of these user provided functions is now:



Where polyvox_function is a typedef for std::function or boost::function. The example has been updated to demonstrate.

@ker - It's taking a bit longer that I thought to make these changes so it'll probably be a few more days before I hand it back to you. Is it all still working for you so far?

sorry I was on a LAN party
I'll update tomorrow and test it out, but except for changing the calls to the loader/unloader function I don't see what should break (especially since I'm not using much of polyvox except for volume and SurfaceExctractor)

about the float issues at large distances, the extracted vertices are in relative coordinates to the region you're extracting, therefore you won't ever run into trouble there if you start some "shifting" after maybe... 50k blocks?

No problem, I just don't want to be holding you up. But I don't think I've broken anything yet. Next I'll add something to optionally disable the streaming, then maybe some optimisations. Then we can think about merging it back to the trunk, unless you want to make more changes yourself.

As for the float issues, I don't expect PolyVox will have any issues because the offset of each region can be stored with integers, and the floating point vetex positions are then all small values relative to this offset. But for rendering they will be converted to floats, so the users code will probably have to deal with some precision issues. There are solutions of course (perhaps along the lines of what you describe) though I'm not particularly familiar with this area. But that problem lies outside of PolyVox anyway.

I think we have 2 different opinions on what a Region is ^^
I always thought it is minimum position until maximum position INCLUDING the maximum position (that is how the callback gets it).
but for the surface extractors you are using minimum position until maximum position minus 1.
this should be consistent, and I believe my way to be the one to get rid off, because I only messed up (and noticed therefore) in the load-callback (and I believe I should be able to use my own code)

also for linux compilation some more includes are needed (see attachement)

no I have no further changes related to streaming.

I am using those two functions that make life easier when switching between region and voxel coordinates, maybe someone finds them useful:

Actually you will find there is a difference between the way the marching cubes SurfaceExtractor works compared to the CubicSurfaceExtractor. This is because the CubicSurfaceExtractor can generate polygons by looking at only a single voxel (i.e. should it generate a cube or not) where as the marching cubes SurfaceExtractor can only generate polygons by looking at groups of 8 neighbouring voxels.

So if you generate a mesh for the range 1-5, and then for the range 6-10, with the CubicSurfacExtractor it will look continous but with the marching cubes SurfaceExtractor there will be a gap. This is just a difference in the way the algorithms work, and it should probably be documented rather than trying to hide it from the user.

Anyway, I think that for your load/save functions it makes sense if the Region is indeed inclusive of the maximum position.

[Edit:] Oh, and I applied your patch.

Following on from the confusion over the Regions, I have actually deprecated the width(), height(), and depth() functions of the Region. If users want these values they now need to calculate them themself from the upper and lower bounds.

Ok, now I have a question. So far we have been refering to ker's work as streaming, but I'm wondering whether that is the most appropriate term for it.

What ker has added is basically support for very large volumes, so large that it may be difficult to fit all the data in memory at once. When the volume runs out of memory it is now able to call a user-provide function to say 'hey I'm out of memory and so I'm dumping this piece of data which hasn't been used for a while'. It's then the users responsibility to do something with that data if they don't want it to be lost.

On the other hand, the volume might try to use a piece of data and find it is missing. This could be because it was dumped out of memory earlier, or because it hasn't been loaded yet. Either way, the volume can now call another user-provided function to ask for this data to be provided.

So, is streaming really the right term for this behaviour? Or is it just 'Large Volume Supprot'? Or 'On Demand Volumes'. Or 'Application based data handling'? I'm open to suggestions here

I dunno but there is a slight problem here!

For example, in my engine I'd like the data being dumped to be saved to file, like many others. But I'd probably like it to be done in a different thread, to do that the callback function needs to call to a thread to do the work... but by the time that happens I have a feeling my data would be gone already? unless there would be a shared_ptr involved?

you'll have several troubles with using threads. the major one being that even multiple simultaneous read access to the volume will cause horrible horrible behavior.
so you have to lock the entire volume when accessing any part of this...

the currently only solution I can think of:
discard the load function, only have a custom unload function.
2 threads, we'll call them "main" and "other".
in "main" you access only voxels that are already loaded (you'll have to implement this yourself)
"main" sends an ansychronous request to "other" that then fills the requested area with data.
in the case where something needs to be unloaded, this will be happening in "other".
you will need to lock any access to the entire volume both when accessing from "other" and from "main"

well... streaming really is the wrong name, I believe "paging" to be the one you're looking for.

[edit]
if multithreading should be supported inside polyvox, this will need a complete overhaul of RLE and paging/streaming. probably with a custom implementation of map...
[/edit]

In general PolyVox is not thread safe, so you need to be careful here. You should performa all volume accesses from the same thread. I would imagine the best approach was simply to copy the data when PolyVox tries to dump it, then in a seperate thread you can work on serialising your copy of the data.