Hey everyone,

I've been reading the different threads that talk about streaming, and I don't think any of them are actually relevant to my problem, since I do not care about storing parts of the volume or loading parts of it.

My setup:
* a server (minecraft beta server) that sends me 16x128x16 chunks of voxel data
* a Volume<Material8>(1024,128,1024,64)
* a usually occupied area of 160x128x160 centered on 512,64,512
* fast rendering on block changes (Regions that contain exactly one block, renders almost as fast as the server can send the blocks (almost b/c I need to render the 8 chunks around the changed chunk, too, otherwise "inside" the mesh some weird areas/lines appear, but that is basically irrelevant, I know why they appear)

If I move towards the border of my occupied area, the server will send me new chunks on that side and send me removal orders for chunks that I move away from.

now to my question:
would it suffice to edit Volume::m_pBlocks to shift the entire occupied area one block in some direction? or are there any things that I'd mess up by doing so? (m_vecBlockIsPotentiallyHomogenous would be invalid of course, but it doesn't matter or I could modify that one, too)

/ker

I think in principle it might be possible to adjust the block pointers to do the kind of shifting you describe... but maybe it gets messy. I guess you'll also have to adjust the offset which are part of the SurfaceMesh... I'm not sure.

Overall I would recommend you switch to the new Volume class (the old one will be removed when the new one comes in):

• It support much larger volumes. For Minecraft style terrain an 8192x8192x256 volume takes about 50mb of memory.
• When you receive your removal commands from the server you can't actually unload from memory, but by setting the block to be empty you will get very good compression.
• If you do need larger volumes it's a step in the right direction. It will be easier to implement paging inside the new volume than the old one. Paging here doesn't have to mean paging to/from disk, if done right it could be any source including a server.
• You will continue to benefit from PolyVox updates, which might not work with your system otherwise.

I realise your not actually asking for infinite volumes, but I still think it would be easier and more beneficial to impement that rather than the shifting system you describe.

ok, yes I believe it would get very messy indeed.
I'm already using the new volume.

wow, I'll have to test out those sizes asap (at one point we'll reach the unsigned short boundary thou^^)

so what are my possibilities?
* completely ditching the nicely ordered block array and using a map to index into the block array and implement some caching techniques
* telling volumes about each other and modifying getVoxel to instead of getting border values, getting voxel values from neighboring volumes
* 4d-torus

anything else?

Yeah, we can change to unsigned long when necessary - I just didn't foresee that people would want volumes that big

This. As the volume gets large there's currently going to be a huge nuber of blocks in the array, even though each block is small it will end up taking a lot of space. You should look at replacing the vector of blocks with an std::map or probably an std::hash_map. Then you can remove blocks completely when they are really old.

Then you need some kind of callback mechanism so that if data is missing from memory then it can be requested from the application, and if data is removed from memory because it is not being used the applications should be notified in case it needs to be saved somewhere. This callback stuff may not be strictly necessary for your own work (as it's being driven by the server) so it need not be a priority, but it will be necessary if you want to merge it back into PolyVox as then it needs to work in a range of applications.

Not very keen on this, it will be a lot of work and probably messy

You mean a toroidal wrap? It's probably more complex than the first approach but you can experiment if you like.

ok, I have implemented this... but did something stupid while I was at it... using signed coordinates instead of unsigned.
this works perfectly until you try to feed that to a surface extractor...
(it creates really bad surfaces... as in it kills my ogre renderer... I'm assuming some unsigned values getting casted to signed, because I've found threads about someone having the same crashes that I do b/c of unsigned->signed casts)
now... Regions support negative values... so they're not the problem...
and VolumeSampler I changed to best of my knowledge to not care about the lower border needing to be posititve)
do you have hints on how to modify the different surface extractors to support this? or should I stay with unsigned positions?

Excellent

I think in principle it should be possible to modify the surface extractors to work on signed positions, and I think this should probably be done because it doesn't really make sense for an infinite volume to stop at zero.

However, I suspect it will be much easier for you to change the volume to use unsigned ints for now (as I guess you are now familier with that code) and we'll handle the signed issue later. If you do want to face it now I'd suggest starting with the cubic surface extractors as the Marching Cubes one is complex and messy.

Are you intending to contribute your changes back to PolyVox (you don't have to of course, but I'd be interested in them)? I'm probably going to merge the large volume stuff into the truck this weekend, but you can carry on working against the branch and we can merge across again later.

So wait, did you make it that a hash map of coordinates each has a block of voxel data which can be loaded and unloadted from stream? does it update just one volume for it all? *confused*

eeeeh, yes I'd be happy to supply my streamed volume, but FIRST I gotta clean up that code, I am just starting to use somewhat proper variable naming (again)... + I forgot many probably useful comments


the following three lines are my changes to the interface of Volume. (and lots of stuff removed that was just useful for boundaries, sorry no getEnclosingRegion anymore)


now... m_pBlocks is empty at creation of the Volume.
there's seriously nothing in there, not even empty blocks.

you are allowed to register your own loadBlock and unloadBlock functions (using boost::bind, if this is an issue due to dependencies on boost, I can modify this to some messy pointer arithmetic)
also unordered_map is only in c++0x

anyway, loadBlock is called, when you try to access (set/getVoxel) a nonexisting/not_yet_loaded block.
it gives you a block reference which you can fill with your own data if you wish and a Vector3DInt64 which is the position of the Block in Block-coordinates (shift this by your Block-sidelength to get Volume coordinates)
if you don't set a loadBlock function, the default is used, which is filling the block with VoxelType().
unloadBlock works almost the same... you get block coordinates so you know where this block belongs and you get the Block to somehow extract the data from and save however you want. doing nothing in the unloadBlock function causes the data to be lost.

unloadBlock is called whenever the size of the unordered_map reaches some threshhold. least recently used then decides which block to drop.

and behind all that there's still all the compression working without any problems
(un)loadBlock always gets an uncompressed block, so no worries there...

that's cool. So basically with this small modification (storing the blocks into unordered_maps) one can stream "infinite" terrain directly into the Volume class? That's cool. Dang it. I have a system of multiple volumes... Well I didn't want to touch the Volume class at the time.

Anyway, I think one can still generate the data in threads with this system. Perhaps just have to have a call back function to inject the data into the block after the data becomes available. Then maybe have a SIMPLE listeners class to inform the neighbors that the data is now available, since I see there may be some cases where a neighbor block tries to get a voxel next to it, triggering an loadBlock, and once the data becomes available the neighbor can access it.

I don't know yet but I'm definitely interested in this if it gets merged back into PolyVox.

Very nice. Wondering when it will be merged in.

You'll probably need to write a small tutorial on how to use this properly