Hi all,

The LargeVolume class has the potential to be the recommended volume class in PolyVox, but I'm sure many of you are aware that it has some issues. So far we have not made much use of it ourself because the SimpleVolume has been sufficient, but as work proceeds on Cubiquity we're going to need to load larger amounts of data.

We have already made some improvements to the LargeVolume since the last release of PolyVox. In particular:

• Bug fixes: We fixed a number of small issues which could cause crashes, and added unit tests to verify all the volume classes.
  
• Voxel access: The getVoxelAt() function is replaced by getVoxel() (fast, with no bounds checking) and getVoxelWithWrapping() which lets you specify clamp or border modes for voxels which lie outside the volume. This affects all volume classes but the changes are not final yet.
  
• Compression: The compression algorithm used by the LargeVolume is now pluggable, with ZIP and RLE compressors provided for smooth and cubic-style data respectively.

However, there are a couple of areas where more work is needed (at least before we can adopt it in Cubiquity). In particular:

• Paging: The overflow and underflow handlers are difficult to use and also seem to cause problems on VS2012. It seems a more elegant solution would be to provide a base 'Pager' class with virtual overflow() and underflow() member functions. Maybe 'Pager' is the wrong name for this? PolyVox could also provide a 'FilePager' implementation which would read and write blocks to disk. Also, I think the current implementation always uncompresses blocks before paging them out and this may not be desirable.
  
• Threading: Currently the LargeVolume is not thread safe, and if people want to run multiple surface extractors then the data needs to be copied into seperate volumes. I can see this is not ideal and that perhaps a more thread safe LargeVolume is the way forward. Standard C++ threads are not widely supported on Windows (at least not in VS2010) so this would force a dependancy on Boost. I don't know how closely boost threads match std threads, but we'd abstract the differences if possible so that GCC users don't need boost.

So any thoughts on the above? Any other concerns about the LargeVolume or areas where it could be improved?

In regards with the bounds checking for Large volume and maybe the other classes as well, how about having it be conditionally compiled based on a cmake setting or debug/release build?

I second the idea of an abstract Paging class that the user can implement. This will make using LargeVolume easier and cleaner. I think 'Paging' sounds better than 'Pager', especially since the latter is associated with the antiquated wireless messaging device.

Actually is kind of how it works, in that the getVoxel() function (which does not perform bounds checking) does still have an assert() present. So Bounds checking actually is performed if asserts are enabled, but not otherwise, and it will never throw an exception regardless. You can see it here: https://bitbucket.org/volumesoffun/poly ... elop#cl-82

But you've got me thinking, maybe getVoxel() should instead take a parameter to control whether bounds checks are supressed. Untested code:



In practice 'suppressBoundsChecks' should be a template parameter so that checking it is free, but I can't write the syntax for that off the top of my head. We could then also have another overload:



Which includes a wrap mode, and hence never needs bounds checking anyway (but the wrapping has some small overhead). this would replace getVoxelWithWrapping() which is in develop at the moment.

Ok, I will play with this and see what I can come up with.



Actually I think a classname should always be a noun, but something like PagingManager would be possible. I was actually wondering whether 'page' is the right word at all... maybe 'Streamer' or 'OverflowManager' or something. Hmmm... probably not. I'll think about it but maybe Pager is best after all.

Ok, I'm making some changes in a branch called bounds-checks. For each volume class there are now two versions of the getVoxel() function:



So for example:



I said earlier the parameter to control bounds checking should be a template parameter. Actually I haven't done this, because C++ does not support default parameters for template functions (though C++11 fixes this). I don't want to force users to call getVoxel<BoundsChecks::Full>(x, y, z,) as I think that's confusing.

The old getVoxelAt(...) functions are still in place and behave as before, but I expect they will be deprecated. I need to do more testing though and to port internal code to the new system for testing.

Anyway, skipping these bounds tests does seem to help performance

@Matt - I hope these function signatures don't cause any problems for SWIG bindings. I think they are probably OK? Actually that's another reason to avoid the template parameter solution - I expect that would have complicated SWIG.

I think they should be fine for SWIG. It supports enums just fine as well as default parameters for functions. If it were to be a template parameter then I would just have to generate a wrapped function for each state (i.e. one with checking and one without). This would still be fine but the API would be a little larger.

_________________
Matt Williams
Linux/CMake guy

I realise now that both options are probably possible - i.e. a non templatised getVoxel(..., BoundsCheck eBoundsCheck = BoundsChecks::Full) version and also a templatised getVoxel<BoundsCheck>(...) version. Still, it complicates things and there's no indication that it's needed at the moment, but it could probably be added at a later date without breaking backwards compatibility.

Also, is the idea of a Pager base class going to be easier for SWIG to wrap than the std::functions? Do you think it will be possible to define a Pager implementation in Python/C#? If it's not possible to provide this then it should be fine to just wrap a version with the default FilePager being used.

Hi all,

I've just merged in some work which aims to reduce the amount of bounds checking which is performed when accessing a voxel.The getVoxelAt() function is replaced by getVoxel(), and this provides explicit control over when bounds checking is performed. Please see these docs for more information:

• Basic access to volume data

The implementation of the templatised versions of the functions proved to be a bit complex due to this C++ limitation, but I worked around that with the method suggested in that thread and the implementation is hidden from the user.

Although the framework is in place I haven't made proper use of the new functions inside PolyVox, so you won't really see a speed benefit yet. I'll update the PolyVox internals in time.

I've also started replacing the std::functions in LargeVolume with a Pager class. This work is taking place in a seperate branch here: Paging branch

I just merged the work from the paging branch back into develop. Note that this work is not finished, but the tricky part is over and the tests now pass again. It is more convenient for me to work in develop now so I can properly test the changes within Cubiquity. If you are working with the LargeVolume you should probably hold off updating for a couple of weeks until it settles down and stablizes.

• The use of dataOverflowHandler() and dataRequiredHandler() for paging is now completely removed, and is replaced by the Pager class. We provide an implementation called 'FilePager' which saves blocks to disk when they get moved out of memory, but you could also implement your own database or network pagers.
  
• The use of ConstVolumeProxy for implementing the paging is also gone. It was kind of clever but rather convoluted and potentially slow. The new paging system just gives you a chuck of compressed data to save directly without having to iterate over and retrieve each voxel individually.
  
• I'm also planning to remove setMaxNumberOfUncompressedBlocks() and setMaxNumberOfBlocksInMemory() as I think it is hard for the user to know what these should be set to. Instead there will be something like 'setMaxMemoryUsage()' and the system will work out the required values automatically.

Anyway, it's work in progress so you probably don't want to use it yet, but it should come together soon.

A quick question for Matt - do you know if we have permission to write to the working directory on the build machine? I guess we do? I'd quite like to make the LargeVolume unit tests make use of the new FilePager, which will mean dumping some data to a disk and then reading it back again. A few megabytes I guess.

Current working directory is the easiest place to do this, as C/C++ doesn't provide support for creating folders etc and I'd also like it to just work on Windows. I assume the build machine will delete the whole build folder after the tests and test in a clean folder each time?

Maybe we just try it and see...