Volumes Of Fun
http://www.volumesoffun.com/phpBB3/

Paging Volume (Previously: Streaming Volume)
http://www.volumesoffun.com/phpBB3/viewtopic.php?f=2&t=145
Page 2 of 15

Author:  David Williams [ Sat Feb 26, 2011 10:15 am ]
Post subject:  Re: Streaming Volume

This all sounds very promising, and in line with what I was planning :-)

ker wrote:
eeeeh, yes :D I'd be happy to supply my streamed volume, but FIRST I gotta clean up that code, I am just starting to use somewhat proper variable naming (again)... + I forgot many probably useful comments


Sure, there is no hurry, and it's always suprising how long it takes to tidy up code. But I think I will avoid commiting anything to the new volume branch in order to avoid conflicts with your work. That way you can send a patch and it should apply without problems.

ker wrote:
you are allowed to register your own loadBlock and unloadBlock functions (using boost::bind, if this is an issue due to dependencies on boost, I can modify this to some messy pointer arithmetic)
also unordered_map is only in c++0x


std::function is part of C++0x anyway so there shouldn't be a problem here. And is there a version of unordered_map in Boost as well?

ker wrote:
anyway, loadBlock is called, when you try to access (set/getVoxel) a nonexisting/not_yet_loaded block.
it gives you a block reference which you can fill with your own data if you wish and a Vector3DInt64 which is the position of the Block in Block-coordinates (shift this by your Block-sidelength to get Volume coordinates)
if you don't set a loadBlock function, the default is used, which is filling the block with VoxelType().
unloadBlock works almost the same... you get block coordinates so you know where this block belongs and you get the Block to somehow extract the data from and save however you want. doing nothing in the unloadBlock function causes the data to be lost.

unloadBlock is called whenever the size of the unordered_map reaches some threshhold. least recently used then decides which block to drop.

Ok, the only thing I would say here is that it is exposing the Block class to the user, when it was previously an implementation detail. Now maybe there is no way around that, but think whether it is possible to pass a Region instead of a Block. Then the user code could use the Volume's setVoxelAt() methods to fill in the data in the specified region. This allows us more flexibility in the future if we want to change the internal representation.

Also, maybe just 32 bit ints for positions? 64 seems a bit excessive, and I don't know how well they work on mobile platforms for example.

Author:  ker [ Sat Feb 26, 2011 5:59 pm ]
Post subject:  Re: Streaming Volume

1. use uint32_t
2. pass a Region and a reference to the volume instead of the block
3. check boost for unordered_map

will come back when I'm done with it ^^

Author:  ker [ Tue Mar 01, 2011 1:39 pm ]
Post subject:  Re: Streaming Volume

ok.. I got into some trouble with the 32bit ints... I'm working on it thou...
here is a 16 bit streaming world that will probably crash if you move to the upper range of a uint16

I'm writing an example currently to show off the streaming feature (like create perlin noise on the fly when requested instead of precomputing the volume)

I'm unable to upload files... posting it somewhere else
it looks a little odd... I hope this didn't mess up the diff file (the upload added blank lines everywhere)

http://paste.pocoo.org/raw/JWzOXN5vup9oVhRBHysH/

ok... this was easier than expected:

this is an example callback for volume data creation
Code:
void createPerlinTerrainChunk(Volume<MaterialDensityPair44>& volData, PolyVox::Region reg)
{
    Perlin perlin(2,2,1,234);
    for(int x = reg.getLowerCorner().getX(); x < reg.getUpperCorner().getX(); x++) {
        for(int y = reg.getLowerCorner().getY(); y < reg.getUpperCorner().getY(); y++) {
            float perlinVal = perlin.Get(x / 254.0f , y / 254.0f);
            perlinVal += 1.0f;
            perlinVal *= 0.5f;
            perlinVal *= 255.0f;

            for(int z = reg.getLowerCorner().getZ(); z < reg.getUpperCorner().getZ(); z++) {
                MaterialDensityPair44 voxel;
                if(z < perlinVal)
                {
                    voxel.setMaterial(245);
                    voxel.setDensity(MaterialDensityPair44::getMaxDensity());
                }
                else
                {
                    voxel.setMaterial(0);
                    voxel.setDensity(MaterialDensityPair44::getMinDensity());
                }

                volData.setVoxelAt(x, y, z, voxel);
            }
        }
    }
}





and this is how you use it:

Code:
int main(int argc, char *argv[])
{
    //Create and show the Qt OpenGL window
    QApplication app(argc, argv);
    OpenGLWidget openGLWidget(0);
    openGLWidget.show();

    //Create an empty volume and then place a sphere in it
    Volume<MaterialDensityPair44> volData;
    volData.setBlockCacheSize(4096);
    volData.m_fLoad = boost::bind(&createPerlinTerrainChunk, boost::ref(volData), _1);
    //createSphereInVolume(volData, 30);
    //createPerlinTerrain(volData, Vector3DInt16(255,255,255));
    //createPerlinVolumeSlow(volData);
    std::cout << "Memory usage: " << volData.calculateSizeInBytes() << std::endl;
    //volData.setBlockCacheSize(8);
    std::cout << "Memory usage: " << volData.calculateSizeInBytes() << std::endl;
    std::cout << "Compression ratio: " << volData.calculateCompressionRatio() << std::endl;

    //Extract the surface
    SurfaceMesh<PositionMaterialNormal> mesh;
    SurfaceExtractor<MaterialDensityPair44> surfaceExtractor(&volData, PolyVox::Region(Vector3DInt16(1,1,1), Vector3DInt16(1022,253,253)), &mesh);
    surfaceExtractor.execute();

    //Pass the surface to the OpenGL window
    openGLWidget.setSurfaceMeshToRender(mesh);

    //Run the message pump.
    return app.exec();
}

Author:  David Williams [ Tue Mar 01, 2011 9:25 pm ]
Post subject:  Re: Streaming Volume

Excellent! But unfortunatly the patch does seem corrupted. Matt pointed out to me that the forum doesn't allow plain text attachments - I think I've fixed this now so please try attatching again. Otherwise you can send it to david at thermite3d dot org.

Author:  ker [ Tue Mar 01, 2011 9:39 pm ]
Post subject:  Re: Streaming Volume

actually I got 32 bit integers to work with surface extractor just a moment ago :D
so here is the patch I have only tested CubicSurfaceExtractorWithNormals and SurfaceExtractor

hey the adding worked :)

on another note: I believe the extractors to work with signed integers. I read through them and if I understood them correctly they're using signed integers everywhere where it's relevant anyway...

+ another thing:
blocks have a "modified" attribute. I would like to pass this to the callback functions... but in the future the callback function might get more than one block... since it's getting regions currently... do you have any thoughts on this?
the reason I'm asking is, that parts that were generated, won't need to be saved to disk/anywhere if they weren't modified. I'm still not sure if it would be really useful due to the computations required for regenerating blocks (where just loading from disk is probably faster)

[edit]
damn it... Netbeans messed up the alignment of lines and added random spaces... which causes diff to find differences in much more places than intended... do you want me to fix this?
[/edit]

Attachments:
polyvox_uint32_streaming.diff [72.51 KiB]
Downloaded 236 times

Author:  David Williams [ Tue Mar 01, 2011 11:23 pm ]
Post subject:  Re: Streaming Volume

Ok, it basically works :-)

There's quite a few things I would change/tidy (including that it now requires Boost, but of course this can be worked around with C++0x equivalents). But I don't think it's useful to list them all here, so I suggest you get it to a point where it is working in your project, tidy as much as possible, and then you send me another patch. The last one did indeed seem to have some alignment problems so it would be good if those can be sorted so I can see what has really changed.

It would also be nice if the examples were updated to work, but I appriciate than the OpenGL one is a bit of a mess. If you can't fix it I'll have a go myself.

I'll then commit your code and go over it myself. Give me a week or so to do this, at the end of which I'll probably have broken it for you but it can then be handed back to you for another pass.

Obviously we can discuss here all the changes we're making.

ker wrote:
+ another thing:
blocks have a "modified" attribute. I would like to pass this to the callback functions... but in the future the callback function might get more than one block... since it's getting regions currently... do you have any thoughts on this?
the reason I'm asking is, that parts that were generated, won't need to be saved to disk/anywhere if they weren't modified. I'm still not sure if it would be really useful due to the computations required for regenerating blocks (where just loading from disk is probably faster)


I'd say leave this out for now - a lot has already changed and we really want to keep this as simple as possible.

Author:  Shanee [ Wed Mar 02, 2011 8:38 am ]
Post subject:  Re: Streaming Volume

Ok I think I will delay my PolyVox implementation by a month or so, so I can get right on with streams ;)

Author:  ker [ Wed Mar 02, 2011 9:55 am ]
Post subject:  Re: Streaming Volume

sooo... now I checked out RLE again and made the changes only using gedit :D
it's a much neater diff file now

I couldn't figure out how to make the open gl example work... maybe my camera is simply pointing somewhere where there is nothing to see...
it compiles and it generates meshes... maybe you can figure it out
also I did not show off the streaming feature in the opengl example, I just tried to make it work like before.

I had to remove some constness from functions (getVoxel and getUncompressedBlock), because it's impossible to load a new block by setting coordinates in a const function... since that would require calling non const functions...
do you have an idea how to fix this or make it pretty at least?

Shanee wrote:
Ok I think I will delay my PolyVox implementation by a month or so, so I can get right on with streams ;)

try waiting until signed coordinates work, then it'll be really fun and simple to use (until you get to the point where your floating points start getting odd (which happens long before you reach the maximum int32 value) ) ;)

signed streaming volumes work with CubicSurfaceExtractor without even touching it... SurfaceExtractor has some problems thou... in the negative space, there is a problem at the block borders

Attachments:
File comment: hey at least you can see the stuff that used to be nonexistant :D
near zero crossing.png
near zero crossing.png [ 180.13 KiB | Viewed 3568 times ]
File comment: no problems with cubic surface extractor
cubic without problems.png
cubic without problems.png [ 133.59 KiB | Viewed 3568 times ]
polyvox_streaming_32.diff [70.2 KiB]
Downloaded 229 times

Author:  Shanee [ Wed Mar 02, 2011 12:49 pm ]
Post subject:  Re: Streaming Volume

Looks nice :)

Why would anyone expect to have floating points problems? You'd need to have something nearing the 100,000+ sized terrain for that I think?

I would be satisfied with a 20km sized terrain with a resolution of 1 unit = 1 meter :)

Reason I said I'd wait is because I imagine there is quite some cleaning and simplifying to do ;)

Author:  ker [ Wed Mar 02, 2011 1:14 pm ]
Post subject:  Re: Streaming Volume

well... I was thinking along the lines of really large maps where most of the map is maybe not filled with points of interest (like in a lord of the rings map where you'd walk from the hobbits' home to mordor... but still can start digging like crazy anywhere you want... HEY someone might implement a game like that ^^)

but yes, it's rather irrelevant at this point...

hehe yes, I'm sure I broke more than just the different renderers... it's better to wait

about the resolution... I have been talking with some pure gamers (who got no clue about code whatsoever)... they'd like voxel side lengths of about 10-20 cm

Page 2 of 15 All times are UTC
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group
http://www.phpbb.com/