It is currently Sat Aug 22, 2020 12:29 pm


All times are UTC




Post new topic Reply to topic  [ 150 posts ]  Go to page Previous  1, 2, 3, 4, 5 ... 15  Next
Author Message
 Post subject: Re: Streaming Volume
PostPosted: Sat Feb 26, 2011 10:15 am 
Developer
User avatar

Joined: Sun May 04, 2008 6:35 pm
Posts: 1827
This all sounds very promising, and in line with what I was planning :-)

ker wrote:
eeeeh, yes :D I'd be happy to supply my streamed volume, but FIRST I gotta clean up that code, I am just starting to use somewhat proper variable naming (again)... + I forgot many probably useful comments


Sure, there is no hurry, and it's always suprising how long it takes to tidy up code. But I think I will avoid commiting anything to the new volume branch in order to avoid conflicts with your work. That way you can send a patch and it should apply without problems.

ker wrote:
you are allowed to register your own loadBlock and unloadBlock functions (using boost::bind, if this is an issue due to dependencies on boost, I can modify this to some messy pointer arithmetic)
also unordered_map is only in c++0x


std::function is part of C++0x anyway so there shouldn't be a problem here. And is there a version of unordered_map in Boost as well?

ker wrote:
anyway, loadBlock is called, when you try to access (set/getVoxel) a nonexisting/not_yet_loaded block.
it gives you a block reference which you can fill with your own data if you wish and a Vector3DInt64 which is the position of the Block in Block-coordinates (shift this by your Block-sidelength to get Volume coordinates)
if you don't set a loadBlock function, the default is used, which is filling the block with VoxelType().
unloadBlock works almost the same... you get block coordinates so you know where this block belongs and you get the Block to somehow extract the data from and save however you want. doing nothing in the unloadBlock function causes the data to be lost.

unloadBlock is called whenever the size of the unordered_map reaches some threshhold. least recently used then decides which block to drop.

Ok, the only thing I would say here is that it is exposing the Block class to the user, when it was previously an implementation detail. Now maybe there is no way around that, but think whether it is possible to pass a Region instead of a Block. Then the user code could use the Volume's setVoxelAt() methods to fill in the data in the specified region. This allows us more flexibility in the future if we want to change the internal representation.

Also, maybe just 32 bit ints for positions? 64 seems a bit excessive, and I don't know how well they work on mobile platforms for example.


Top
Offline Profile  
Reply with quote  
 Post subject: Re: Streaming Volume
PostPosted: Sat Feb 26, 2011 5:59 pm 
User avatar

Joined: Wed Jan 26, 2011 3:20 pm
Posts: 203
Location: Germany
1. use uint32_t
2. pass a Region and a reference to the volume instead of the block
3. check boost for unordered_map

will come back when I'm done with it ^^


Top
Offline Profile  
Reply with quote  
 Post subject: Re: Streaming Volume
PostPosted: Tue Mar 01, 2011 1:39 pm 
User avatar

Joined: Wed Jan 26, 2011 3:20 pm
Posts: 203
Location: Germany
ok.. I got into some trouble with the 32bit ints... I'm working on it thou...
here is a 16 bit streaming world that will probably crash if you move to the upper range of a uint16

I'm writing an example currently to show off the streaming feature (like create perlin noise on the fly when requested instead of precomputing the volume)

I'm unable to upload files... posting it somewhere else
it looks a little odd... I hope this didn't mess up the diff file (the upload added blank lines everywhere)

http://paste.pocoo.org/raw/JWzOXN5vup9oVhRBHysH/

ok... this was easier than expected:

this is an example callback for volume data creation
Code:
void createPerlinTerrainChunk(Volume<MaterialDensityPair44>& volData, PolyVox::Region reg)
{
    Perlin perlin(2,2,1,234);
    for(int x = reg.getLowerCorner().getX(); x < reg.getUpperCorner().getX(); x++) {
        for(int y = reg.getLowerCorner().getY(); y < reg.getUpperCorner().getY(); y++) {
            float perlinVal = perlin.Get(x / 254.0f , y / 254.0f);
            perlinVal += 1.0f;
            perlinVal *= 0.5f;
            perlinVal *= 255.0f;

            for(int z = reg.getLowerCorner().getZ(); z < reg.getUpperCorner().getZ(); z++) {
                MaterialDensityPair44 voxel;
                if(z < perlinVal)
                {
                    voxel.setMaterial(245);
                    voxel.setDensity(MaterialDensityPair44::getMaxDensity());
                }
                else
                {
                    voxel.setMaterial(0);
                    voxel.setDensity(MaterialDensityPair44::getMinDensity());
                }

                volData.setVoxelAt(x, y, z, voxel);
            }
        }
    }
}





and this is how you use it:

Code:
int main(int argc, char *argv[])
{
    //Create and show the Qt OpenGL window
    QApplication app(argc, argv);
    OpenGLWidget openGLWidget(0);
    openGLWidget.show();

    //Create an empty volume and then place a sphere in it
    Volume<MaterialDensityPair44> volData;
    volData.setBlockCacheSize(4096);
    volData.m_fLoad = boost::bind(&createPerlinTerrainChunk, boost::ref(volData), _1);
    //createSphereInVolume(volData, 30);
    //createPerlinTerrain(volData, Vector3DInt16(255,255,255));
    //createPerlinVolumeSlow(volData);
    std::cout << "Memory usage: " << volData.calculateSizeInBytes() << std::endl;
    //volData.setBlockCacheSize(8);
    std::cout << "Memory usage: " << volData.calculateSizeInBytes() << std::endl;
    std::cout << "Compression ratio: " << volData.calculateCompressionRatio() << std::endl;

    //Extract the surface
    SurfaceMesh<PositionMaterialNormal> mesh;
    SurfaceExtractor<MaterialDensityPair44> surfaceExtractor(&volData, PolyVox::Region(Vector3DInt16(1,1,1), Vector3DInt16(1022,253,253)), &mesh);
    surfaceExtractor.execute();

    //Pass the surface to the OpenGL window
    openGLWidget.setSurfaceMeshToRender(mesh);

    //Run the message pump.
    return app.exec();
}


Top
Offline Profile  
Reply with quote  
 Post subject: Re: Streaming Volume
PostPosted: Tue Mar 01, 2011 9:25 pm 
Developer
User avatar

Joined: Sun May 04, 2008 6:35 pm
Posts: 1827
Excellent! But unfortunatly the patch does seem corrupted. Matt pointed out to me that the forum doesn't allow plain text attachments - I think I've fixed this now so please try attatching again. Otherwise you can send it to david at thermite3d dot org.


Top
Offline Profile  
Reply with quote  
 Post subject: Re: Streaming Volume
PostPosted: Tue Mar 01, 2011 9:39 pm 
User avatar

Joined: Wed Jan 26, 2011 3:20 pm
Posts: 203
Location: Germany
actually I got 32 bit integers to work with surface extractor just a moment ago :D
so here is the patch I have only tested CubicSurfaceExtractorWithNormals and SurfaceExtractor

hey the adding worked :)

on another note: I believe the extractors to work with signed integers. I read through them and if I understood them correctly they're using signed integers everywhere where it's relevant anyway...

+ another thing:
blocks have a "modified" attribute. I would like to pass this to the callback functions... but in the future the callback function might get more than one block... since it's getting regions currently... do you have any thoughts on this?
the reason I'm asking is, that parts that were generated, won't need to be saved to disk/anywhere if they weren't modified. I'm still not sure if it would be really useful due to the computations required for regenerating blocks (where just loading from disk is probably faster)

[edit]
damn it... Netbeans messed up the alignment of lines and added random spaces... which causes diff to find differences in much more places than intended... do you want me to fix this?
[/edit]


Attachments:
polyvox_uint32_streaming.diff [72.51 KiB]
Downloaded 236 times
Top
Offline Profile  
Reply with quote  
 Post subject: Re: Streaming Volume
PostPosted: Tue Mar 01, 2011 11:23 pm 
Developer
User avatar

Joined: Sun May 04, 2008 6:35 pm
Posts: 1827
Ok, it basically works :-)

There's quite a few things I would change/tidy (including that it now requires Boost, but of course this can be worked around with C++0x equivalents). But I don't think it's useful to list them all here, so I suggest you get it to a point where it is working in your project, tidy as much as possible, and then you send me another patch. The last one did indeed seem to have some alignment problems so it would be good if those can be sorted so I can see what has really changed.

It would also be nice if the examples were updated to work, but I appriciate than the OpenGL one is a bit of a mess. If you can't fix it I'll have a go myself.

I'll then commit your code and go over it myself. Give me a week or so to do this, at the end of which I'll probably have broken it for you but it can then be handed back to you for another pass.

Obviously we can discuss here all the changes we're making.

ker wrote:
+ another thing:
blocks have a "modified" attribute. I would like to pass this to the callback functions... but in the future the callback function might get more than one block... since it's getting regions currently... do you have any thoughts on this?
the reason I'm asking is, that parts that were generated, won't need to be saved to disk/anywhere if they weren't modified. I'm still not sure if it would be really useful due to the computations required for regenerating blocks (where just loading from disk is probably faster)


I'd say leave this out for now - a lot has already changed and we really want to keep this as simple as possible.


Top
Offline Profile  
Reply with quote  
 Post subject: Re: Streaming Volume
PostPosted: Wed Mar 02, 2011 8:38 am 

Joined: Fri Feb 18, 2011 8:41 pm
Posts: 173
Ok I think I will delay my PolyVox implementation by a month or so, so I can get right on with streams ;)


Top
Offline Profile  
Reply with quote  
 Post subject: Re: Streaming Volume
PostPosted: Wed Mar 02, 2011 9:55 am 
User avatar

Joined: Wed Jan 26, 2011 3:20 pm
Posts: 203
Location: Germany
sooo... now I checked out RLE again and made the changes only using gedit :D
it's a much neater diff file now

I couldn't figure out how to make the open gl example work... maybe my camera is simply pointing somewhere where there is nothing to see...
it compiles and it generates meshes... maybe you can figure it out
also I did not show off the streaming feature in the opengl example, I just tried to make it work like before.

I had to remove some constness from functions (getVoxel and getUncompressedBlock), because it's impossible to load a new block by setting coordinates in a const function... since that would require calling non const functions...
do you have an idea how to fix this or make it pretty at least?

Shanee wrote:
Ok I think I will delay my PolyVox implementation by a month or so, so I can get right on with streams ;)

try waiting until signed coordinates work, then it'll be really fun and simple to use (until you get to the point where your floating points start getting odd (which happens long before you reach the maximum int32 value) ) ;)

signed streaming volumes work with CubicSurfaceExtractor without even touching it... SurfaceExtractor has some problems thou... in the negative space, there is a problem at the block borders


Attachments:
File comment: hey at least you can see the stuff that used to be nonexistant :D
near zero crossing.png
near zero crossing.png [ 180.13 KiB | Viewed 3565 times ]
File comment: no problems with cubic surface extractor
cubic without problems.png
cubic without problems.png [ 133.59 KiB | Viewed 3565 times ]
polyvox_streaming_32.diff [70.2 KiB]
Downloaded 229 times
Top
Offline Profile  
Reply with quote  
 Post subject: Re: Streaming Volume
PostPosted: Wed Mar 02, 2011 12:49 pm 

Joined: Fri Feb 18, 2011 8:41 pm
Posts: 173
Looks nice :)

Why would anyone expect to have floating points problems? You'd need to have something nearing the 100,000+ sized terrain for that I think?

I would be satisfied with a 20km sized terrain with a resolution of 1 unit = 1 meter :)

Reason I said I'd wait is because I imagine there is quite some cleaning and simplifying to do ;)


Top
Offline Profile  
Reply with quote  
 Post subject: Re: Streaming Volume
PostPosted: Wed Mar 02, 2011 1:14 pm 
User avatar

Joined: Wed Jan 26, 2011 3:20 pm
Posts: 203
Location: Germany
well... I was thinking along the lines of really large maps where most of the map is maybe not filled with points of interest (like in a lord of the rings map where you'd walk from the hobbits' home to mordor... but still can start digging like crazy anywhere you want... HEY someone might implement a game like that ^^)

but yes, it's rather irrelevant at this point...

hehe yes, I'm sure I broke more than just the different renderers... it's better to wait

about the resolution... I have been talking with some pure gamers (who got no clue about code whatsoever)... they'd like voxel side lengths of about 10-20 cm


Top
Offline Profile  
Reply with quote  
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 150 posts ]  Go to page Previous  1, 2, 3, 4, 5 ... 15  Next

All times are UTC


Who is online

Users browsing this forum: No registered users and 5 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
cron
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group
Theme created StylerBB.net