It is currently Sat Aug 22, 2020 2:02 pm


All times are UTC




Post new topic Reply to topic  [ 23 posts ]  Go to page Previous  1, 2, 3  Next
Author Message
 Post subject: Re: Save Voxel Data to Disk
PostPosted: Sat Mar 19, 2011 1:34 pm 

Joined: Sun Jan 23, 2011 6:06 am
Posts: 92
Here is the code example for compressing data with zlib using POCO.

Code:
#include <Poco\DeflatingStream.h>
#include <Poco\InflatingStream.h>
#include <Poco\BinaryReader.h>
#include <Poco\BinaryWriter.h>

#include <iostream>
#include <fstream>

#include <stdint.h>
#include <string>

// Using 1D array for passing material data for simplicity, not bothering to check pointer.
void saveData(uint8_t* materialData, uint32_t arraySize, std::string fileName)
{
   std::ofstream outputStream(fileName.c_str(), std::ios::binary);
   Poco::DeflatingOutputStream deflator(outputStream, Poco::DeflatingStreamBuf::STREAM_ZLIB);
   Poco::BinaryWriter binaryWriter(deflator, Poco::BinaryWriter::NATIVE_BYTE_ORDER);

   for(uint32_t i = 0 ; i < arraySize ; i++)
      binaryWriter << materialData[i];
   
   deflator.close();   // this will perform the actual compression
   outputStream.close();
}

// Assumes that arraySize and file data match. Not taking EOF into account for example code.
void loadData(uint8_t* materialData, uint32_t arraySize, std::string fileName)
{
   std::ifstream inputStream(fileName.c_str(), std::ios::binary);
   Poco::InflatingInputStream inflator(inputStream, Poco::InflatingStreamBuf::STREAM_ZLIB);
   Poco::BinaryReader binaryReader(inflator, Poco::BinaryReader::NATIVE_BYTE_ORDER);

   for(uint32_t i = 0 ; i < arraySize ; i++)
      binaryReader >> materialData[i];

   inputStream.close();
}


POCO can be downloaded from http://pocoproject.org/


Top
Offline Profile  
Reply with quote  
 Post subject: Re: Save Voxel Data to Disk
PostPosted: Sat Mar 19, 2011 2:09 pm 
User avatar

Joined: Wed Jan 26, 2011 3:20 pm
Posts: 203
Location: Germany
for the lazy (ubuntu) ones among us: "sudo apt-get install libpoco-dev"
thx @ DefiniteIntegral, I've so far been manually using zlib and some other things that poco provides..
also this might solve shanee's problem as poco contains databases for raw data


Top
Offline Profile  
Reply with quote  
 Post subject: Re: Save Voxel Data to Disk
PostPosted: Sat Mar 19, 2011 8:19 pm 

Joined: Sat Sep 18, 2010 9:45 pm
Posts: 189
Please don't do what Minecraft is doing...

Just use B-tree (B+ tree). Tokyo cabinet use it. I would say for key/value store with respect to streaming to disk B-tree is the way to go. (Of course you could also use a hash table scheme but that requires more disk space; I'm not sure how well it works for harddisk read/write but Tokyo cabinent also support hash table scheme so I guess it must be alright).


Top
Offline Profile  
Reply with quote  
 Post subject: Re: Save Voxel Data to Disk
PostPosted: Sun Mar 20, 2011 2:58 am 

Joined: Sun Jan 23, 2011 6:06 am
Posts: 92
Tokyo Cabinet eh. Sounds interesting. I don't really have time to try it now, but I shall have to do some speed testing later to see how B+Tree compares to lazy minecraft style data storage.


Top
Offline Profile  
Reply with quote  
 Post subject: Re: Save Voxel Data to Disk
PostPosted: Tue Mar 22, 2011 7:26 pm 

Joined: Fri Feb 18, 2011 8:41 pm
Posts: 173
What is Poco? what can I do with it? Do I need the "complete" version or the starter one?

What does each support? Didn't find all the info on the site


Top
Offline Profile  
Reply with quote  
 Post subject: Re: Save Voxel Data to Disk
PostPosted: Sun Mar 27, 2011 3:37 am 
User avatar

Joined: Tue Feb 22, 2011 8:04 am
Posts: 101
I have been going through the POCO site and reading through their documentation. They are very very sparse on details.

Do they have database capability, or just an interface to other database libraries?

From my readings, and what has been stated on this thread, I think a database would be the best way to save my volume data. That way I could keep it in one file, and use location as a key value.

However POCO never seems to come out and say whether they have database capability, a database interface, or both.

Does anyone have any experience in this area?

Also it seems this 'Tokyo Cabinet' is free and has some nice database features. Has anyone used this before? It might be a good substitute for POCO's features.

_________________
--Real Programmers use a magnetized needle and a steady hand.
--xkcd--


Top
Offline Profile  
Reply with quote  
 Post subject: Re: Save Voxel Data to Disk
PostPosted: Sun Mar 27, 2011 4:33 am 

Joined: Sun Jan 23, 2011 6:06 am
Posts: 92
I haven't used POCO for database access. I just use the standard version for stuff unrelated to databases, like thread/mutex/event/zlib/xml/hashing/filesystem/timestamps etc, because it is convenient.

I believe POCO is just an interface for MySQL anyway.... ? Tokyo Cabinet looks better for database access.

My main worry with a database is that a tiny bit of data corruption might wreck the entire save, which wouldn't be such an issue with separate files.

Beyzend can you comment on this?


Top
Offline Profile  
Reply with quote  
 Post subject: Re: Save Voxel Data to Disk
PostPosted: Sun Mar 27, 2011 7:52 am 

Joined: Sat Sep 18, 2010 9:45 pm
Posts: 189
DefiniteIntegral,

I'm not sure tell you the truth. I have very little experience with filesystem and db systems and no experience with B+ and B- trees. I google it, it seems CouchDB's implementation is very nice, it's only does appends to file and other safeguards--almost requiring no transactional redundancies. Plus CouchDB's system is also effective at concurrency. May want to look into it.

But just looking at B+ tree, it looks like things like safety and robustness can be implemented. I don't know, I have to look at it further. (Well, as long as your interior nodes don't get messed up, you can always just regenerate a chunk--the leaf node--in the rare case that is messed up.)I was just going to use some filesystem library built on top of B+ tree and do a fire and forget like I always do, i.e. I don't really care. But perhaps one can build some safety on top of the said filesystem, just to be more secure, I guess.

Another option I considered is building Tokyo Cabinet. I's Linux only but after that googling it seems as long you don't run the networking part of that system you can use it under Windows.


Top
Offline Profile  
Reply with quote  
 Post subject: Re: Save Voxel Data to Disk
PostPosted: Tue Mar 29, 2011 5:55 am 
User avatar

Joined: Tue Feb 22, 2011 8:04 am
Posts: 101
Quote:
My main worry with a database is that a tiny bit of data corruption might wreck the entire save, which wouldn't be such an issue with separate files.


Hmm you have a point with the reliability of a database. It could be a problem if it became corrupted. I'm sure software problems could be handled by good code, and hardware could be resolved by automatic backups. Still it would be nice to only lose part of the data in the event of a crash.

It also seems the possibility of using Tokyo Cabinet is slim to none. It is very linux dependent, and I can't figure out how to get around that fact. No one has developed a port to Windows either.

With this being the case, this BTree organization sounds promising, but I don't really understand how to apply this concept.

Do you mind giving me an overview on how to apply this kind of file saving? I am unsure how to create this kind of file.

Thanks for your help so far!

_________________
--Real Programmers use a magnetized needle and a steady hand.
--xkcd--


Top
Offline Profile  
Reply with quote  
 Post subject: Re: Save Voxel Data to Disk
PostPosted: Tue Mar 29, 2011 3:42 pm 

Joined: Sat Sep 18, 2010 9:45 pm
Posts: 189
The thing about it is regarding couchdb's implementation of append only B-tree is the way they do it it will get rid of problems with consistency (not sure if that's the correct usage) since for each update you don't touch what has been committed to disk--and this does a lot for safety if no previous written data are touched (not sure what they do to clean up old data). The index writing on the other hand, has redundancy build-in that also serves this purpose. At least that's what I got just from a 5 min reading of different blogs.

http://guide.couchdb.org/draft/btree.html#figure/1

http://horicky.blogspot.com/2008/10/cou ... ation.html

http://jchrisa.net/drl/_design/sofa/_li ... ithm%22%5D

But for my game I'm not going to do append only because I'm not making a DB I don't need snapshots, revisions, and all that other crap. I was going to use:

http://www.scalingweb.com/bplus_tree.php

I was hoping it has some safety build-in.

Anyway, I think in a game, you can just pass off responsibility to the player. Let them hit save. Let them think they are responsible...lol. So just backup a little once in awhile. If something screws up, go back a previous backup save. Tell the player, well you should have saved often. But since we're the shit, here is a backup...don't forget to save..smile.

The backup can be fast with B-trees because it's just backing up the index. In the case that a leaf-node is corrupt...do what Minecraft... regenerate chunk like a boss. (Wait, but how would you know if a chunk is corrupt? some sort of check?)

1. Backup index.
2. ?
3. ?
4. PROFIT like a boss.


Top
Offline Profile  
Reply with quote  
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 23 posts ]  Go to page Previous  1, 2, 3  Next

All times are UTC


Who is online

Users browsing this forum: No registered users and 7 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
cron
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group
Theme created StylerBB.net