| Volumes Of Fun http://www.volumesoffun.com/phpBB3/ |
|
| Save Voxel Data to Disk http://www.volumesoffun.com/phpBB3/viewtopic.php?f=14&t=176 |
Page 2 of 3 |
| Author: | DefiniteIntegral [ Sat Mar 19, 2011 1:34 pm ] |
| Post subject: | Re: Save Voxel Data to Disk |
Here is the code example for compressing data with zlib using POCO. Code: #include <Poco\DeflatingStream.h> #include <Poco\InflatingStream.h> #include <Poco\BinaryReader.h> #include <Poco\BinaryWriter.h> #include <iostream> #include <fstream> #include <stdint.h> #include <string> // Using 1D array for passing material data for simplicity, not bothering to check pointer. void saveData(uint8_t* materialData, uint32_t arraySize, std::string fileName) { std::ofstream outputStream(fileName.c_str(), std::ios::binary); Poco::DeflatingOutputStream deflator(outputStream, Poco::DeflatingStreamBuf::STREAM_ZLIB); Poco::BinaryWriter binaryWriter(deflator, Poco::BinaryWriter::NATIVE_BYTE_ORDER); for(uint32_t i = 0 ; i < arraySize ; i++) binaryWriter << materialData[i]; deflator.close(); // this will perform the actual compression outputStream.close(); } // Assumes that arraySize and file data match. Not taking EOF into account for example code. void loadData(uint8_t* materialData, uint32_t arraySize, std::string fileName) { std::ifstream inputStream(fileName.c_str(), std::ios::binary); Poco::InflatingInputStream inflator(inputStream, Poco::InflatingStreamBuf::STREAM_ZLIB); Poco::BinaryReader binaryReader(inflator, Poco::BinaryReader::NATIVE_BYTE_ORDER); for(uint32_t i = 0 ; i < arraySize ; i++) binaryReader >> materialData[i]; inputStream.close(); } POCO can be downloaded from http://pocoproject.org/ |
|
| Author: | ker [ Sat Mar 19, 2011 2:09 pm ] |
| Post subject: | Re: Save Voxel Data to Disk |
for the lazy (ubuntu) ones among us: "sudo apt-get install libpoco-dev" thx @ DefiniteIntegral, I've so far been manually using zlib and some other things that poco provides.. also this might solve shanee's problem as poco contains databases for raw data |
|
| Author: | beyzend [ Sat Mar 19, 2011 8:19 pm ] |
| Post subject: | Re: Save Voxel Data to Disk |
Please don't do what Minecraft is doing... Just use B-tree (B+ tree). Tokyo cabinet use it. I would say for key/value store with respect to streaming to disk B-tree is the way to go. (Of course you could also use a hash table scheme but that requires more disk space; I'm not sure how well it works for harddisk read/write but Tokyo cabinent also support hash table scheme so I guess it must be alright). |
|
| Author: | DefiniteIntegral [ Sun Mar 20, 2011 2:58 am ] |
| Post subject: | Re: Save Voxel Data to Disk |
Tokyo Cabinet eh. Sounds interesting. I don't really have time to try it now, but I shall have to do some speed testing later to see how B+Tree compares to lazy minecraft style data storage. |
|
| Author: | Shanee [ Tue Mar 22, 2011 7:26 pm ] |
| Post subject: | Re: Save Voxel Data to Disk |
What is Poco? what can I do with it? Do I need the "complete" version or the starter one? What does each support? Didn't find all the info on the site |
|
| Author: | GM_Riscvul [ Sun Mar 27, 2011 3:37 am ] |
| Post subject: | Re: Save Voxel Data to Disk |
I have been going through the POCO site and reading through their documentation. They are very very sparse on details. Do they have database capability, or just an interface to other database libraries? From my readings, and what has been stated on this thread, I think a database would be the best way to save my volume data. That way I could keep it in one file, and use location as a key value. However POCO never seems to come out and say whether they have database capability, a database interface, or both. Does anyone have any experience in this area? Also it seems this 'Tokyo Cabinet' is free and has some nice database features. Has anyone used this before? It might be a good substitute for POCO's features. |
|
| Author: | DefiniteIntegral [ Sun Mar 27, 2011 4:33 am ] |
| Post subject: | Re: Save Voxel Data to Disk |
I haven't used POCO for database access. I just use the standard version for stuff unrelated to databases, like thread/mutex/event/zlib/xml/hashing/filesystem/timestamps etc, because it is convenient. I believe POCO is just an interface for MySQL anyway.... ? Tokyo Cabinet looks better for database access. My main worry with a database is that a tiny bit of data corruption might wreck the entire save, which wouldn't be such an issue with separate files. Beyzend can you comment on this? |
|
| Author: | beyzend [ Sun Mar 27, 2011 7:52 am ] |
| Post subject: | Re: Save Voxel Data to Disk |
DefiniteIntegral, I'm not sure tell you the truth. I have very little experience with filesystem and db systems and no experience with B+ and B- trees. I google it, it seems CouchDB's implementation is very nice, it's only does appends to file and other safeguards--almost requiring no transactional redundancies. Plus CouchDB's system is also effective at concurrency. May want to look into it. But just looking at B+ tree, it looks like things like safety and robustness can be implemented. I don't know, I have to look at it further. (Well, as long as your interior nodes don't get messed up, you can always just regenerate a chunk--the leaf node--in the rare case that is messed up.)I was just going to use some filesystem library built on top of B+ tree and do a fire and forget like I always do, i.e. I don't really care. But perhaps one can build some safety on top of the said filesystem, just to be more secure, I guess. Another option I considered is building Tokyo Cabinet. I's Linux only but after that googling it seems as long you don't run the networking part of that system you can use it under Windows. |
|
| Author: | GM_Riscvul [ Tue Mar 29, 2011 5:55 am ] |
| Post subject: | Re: Save Voxel Data to Disk |
Quote: My main worry with a database is that a tiny bit of data corruption might wreck the entire save, which wouldn't be such an issue with separate files. Hmm you have a point with the reliability of a database. It could be a problem if it became corrupted. I'm sure software problems could be handled by good code, and hardware could be resolved by automatic backups. Still it would be nice to only lose part of the data in the event of a crash. It also seems the possibility of using Tokyo Cabinet is slim to none. It is very linux dependent, and I can't figure out how to get around that fact. No one has developed a port to Windows either. With this being the case, this BTree organization sounds promising, but I don't really understand how to apply this concept. Do you mind giving me an overview on how to apply this kind of file saving? I am unsure how to create this kind of file. Thanks for your help so far! |
|
| Author: | beyzend [ Tue Mar 29, 2011 3:42 pm ] |
| Post subject: | Re: Save Voxel Data to Disk |
The thing about it is regarding couchdb's implementation of append only B-tree is the way they do it it will get rid of problems with consistency (not sure if that's the correct usage) since for each update you don't touch what has been committed to disk--and this does a lot for safety if no previous written data are touched (not sure what they do to clean up old data). The index writing on the other hand, has redundancy build-in that also serves this purpose. At least that's what I got just from a 5 min reading of different blogs. http://guide.couchdb.org/draft/btree.html#figure/1 http://horicky.blogspot.com/2008/10/cou ... ation.html http://jchrisa.net/drl/_design/sofa/_li ... ithm%22%5D But for my game I'm not going to do append only because I'm not making a DB I don't need snapshots, revisions, and all that other crap. I was going to use: http://www.scalingweb.com/bplus_tree.php I was hoping it has some safety build-in. Anyway, I think in a game, you can just pass off responsibility to the player. Let them hit save. Let them think they are responsible...lol. So just backup a little once in awhile. If something screws up, go back a previous backup save. Tell the player, well you should have saved often. But since we're the shit, here is a backup...don't forget to save..smile. The backup can be fast with B-trees because it's just backing up the index. In the case that a leaf-node is corrupt...do what Minecraft... regenerate chunk like a boss. (Wait, but how would you know if a chunk is corrupt? some sort of check?) 1. Backup index. 2. ? 3. ? 4. PROFIT like a boss. |
|
| Page 2 of 3 | All times are UTC |
| Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group http://www.phpbb.com/ |
|