FOSS gamedev and creative worlds

Hello everyone,

I'm the author of WorldPainter, an interactive map generator for Minecraft. I've been asked a few times whether I could support Minetest as well, and decided I'd look into it.

One thing I want to do is gauge the level of interest. Are many people interested in this? Also, are there already other tools for Minetest which do the same or similar things?

Another question is whether someone can direct me to information about the file format. I can probably figure it out from the source code, but if there is any documentation out there describing the details that might speeds things up considerably.

Thanks in advance for any answers and information you can give!

Cheers,
Captain_Chaos

Hello and welcome to Minetest.

Do I see it right that this mod is able to use images to generate maps, similar to this mod?
The videos I've seen look very promising, with the advanced terrain editor that allows users to easily draw and generate maps. I'm interested in map generation and especially in generating custom maps that do not base on perin noise - to get a variety. Hopefully other people think so too.

What do you mean with "direct me to information about the file format"? If you're looking for the method to generate terrain, you will find the functions in the lua_api.txt file and further descriptions in the wiki.

Krock wrote:Do I see it right that this mod is able to use images to generate maps, similar to this mod?

I'm not sure what that mod does, but I'll explain more fully what WorldPainter does: WorldPainter is not a mod, but a stand alone program, with which you generate maps for Minecraft. So instead of Minecraft generating the blocks, or Minecraft mods generating the blocks, WorldPainter generates them according to your instructions about the terrain height and type, and any additional "layers" you've painted which can add things like trees and other vegetation, caves or any type of custom object.

You can paint and sculpt the terrain by hand, but you can also use height maps and/or image masks, which I think answers your question.

Krock wrote:What do you mean with "direct me to information about the file format"?

WorldPainter is a stand alone program, which pre-generates all the blocks for the entire map and stores them on disk in Minecraft's native format. In order to support Minetest I would have to know what Minetest's on-disk file format is so I can do the same for it. WorldPainter is not a mod, and as such does not run inside Minetest and cannot make use of its runtime APIs. In theory something like that might be possible, but an additional hurdle for that is that Minetest is written in C++ and WorldPainter in Java. So the best alternative is probably for WorldPainter to directly write the blocks to disk in Minetest's native block storage format. What I'm looking into now is what that format is and therefore how feasible that would be.

I think I have seen that Minetest actually supports different backends for storing the blocks. Is there a de facto standard which WorldPainter could support and be useful to most users? Can one of the formats easily be converted to the other formats so that it would make sense to support that format? Is there any detailed documentation about the technical implementation details (data models, data structures, etc.) of these storage formats/backends?

The format is more or less always the same, (almost) the only thing that changes is how the blocks are stored in the db. More precisely, each 16*16*16 block is first serialized, then inserted into the db, the serialization being the same for all db backends. The most used one is sqlite3, as it was originally the only one Minetest supported, so you can be sure most if not all users will be able to open such maps.

As for the precise block format, there unfortunately is no documentation that I know of. However you can look at some code producing it, such as https://github.com/minetest/minetest/blob/master/src/mapblock.cpp#L556 or https://github.com/minetest-tools/mcimport/blob/master/block.py#L344.

Also, beware the following differences between MC and MT:
- The x and z axes do not work exactly as they work in MT: if you proceed the same way, then the map will be flipped.
- Light is stored as light on day and night, not as sunlight and artificial light.

PRE-EDIT: Nice ninja, Nore ;)

Thanks for your explanation. I assumed it would be imported with a mod - but actually you're writing the stuff directly to the database.
There are multiple backends, such as SQLite3, LevelDB, Redis and PostgreSQL (sorted by addition date, where SQLite3 is the longest and likely the most used one). A full documentation of how the database structure looks like can be found in the file doc/world_format.txt. This structure is the same for all backends, only the functions to insert, delete or update the data varies.

C++ code that could be helpful:
https://github.com/minetest/minetestmap ... r.cpp#L362
This tool reads the database contents and generates a map from it. My knowledge about this is near to zero, thus it might be better to ask some core developers on freenode/#minetest-dev.

Pre-edit: oh, looks like his response is already complete :rotfl:

Just FYI, I for one would be very interested in seeing something like this. However, the MT developers can give a lot more info on this than I can, so I'll leave it to them (and Krock, seeing as told me he was preparing a response).

Thanks guys, this is all very useful and exactly what I was looking for. I'll look into how feasible it would be to support Minetest.

How do the available blocks compare between Minetest and Minecraft? Do the blocks have numerical ids similar to those of Minecraft? Is there a handy overview somewhere of all the blocks Minetest knows about? Are there similar mechanisms for more complex blocks containing data, such as chests, and for mobs?

There are no blocks id per se, all mapblocks contain something called "name-id mapping", that explain the mapping between the id of a node in the mapblock, and its name (like, "default:stone" for stone). To know the name of the various nodes, you can look that up on the wiki (http://wiki.minetest.net/Stone). Finally, for chests and the like, their additional information is stored in metadata; the easiest way to understand the format for MC-like nodes like chests would perhaps be to look at the code of mcimport. For mobs, well... there are no mobs in the "default" game (Minetest Game), so you need to look at some mob mod for that.

Ah, interesting. That is something that is sorely missing in Minecraft's file format, but it does mean some significant changes for WorldPainter, as it is based on the assumption of a static block to id mapping. Nevertheless it seems doable, especially if I leave mod support for later.

Wow this is great news.
It would be wonderfull if the WorldPainter could export to Minetest. I used the program with Minecraft and have used it to create content for Minetest by using some of the converters listed below

mcimport - import Minecraft classic maps by Sokomine
Converter Minecraft world to Minetest world by dgm5555
Convert parts of Minecraft maps to Minetest WorldEdit schematics by sfan5
Convert Minecraft maps to Minetest worlds by sofar
Minecraft to Minetest Block Converter (mcblocks) by dgm5555
compatibility nodes for MC schematics [mccompat] by Sokomine
fork of mcimport by dgm3333

I guess for a start this would be the easiest way as the node organization between mt and mc differs quite a lot. Basically mc has a fixed set of posible nodes if I remember correctly, while mt has a very open organization. Anybody can and countless have added mods for minetest which then have added an completetly unorganized number of new nodes to mt that number in the thousands. There is no central list and some are unique to single servers which may or may not have published the relevant source code.

So as stated above the easiest way would probably be for a start to be able to export a map created as for mc to the mt format. I would also suggest to stick to sqlite for a start as the backend formats can be converted to other formats by mt itself quite easily.

Minetest is organized as an engine that is expanded by some Lua scripts. The game content itself such as nodes and mobs is provided as mods. A subgame is a collection of mods that are designed to work well together and provide a complete gameplay in contrast to just a collection of mods such as mod-packs that might provide a group of mods about a certain subject or feature. So in contrast to mc there is no standalone version of mt. Minetest is always a combination of "Minetest the engine" that runs a "Minetest subgame" and this may or may not also include some more mod-packs or single mods. The Minetest Server List might give an insight to the great mods variety. You may view some of the mods listed that are installed by hovering along the column named "Version, Subgame, Mapgenerator"

There have been some efforts to create subgames that either recreate the gameplay of mc or to allow easily to play maps converted from mc. MineClone 2 by Wuzzy and others is the most recent project that might be of interest here. Others may be found in this List of Subgames

MinetestMapper is a group of programs that can draw overview maps of minetest worlds, and as such can be used to analyze them. For example if you run the MinetestMapperGui - Graphical userinterface for mtmapper you can create a colors.txt file that could be simply used as node list. If you supply an empty colors.txt file and try to map a world the program will complain and provide a complete list of all nodes present in the map. A very large but far from complete colors.txt file that focuses on the "Dreambuilder Modpack" that contains many of the more popular mods can be found on digitalaudioconcepts or another that focuses on the Ethereal mods is the xaagle-colorfile.

Thanks for the extensive info, twoelk!

It sounds like I would have to support Minetest mods in WorldPainter for it to have any practical use. Would you agree? Although I do understand there is a default subgame with the basic Minecraft-like blocks in it (the blocks I see if I just download and run Minetest); would it be of any use to support only those blocks, at least initially, or is the actual use of that block set very limited in practice?

I'm not an expert in this area but from what I've seen most mods actually depend on the minetest_game blocks and resources. In my opinion you could definitely initially start off with supporting those blocks and they are most common such as dirt and trees and such.

Also this sounds super exciting and useful for my students as well as we could recreate existing and past areas very quickly or even our own new creations. Thank you for taking an interest in supporting Minetest!

Thanks Captain_Chaos for your interest in minetest It really makes me happy!

Captain_Chaos wrote:Thanks for the extensive info, twoelk!

It sounds like I would have to support Minetest mods in WorldPainter for it to have any practical use. Would you agree? Although I do understand there is a default subgame with the basic Minecraft-like blocks in it (the blocks I see if I just download and run Minetest); would it be of any use to support only those blocks, at least initially, or is the actual use of that block set very limited in practice?

The `factory` game which most mods rely on is `minetest_game`. (github.com/minetest/minetest_game)

Extracting the blocks is relatively simple because we can easily run some quick code in minetest and have it spit out a list:

Put this in ~/.minetest/worlds/test/worldmods/test/init.lua:



And it spits out the full list that your minetest currently understands.

My output for 0.4.15 minetest:

https://gist.github.com/sofar/1c9475bea ... 25f89ed669

I think you could probably settle on a basic set of landscape blocksnodes (dirt/stones, plants, liquids, etc.) to start with since that will already get you all the blocks that most people need. The rest of the blocks could be optionally input by a user.

We could also make some trivial tooling to output these lists (and add e.g. colors to the exported list) so that adding more would become trivial for users to do themselves.

This would be a really great thing to have! Thanks for considering supporting minetest

This is a very nice gesture.
And I think the interest is fairly solid. And if the option does become available the interest will only grow from there.

More resources that might help:
Basic usage of the SQLite3 storage of MapBlocks
Converting between pos in the sqlite3 and MapBlock positions
deserializing MapBlocks
A list of (almost) all default nodes including their rough RGB colors

Another question: I see that there are is a whole list of different versions of the serialization format. Could I get away with supporting only one version, for instance the latest one? Is there a current de facto standard version which most people are using? Or are people using wildly different versions?

I suggest you use the oldest version that supports all the features you want; version 25 may be a good choice since it is already quite old (at least a few years), while version 27 was introduced just a few days ago to fix lighting problems, so pre-v27 mapblocks may be slightly slower to load on new builds (but only the first time). If you want to support old versions, such as the stable one, I think it is best to write v25 maps; if you care less about that, then you can support the lastest version, but only new servers will be able to open the maps.
(Also, there is a version 26, but I don't remember the difference with version 25.)

Nore wrote:I suggest you use the oldest version that supports all the features you want; version 25 may be a good choice since it is already quite old (at least a few years), while version 27 was introduced just a few days ago to fix lighting problems, so pre-v27 mapblocks may be slightly slower to load on new builds (but only the first time). If you want to support old versions, such as the stable one, I think it is best to write v25 maps; if you care less about that, then you can support the lastest version, but only new servers will be able to open the maps.
(Also, there is a version 26, but I don't remember the difference with version 25.)

Shouldn't he support v0.4.15 aka the newest stable

And one more. :) Minetest has negative z (vertical) coordinates if I understand correctly, which Minecraft doesn't, so that would also be a challenge. Is the water level at z == 0? When does Minetest generate new map blocks? Is it in a sphere around the player? What is the default radius in which it will generate map blocks?

Default water level is z = 0. About generating new mapblocks, it happens when the player is close enough; I think it can be configured but I don't remember the default radius. Also, the map is generated in mapchunks of 5*5*5 mapblocks (that's the default; if you produce your own maps, it is advised to set this number (chunksize or something like this IIRC) to 1, and to set mapgen to "singlenode" so that the map generation will only generate air and will node interfere with your map generation).

In MT X is Eastwards, Z is Northwards and Y is up.
(0,0,0) is world centre and the world extends 32000 nodes (we call the basic unit-cubes in the world 'nodes') in all 6 directions.
Sea level is at Y = 1.
Our 'mapblocks' are cubic, 16 * 16 * 16 nodes in size.
World is generated in a cube around the player default radius 6 * 16 = 96 nodes, but of course world is generated in batches of mapblocks called 'mapchunks', each 5 * 5 * 5 mapblocks in size, so i assume whenever the 96 node cube around the player touches a mapchunk that mapchunk is generated.

Questions, questions... The world_format.txt file describes a list of files which make up a map, in addition to the block database. Files such as auth.txt, env_meta.txt, ipban.txt, etc.

What would be the absolute minimum I could get away with for generating these files? In other words, which could I leave out because they are optional or Minetest will generate them itself if they are missing, and of the files which must be present, what content could I leave out of them which will be supplemented by Minetest?

Also, I have the basic game running, but I'm trying to compile it myself because that would make it much easier to test my understanding of the source code. I'm having trouble though. I'm compiling it with MinGW, and it gets to the minetest object file stage, but it won't link with the precompiled binaries I downloaded of the libraries, presumably because they are meant for MSVC.

Are there precompiled versions of the libraries for MinGW? What is the easiest way to build Minetest (and minetest_game) on Windows 10, for someone not in possession of Microsoft Visual Studio?

What does Minetest do if it encounters unknown node names in the map? Will it crash, ignore them, or display them as some standard block? Will it keep the unknown node, or will it remove it?

The libraries that are labeled as "pre-compiled libraries" are indeed compiled with MSVC (2010) for 32-bit MSVC builds only. I'd be surprised when one gets it working with MinGW.

Check out the buildbot script. That script and the 32-bit pendant download cross-compiled libraries for MinGW and compile the binary automatically.

Captain_Chaos wrote:What does Minetest do if it encounters unknown node names in the map? Will it crash, ignore them, or display them as some standard block? Will it keep the unknown node, or will it remove it?

In-game they get the "unknown node" texture. When you dig them, they will convert to an "unknown item", which can not be used anywhere but it's still possible to get its original name (whatever it was before it turned into an unknown thing).

Captain_Chaos wrote:In other words, which could I leave out because they are optional or Minetest will generate them itself if they are missing?

Everything is optional. I have a test case that runs a minetest server only from an empty folder. So all you'd have to do is create (on Linux) ~/.minetest/worlds/new_world/ and put stuff in there.


(runs)