Minetestmapper rewritten into C++

[Jordach]

Hey, um.. I'm not sure if it's okay to bring back an old topic, but when I run it, it turns and right back off after about a frame, or tick, or whatever you want to call it. May I have some help? I use Windows 8

same happens here. I'm windows 8 as .well

It closes instantly because the program isn't doing anything special. That's why if the mapper is fed a world directory, it will produce a overhead image.

[miner65536]

Hi,
Is there a way to get the mapper to output a fixed sized tile even if there is no content there, IE to respect the geometry option & not auto-crop to the available world content?

[LazyJ]

Old thread but still a good program... and it needs some updating and improvement TLC.

For several months I have been using this program along with the python version to generate maps of our server. I have to resort to the python version because of a bug in the cpp version when using the geometry option. When positive x and/or positive z coords are entered into the geometry option, the cpp minetestmapper generates white space from 0,0 to the specified positive coord and then it renders the area image.

Below are two sample images. The goal is to produce a 100x100 area image starting at x50,z50 and extending to x150,z150.

First the cpp minetestmapper version:

Your phone or window isn't wide enough to display the code box. If it's a phone, try rotating it to landscape mode.

Code: Select all

--geometry 50:50+100+100

Note the excess white space and the area image starting 14 farther in than specified (x64,z64)



Now the python mintestmapper version:

Your phone or window isn't wide enough to display the code box. If it's a phone, try rotating it to landscape mode.

Code: Select all

sector_xmin = 50 / 16
sector_xmax = 150 / 16
sector_zmin = 50 / 16
sector_zmax = 150 / 16


This is the image that should be rendered.


The python version of minetestmapper works but is incredibly slow and conusmes excessively large amounts of ram. I have an i7 computer with 16gb of ram running Fluxbox with all other "user" programs shut off and, on some map renders, the python version consumes all 16gb plus 6 to 8gb of swap file all to produce a miniscual 35mb file.

Yeah, that's 35 "megabytes" - a mere fraction of the 16gb of ram.

And while the python version is gobbling up all of my ram and most of my swap file, it is only using 1 of the 8 available cpu cores.

All of this plus over two hours of processing time just to produce a 35mb file!?

The cpp version of minetestmapper is so much faster, much less memory intensive and does the job wonderfully... except for the bug in the geometry.

Are there any cpp code gurus out there that can fix this bug, please?

[VanessaE]

And if someone would fix the bug where the C++ mapper will render horizontal white stripes when the world is too active (I presume it gets hung up trying read while the map is being written, and doesn't bother to wait or try the failed read again), I'd sure appreciate that. Having to shut down the server to ensure a clean map image is not fun.

[Ragnar]

And if someone would fix the bug where the C++ mapper will render horizontal white stripes when the world is too active (I presume it gets hung up trying read while the map is being written, and doesn't bother to wait or try the failed read again), I'd sure appreciate that. Having to shut down the server to ensure a clean map image is not fun.

can't you copy the map when the server is running then do that?

[VanessaE]

I could, yes, but to ensure that the copy isn't broken, it still requires shutting the server down, though for a shorter period of time.

The default python mapper doesn't exhibit this issue, but the slowness of that one is...unappealing.

[VanessaE]

Another update to my colors.txt today. Re-arranged a bit and brought many mods up-to-date, especially things that use moreblocks/stairsplus. Some stuff still needs some attention. Same download link as before.

[Sean884]

For several months I have been using this program along with the python version to generate maps of our server. I have to resort to the python version because of a bug in the cpp version when using the geometry option. When positive x and/or positive z coords are entered into the geometry option, the cpp minetestmapper generates white space from 0,0 to the specified positive coord and then it renders the area image.

<snip>

Note the excess white space and the area image starting 14 farther in than specified (x64,z64)

There's really two issues here. The excess white space between 0,0 and your desired area needs just a small fix; see the patch below. The reason the area you requested started at 64,64 rather than 50,50 is that the C++ code rounds the boundaries to the nearest block of 16x16x16 nodes for better performance. Changing it to use the exact node boundaries without sacrificing speed would NOT be a small fix. :(

Here's the patch for the excess white space issue. If you're not familiar applying patches and recompiling, I'm sorry I'm not able to help you with that at this time; hopefully someone else can.

Your phone or window isn't wide enough to display the code box. If it's a phone, try rotating it to landscape mode.

Code: Select all

diff -u a/TileGenerator.cpp b/TileGenerator.cpp
--- a/TileGenerator.cpp
+++ b/TileGenerator.cpp
@@ -9,6 +9,7 @@
 
 #include <cstdio>
 #include <cstdlib>
+#include <climits>
 #include <fstream>
 #include <gdfontmb.h>
 #include <iostream>
@@ -94,10 +95,10 @@ TileGenerator::TileGenerator():
        m_border(0),
        m_db(0),
        m_image(0),
-       m_xMin(0),
-       m_xMax(0),
-       m_zMin(0),
-       m_zMax(0),
+       m_xMin(INT_MAX),
+       m_xMax(INT_MIN),
+       m_zMin(INT_MAX),
+       m_zMax(INT_MIN),
        m_geomX(-50),
        m_geomY(-50),
        m_geomX2(50),


[Sean884]

And if someone would fix the bug where the C++ mapper will render horizontal white stripes when the world is too active (I presume it gets hung up trying read while the map is being written, and doesn't bother to wait or try the failed read again), I'd sure appreciate that. Having to shut down the server to ensure a clean map image is not fun.

You're correct, the C++ mapper has no code to handle the database being busy, which it very likely will be when the server is running. Unfortunately, the server doesn't have that code either, so if you're running the mapper (even the python version!) and the server concurrently, users of the server also have a chance of seeing missing nodes or having updates (near-)silently fail. It's possible to fix both, but if this is the only use case for doing so, I'm not sure it's worth the time?

[LazyJ]

Thanks, Sean, for the patch. It works! \0/

The 14 block offset still exists but is easily remedied by padding an extra 14 on either side of the geometry option.

So to follow up on my previous example:

If you want a map that starts from x50,z50 and goes to x100,z100 subtract 14 from the 50,50 and add 14 to the 100,100 to get

Your phone or window isn't wide enough to display the code box. If it's a phone, try rotating it to landscape mode.

Code: Select all

--geometry 36:36+114+114

[LazyJ]

Ok, another feature that would be nice (or perhaps it's a code line that can be tweaked as needed) is to specify a height or altitude to ignore things (kind of like how minetestmapper ignores the clouds).

We have players building arial platforms that are high enough not to cast shadows on the ground but are becoming broad enough that they are obscuring large sections of the map.

It would be nice for ground-level players that refer to the overview maps if there were a way to produce a set of maps without the arial platforms and then toggle the option on/off to generate maps with the arial platforms for the sky-builders.

Maybe something like:

Your phone or window isn't wide enough to display the code box. If it's a phone, try rotating it to landscape mode.

Code: Select all

--ignore y:160

[Sokomine]

It would be nice for ground-level players that refer to the overview maps if there were a way to produce a set of maps without the arial platforms and then toggle the option on/off to generate maps with the arial platforms for the sky-builders.

That would be very nice to have. All these sky-platforms make it difficult to navigate on the ground using the map.

[twoelk]

This would be a cool feature indeed and usefull for many things. Slicing at a given hight would not only allow something like hight=30 to exclude platforms above that vallue but hight=-130 would give me a layout of all my tunnels at that level (and show me all caves as well of course)

[VanessaE]

For some reason, I ran into problems again with the mapper producing "holes" in the land underneath unknown-to-the-mapper blocks, so I went to my backups, found a copy that works right, committed the last change (which I think was by ShadowNinja), and uploaded that fork here:

https://github.com/VanessaE/minetest-mapper-cpp

Hope others find it useful.

[Saom3000]

I rewrote the minetestmapper.py tool to C++. It is at least 50 times faster than original.

Your phone or window isn't wide enough to display the code box. If it's a phone, try rotating it to landscape mode.

Code: Select all

./minetest_mapper ...  6,96s user 0,16s system 99% cpu 7,181 total
python minetestmapper.py ... 531,58s user 1,66s system 99% cpu 8:55,13 total


Source code is on github (https://github.com/mireq/minetest-mapper-cpp).

Rquired dependencies are:

• libgd
• boost
• sqlite3

Commandline arguments are that same as used in minetestmapper.py. Input directory (-i) must end with / (on Linux) or with \ (Windows)

Windows binary is available at: https://github.com/mireq/minetest-mapper-cpp/downloads

how to install help me plz

[Sokomine]

With the help of ShadowNinja and VanessaE, I modified a local copy of ShadowNinjas branch of the minetestmapper with the intention of getting a smaller file size where the buildings are easier to see. Of course that map looks far less good than the "normal" one - that is intended. My goal is to get a more abstract, much smaller map that helps players locate roads and towns.

Modifications done:
1. The colors where changed so that grass, dirt and papyrus got the grass-color. Sand, desert_sand, desert_stone, cactus and dry shrub all counts as desert_sand. Stone and gravel retain their color. All other content - possibly player-generated - is drawn in red.
2. The trees turned out to be annoying. Thus, leaves, apples, trees and torches are ignored/treated as air by the mapper.
3. The shading for everythiing but the red content is turned off (the shading indicates height changes on the map).

The result can be seen here: 1.7 MB map of Redcrabs server

It would be more like a real map if the streets where visible as such.

[twoelk]

now that looks really interesting!

I have been playing with the mapper for some while now, hoping not to have to change the code but have since come to the conclusion I can not avoid it to get the results I want.

But if somebody else is allready working on it ;-)

For one I wanted to get rid of the default color table that the program keeps defaulting to. For example if I want to make a map that only shows rails I don't know how to convince it to ignore the rest. If I delete everything else in the color.txt it takes the builtin values for grass,stone,etc. Setting them another color, such as white, blocks everything underneath.

So my sollution was either to define an ignore value such as is used for air or make the program use the values described in the external color.txt file only. (as far as I can see there is no ignore value, it is simply hardcoded to ignore air. With Paramat's Moonrealms and such it might get vital to display air in contrast to vacum/void/undefined)

The other feature I was pondering on was a variable for maximum hight. So that one could for example ignore skyblocks over a certain hight or map a certain underground level.

[Sokomine]

For example if I want to make a map that only shows rails I don't know how to convince it to ignore the rest.

You can invert the exclusion of leaves/apples/etc and let it exclude anything that is not default:rail. That's pretty easy. I did it for a test with torches. Right now, all these changes are done to the source code and cannot be passed as parameters. But then, those are special maps anyway, and compiling is fast. Much faster than the actual map generation.

With Paramat's Moonrealms and such it might get vital to display air in contrast to vacum/void/undefined)

There has to be an internal distinction between air and vacuum. Try to find out the node names!

The other feature I was pondering on was a variable for maximum hight. So that one could for example ignore skyblocks over a certain hight or map a certain underground level.

Oh yes. That sounds very reasonable. All those odd sky plattforms which are not visible from the ground but still block up view have puzzled me a lot. And there have been several players wondering how to get to the space ships visible on the map of Redcrabs.

[A while later]After experimenting a bit with it, drawing for example one layer at a time (nodes at height 0-15 only, or 16-32), part of it turned out to be intresting. Plotting the areas below ground does not help much as that would require inverting what is drawn - instead of normal nodes, air would suddenly become particulary intresting. Else it's all more or less stone.

I'll attach the colors.txt file and a list of the tiny modifications to ShadowNinjas branch that might help to experiment a bit with it.

[Krock]

*BUMP*

It seems like the mapper can't handle LevelDB-worlds, could someone add the support for it?
Thanks in advance.

[sfan5]

When positive x and/or positive z coords are entered into the geometry option, the cpp minetestmapper generates white space from 0,0 to the specified positive coord and then it renders the area image.

This would be a cool feature indeed and usefull for many things. Slicing at a given hight would not only allow something like hight=30 to exclude platforms above that vallue but hight=-130 would give me a layout of all my tunnels at that level (and show me all caves as well of course)

Ok, another feature that would be nice (or perhaps it's a code line that can be tweaked as needed) is to specify a height or altitude to ignore things (kind of like how minetestmapper ignores the clouds).

3. The shading for everythiing but the red content is turned off (the shading indicates height changes on the map).

It seems like the mapper can't handle LevelDB-worlds, could someone add the support for it?

I have implemented the features that were requested
Here's my fork: https://github.com/sfan5/minetest-mapper-cpp

[spillz]

Why doesn't the c++ code use the existing engine code? E.g. database*.(cpp|h)

[sfan5]

Why doesn't the c++ code use the existing engine code? E.g. database*.(cpp|h)

That would require me to include many Minetest headers too..

Random news: I added LevelDB support, but some optiomisations to the code wouldn't hurt.

[sfan5]

I just made a better (automatically generateable) colors.txt for my fork.
Even though the colors.txt was automatically generated using a (pretty primitive) algoritm they still look nice.
Here's a comparison:

[LazyJ]

I've been trying sfan5's fork of MinetestMapper C++.

The reuslts, so far, have been mixed but encouraging.



--backend sqlite3
I tried generating updates to the overview maps that I post in our Minetest.net Forums', LinuxGaming.us server thread. Our server is using the default, sqlite3 backend.

The four, 1,000x1,000 block, 'section' maps take about 2 minutes each when I have used Mireq's or, at present, VanessaE's MinetestMapper C++ fork. sfan5's version took about 8 minutes to produce each 1,000x1,000 section map.

The 'Main Section' overview map is 3,300x2,400. It usually takes about 5 minutes to render. When I tried to render it with sfan5's fork it took over 3 hours.



--backend leveldb
At first I kept getting 'Unknown map backend: leveldb' error messages.

The included README.rst file instructs to compile the minetestmapper by using:

Your phone or window isn't wide enough to display the code box. If it's a phone, try rotating it to landscape mode.

Code: Select all

cmake .

make


To get the '--backend leveldb' to work I had to recompile minetestapper with:

Your phone or window isn't wide enough to display the code box. If it's a phone, try rotating it to landscape mode.

Code: Select all

cmake . -DENABLE_LEVELDB=1

make


A 1,000x1,000, overview map, of a leveldb backend world, took about 5 minutes.

A 2,000x2,000, overview map, of the same leveldb backend world, took about 16 minutes.


Hopefully some of this info will help improve future updates to the minestmapper.

Thanks to Mireq, Sean884, Sokomine, VanessaE and sfan5 for progressing this vital Minetest tool. ;)

[sfan5]

sfan5's version took about 8 minutes to produce each 1,000x1,000 section map.

Could you try reverting https://github.com/sfan5/minetest-mapper-cpp/commit/5bf2039bbb0d1f27141f7962125cc3807659e056 and then trying again. my mapper should not be slower than mireq's or VanessaE's.

Edit: I have found out why mireq's/VanessaE's mapper is faster:

(It does not generate the full map)

[Enke]

Pretty.....

*drools

[Argos]

I added a few patches on top of sfan5's version:

• Fixed the computation of block numbers to be included in the map (off-by-one)
• Failure to open output file is now caught (instead of causing SEGV)
• Fixed --min-y and --max-y (was broken)
• Added option --verbose which:
  
  • reports world and map statistics and coordinate limits
  • reports database access statistics
• Added option --forcegeometry, which creates a map of the requested geometry, even if the world is smaller
  (useful if the map is larger than minetestmapper will handle: generate the map in sections, with --forcegeometry, and glue those together)
• Fixed a memory leak in the sqlite database code (which was also a minor performance issue)
• Improved database performance:
  
  • by default, only blocks that are needed, are requested from the database now.
    leveldb should benefit greatly (I expect)
    as should sqlite when generating partial maps
  • bulk reading of one world row at a time is still an option for sqlite.
    I suspect it might be faster in some cases, depending on the map and on which part of it is being rendered.

Code on github

Miscellaneous info:

I tested my changes with an sqlite database on a system equipped with an SSD.
I'd like to hear about the performance effects when using an HDD.

I haven't gotten around to testing using a leveldb world yet. (feel free to beat me to it :-)

I noticed the code contains two licenses (LGPL 2.1 & BSD-2), without being clear about their respective statuses. That's something to be sorted out / clarified...

[sfan5]

I added a few patches on top of sfan5's version:[...]

I have merged some of them for now, I'll some more later.
Thanks for your contribution

[Amaz]

sfan5, I've been trying to use your auto generate colors, how do I use the avgcolors.py? When I run it like python avgcolors.py, it outputs this: Usage: avgcolor.py <input> So I'm presuming that there is something that I need to add after the avgcolors.py? I'm using Ubuntu. And where should the path $(find /minetest/dir -type f -name $2) in the command file lead to?
Thanks!

[sfan5]

I've clarified the instructions a bit.