Page 1 of 7

Minetestmapper rewritten into C++

PostPosted: Wed Aug 29, 2012 06:08
by mireq
There is now an official repository for this: https://github.com/minetest/minetestmapper

I rewrote the minetestmapper.py tool to C++. It is at least 50 times faster than original.

Your phone or window isn't wide enough to display the code box. If it's a phone, try rotating it to landscape mode.
Code: Select all
./minetest_mapper ...  6,96s user 0,16s system 99% cpu 7,181 total
python minetestmapper.py ... 531,58s user 1,66s system 99% cpu 8:55,13 total


Source code is on github (https://github.com/mireq/minetest-mapper-cpp).

Rquired dependencies are:
  • libgd
  • boost
  • sqlite3

Commandline arguments are the same as used in minetestmapper.py

PostPosted: Wed Aug 29, 2012 16:17
by tonyka
such as do to compile?
there is no readme or install file...
I use linux ...
thanks

PostPosted: Wed Aug 29, 2012 16:21
by PilzAdam
tonyka wrote:such as do to compile?
there is no readme or install file...
I use linux ...
thanks

Your phone or window isn't wide enough to display the code box. If it's a phone, try rotating it to landscape mode.
Code: Select all
$ cd /the/directory/of/minetestmapper/
$ cmake .
$ make .

PostPosted: Wed Aug 29, 2012 18:42
by tonyka
not working
I do not know what I'm doing wrong ...
:(

Your phone or window isn't wide enough to display the code box. If it's a phone, try rotating it to landscape mode.
Code: Select all
tony-ka@tonyka-HP-Compaq-dc7600-Small-Form-Factor:~/Escritorio/Minetest_compilado/minetest-mapper/mireq-minetest-mapper-cpp-5a2375e$ cmake .
-- The C compiler identification is GNU
-- The CXX compiler identification is GNU
-- Check for working C compiler: /usr/bin/gcc
-- Check for working C compiler: /usr/bin/gcc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Found LibSqlite3: /usr/lib/x86_64-linux-gnu/libsqlite3.so
-- Boost version: 1.46.1
-- Found the following Boost libraries:
--   system
--   filesystem
-- Configuring done
-- Generating done
-- Build files have been written to: /home/tony-ka/Escritorio/Minetest_compilado/minetest-mapper/mireq-minetest-mapper-cpp-5a2375e
tony-ka@tonyka-HP-Compaq-dc7600-Small-Form-Factor:~/Escritorio/Minetest_compilado/minetest-mapper/mireq-minetest-mapper-cpp-5a2375e$ make .
tony-ka@tonyka-HP-Compaq-dc7600-Small-Form-Factor:~/Escritorio/Minetest_compilado/minetest-mapper/mireq-minetest-mapper-cpp-5a2375e$

PostPosted: Wed Aug 29, 2012 18:52
by Menche
Don't use the "." after "make".

Also, to speed up a large build, use the "-j#" option, where # is the number of cores + 1. My machine has 4 cores so I use "make -j5". To find the number of cores you have, run "grep -c processor /proc/cpuinfo".

PostPosted: Wed Aug 29, 2012 21:08
by Phitherek_
woah, and c55 added -j2 option permanently in the README for the engine on GitHub... I think he should explain this better...

EDIT: All right, in the -j# the # is number of THREADS and not CORES. So -j2 is for two-threaded one core processor. Menche, you can switch to -j8 then ;).

PostPosted: Thu Aug 30, 2012 01:33
by RabbiBob
Nice work, the Win binary worked right off the mark. I noted that I had an odd issue with the .py that didn't happen in the .exe:

.exe on the left, no odd marks like on the right.

Image

Would you be interesting in adding mod color schemes to the colors.txt (if so, see here)?

PostPosted: Thu Aug 30, 2012 05:50
by mireq
@tonyka: Readme is really missing. README.md will be added in the coming days.

@Menche: In this case, the differences are small:
Your phone or window isn't wide enough to display the code box. If it's a phone, try rotating it to landscape mode.
Code: Select all
make -j 1  7,07s user 0,79s system 98% cpu 7,991 total
make -j 2  7,08s user 0,84s system 121% cpu 6,506 total
make -j 3  7,15s user 0,87s system 135% cpu 5,918 total


@RabbiBob: added, thanks.

PostPosted: Thu Aug 30, 2012 11:08
by Jordach
Can we have some arguments please?

Here is the contents:

Your phone or window isn't wide enough to display the code box. If it's a phone, try rotating it to landscape mode.
Code: Select all
minetest_mapper.exe -i C:\Documents and Settings\Jordan\Desktop\Minetest 0.4.2\worlds\city\ -o output_file.png


And the world SHOULD work. But it does not generate any .png anywhere.

PostPosted: Thu Aug 30, 2012 14:42
by tonyka
thank you all, but I have not managed to compile ...
compile and tells me not find GD.h
according to forums related to GD, gives problems in ubuntu, but can not find the solution...

PostPosted: Thu Aug 30, 2012 19:55
by Nemo08
tonyka wrote:thank you all, but I have not managed to compile ...
compile and tells me not find GD.h
according to forums related to GD, gives problems in ubuntu, but can not find the solution...

Your phone or window isn't wide enough to display the code box. If it's a phone, try rotating it to landscape mode.
Code: Select all
apt-get install libgd2-xpm-dev libboost-all-dev libgd-tools

for 12.04

PostPosted: Thu Aug 30, 2012 20:52
by tonyka
hey thanks ...
has replaced some libraries and ...
It works!
wondering, you get to Rome!

PostPosted: Thu Aug 30, 2012 22:24
by RabbiBob
Jordach wrote:Can we have some arguments please?

Here is the contents:

Your phone or window isn't wide enough to display the code box. If it's a phone, try rotating it to landscape mode.
Code: Select all
minetest_mapper.exe -i C:\Documents and Settings\Jordan\Desktop\Minetest 0.4.2\worlds\city\ -o output_file.png


And the world SHOULD work. But it does not generate any .png anywhere.


Try one of two things:

Add " around the path

Your phone or window isn't wide enough to display the code box. If it's a phone, try rotating it to landscape mode.
Code: Select all
minetest_mapper.exe -i "C:\Documents and Settings\Jordan\Desktop\Minetest 0.4.2\worlds\city" -o output_file.png


or

  • Create a util directory at C:\Documents and Settings\Jordan\Desktop\Minetest 0.4.2\
  • Put minetest_mapper.exe and color.txt in C:\Documents and Settings\Jordan\Desktop\Minetest 0.4.2\util\
  • Create map.bat and put the following in it:
    • minetest_mapper.exe -i ../worlds/city/ -o output.png
  • Save it and run the bat file

The latter way gives you a quick double click way of running the mapper.

PostPosted: Thu Aug 30, 2012 22:52
by Jordach
Thanks RabbiBob.

PostPosted: Fri Aug 31, 2012 12:00
by tonyka
wow is fast and furious :D
is like comparing a moped with a Bugatti Veyron 8.0 W16 Super Sport 1200 hp

PostPosted: Fri Aug 31, 2012 15:20
by Jordach
Here is my colour sheet so far;

Your phone or window isn't wide enough to display the code box. If it's a phone, try rotating it to landscape mode.
Code: Select all
default:stone 128 128 128
default:stone_with_coal 50 50 50
default:water_flowing 39 66 106
default:torch 255 255 0
default:water_source 39 66 106
default:sign_wall 117 86 41
default:chest 128 79 0
default:furnace 118 118 118
default:fence_wood 103 78 42
default:rail 162 119 53
default:ladder 154 110 40
default:lava_flowing 255 100 0
default:lava_source 255 100 0
default:dirt_with_grass 107 134 51
default:tree 86 58 31
default:leaves 48 95 8
default:dirt_with_grass_and_footsteps 102 129 38
default:mese 178 178 0
default:dirt 101 84 36
default:wood 104 78 42
default:sand 210 194 156
default:cobble 123 123 123
default:steelblock 199 199 199
default:glass 183 183 222
default:mossycobble 219 202 178
default:gravel 70 70 70
default:sandstone 204 0 0
default:cactus 0 215 0
default:brick 170 50 25
default:clay 104 78 42
default:papyrus 58 105 18
default:bookshelf 196 160 0
default:jungletree 205 190 121
default:junglegrass 62 101 25
default:nyancat 255 153 255
default:nyancat_rainbow 102 50 255
default:apple 200 0 0
default:desert_sand 210 194 156
default:desert_stone 210 194 156
default:dry_shrub 100 80 40

moreores:gold_block 255 255 51
moreores:mithril_block 51 204 255

moreblocks:junglewood 255 204 153
moreblocks:stonebrick 128 128 128
moreblocks:circlestonebrick 128 128 128
moreblocks:ironstonebrick 180 180 180
moreblocks:stonesquare 128 128 128
moreblocks:splitstonesquare 128 128 128
moreblocks:plankstone 128 128 128
moreblocks:ironglass 120 120 120
moreblocks:coalglass 70 70 70
moreblocks:cleanglass 183 183 222
moreblocks:cactusbrick 0 215 0
moreblocks:cactuschecker 0 215 0
moreblocks:emptybookshelf 196 160 0
moreblocks:oerkkiblock 153 51 204
moreblocks:coalstone 90 90 90
moreblocks:ironstone 120 120 120
moreblocks:coalchecker 120 120 120
moreblocks:ironchecker 170 170 170
moreblocks:trapstone 128 128 128
moreblocks:trapglass 183 183 222
moreblocks:fence_junglewood 255 204 153
moreblocks:horizontaltree 48 95 8
moreblocks:horizontaljungletree 48 95 8
moreblocks:allfacestree 48 95 8
moreblocks:glowglass 240 240 0
moreblocks:superglowglass 240 240 0
moreblocks:rope 0 240 0

stairs:stair_wood 196 160 0
stairs:slab_wood 196 160 0
stairs:stair_stone 128 128 128
stairs:slab_stone 128 128 128
stairs:stair_cobble 128 128 128
stairs:slab_cobble 128 128 128
stairs:stair_brick 170 50 25
stairs:slab_brick 170 50 25
stairs:stair_sandstone 204 0 0
stairs:slab_sandstone 204 0 0
stairs:stair_steelblock 190 190 190
stairs:slab_steelblock 190 190 190
stairs:stair_gold 255 255 51
stairs:slab_gold 255 255 51
stairs:stair_diamond 51 204 255
stairs:slab_diamond 51 204 255

glowstone:glowstone 255 204 0

xfences:fence 103 78 42
xfences:fence_1 103 78 42
xfences:fence_2 103 78 42
xfences:fence_3 103 78 42
xfences:fence_4 103 78 42
xfences:fence_5 103 78 42
xfences:fence_6 103 78 42
xfences:fence_7 103 78 42
xfences:fence_8 103 78 42
xfences:fence_9 103 78 42
xfences:fence_10 103 78 42
xfences:fence_11 103 78 42
xfences:fence_12 103 78 42
xfences:fence_13 103 78 42
xfences:fence_14 103 78 42
xfences:fence_15 103 78 42


Note that steelblock, gold and diamond stairs are only in my own gamemode. And half of moreores is supoorted.

PostPosted: Fri Aug 31, 2012 23:38
by mrtux
This should be included with Minetest's Windows bianaries. +1000

PostPosted: Sat Oct 06, 2012 18:43
by nomohakon
If it only can create heightmaps... can it?

EDIT: greyscale heightmaps.

One question. How to add new blocks to mapping process, ie gloopblocks:cement.

PostPosted: Fri Nov 09, 2012 14:21
by LazyJ
I haven't been able to get a rendering of the full map.

Both mappers (python and c++) only generate an area from (+/-) 1472 x (+/-) 1472. I used the "--drawscale" option to confirm this range.

I've been to two, opposite far corners (-30000 to +30000) and built stuff just to make sure something is registered in the world and yet the mapper programs won't extend to that range.

I've checked the "--help" for clues to see if there was some sort of range option or limiter... no luck.

-i/--input <world_path>
-o/--output <output_image.png>
--bgcolor <color>
--scalecolor <color>
--playercolor <color>
--origincolor <color>
--drawscale
--drawplayers
--draworigin
Color format: '#000000'

Any ideas?

PostPosted: Sat Nov 10, 2012 15:00
by LazyJ
*bump*

PostPosted: Thu Nov 15, 2012 15:33
by mauvebic
where is the default background color set? id like to change it to black i keep forgetting the cmd line argument lol

PostPosted: Thu Nov 15, 2012 17:43
by LazyJ
mauvebic wrote:where is the default background color set? id like to change it to black i keep forgetting the cmd line argument lol


This is from the minetestmapper "--help" from both the python and c++ versions:

-i/--input <world_path>
-o/--output <output_image.png>
--bgcolor <color>
--scalecolor <color>
--playercolor <color>
--origincolor <color>
--drawscale
--drawplayers
--draworigin
Color format: '#000000'


Perhaps "--bgcolor #000000" is what you are looking for? I haven't tried changing the colors yet to know what happens.

PostPosted: Thu Nov 15, 2012 19:03
by mauvebic
i meant change the default in the source code so i dont have to use that argument - cant find what line the defaults' on :p

PostPosted: Tue Nov 20, 2012 03:51
by LazyJ
*bump*

Anyway to expand the minetestmapper's ability to create an image of the full world map (30000+)?

PostPosted: Fri Nov 23, 2012 21:59
by LazyJ
*bump*

PostPosted: Sat Nov 24, 2012 18:41
by mireq
Removed hardcoded size of image and added --geometry x:y+w+h option. New sources and binary are on github (https://github.com/mireq/minetest-mapper-cpp).

PostPosted: Wed Nov 28, 2012 06:06
by LazyJ
Ok, some more kitchen-testing with the latest version from Github (as of 2012_11_27 morning), Minetest v 0.4.3 and Ubuntu 12.04, 4gig ram

I played around with the "--geometry" option and here are some of the results:

--geometry -30000:-30000+100+100
Result: "Segmentation fault (core dumped)". Yes, there are some buildings out at -30,000x,-30,000y.

--geometry -20000:-20000+100+100
Result: About 15 to 20 minutes to produce a 33 byte (yup, 3-3-b-y-t-e) file that wouldn't load into Eye-of-Gnome (eog) nor GIMP.

--geometry -2000:-2000+4000+4000
Result: A 6.7mb file covering a 4000 x 4000 area starting in the lower left corner at -2000x, -2000y

--geometry -10000:-10000+20000+20000
Results: A 20.2mb file covering a 20,000 x 20,000 area staring in the lower left corner at -10,000x, -10,000y

I've noticed that the minetestmapper's "y" is different than Minetest's coordinates "y". In Minetest, the "y" is the vertical (heaven to hell, up and down) where in the minetestmapper, the "y" is the north and south, forward and reverse (like they taught me in geometry class).

The file size seems quite large for a flat, 2D map. I see that each node is already reduced to 1 pixel each. If rendering the full 60,000x60,000 map becomes possible, what other options could be exercised to reduce the file size? Greyscale instead of RGB? Lower resolution? Output format *.jpg instead of *.png? What about splitting the image into sections that could later be knitted together in a photo editor?

I hope this information is usefull in the further develpment of this nifty program.

PostPosted: Wed Nov 28, 2012 12:17
by VanessaE
Program bugs and memory usage aside, about all you can to do shrink the file size further without losing important information is to use 8-bit color instead of 24-bit, assuming it doesn't already do that, use fewer color transitions (e.g. lose the shadows that give hills and tall objects some definition), don't save any metadata, and use maximum PNG compression on the file. You could also try cutting out all parts of the map that haven't been built on, fill those areas with solid white similar to how unexplored regions are drawn. That'll cut the size down some. Large swaths of solid colors compress better than regular map details.

You could indeed try converting such a map to JPG, which will probably cut the file size considerably, but it'll likely suffer horrible quality loss even at high quality settings.

You could also try scaling the map down 2:1 or maybe 3:1, but of course that will result in blurred details.

The best solution is probably to only export the portions of the map that have been built up, and stitch them together on a web page with thumbnails to make a clickable, zoomable map similar to the one ruberwardy maintains for redcrab's server.

Some of this has already been tried as you already know, and the results weren't necessarily all that great.

PostPosted: Wed Nov 28, 2012 21:22
by Iqualfragile
is ruberwardys script avivable to the public?

and try using optipng, it can reduce the filesize to one third

PostPosted: Fri Nov 30, 2012 07:34
by davidpace
I dunno.........
Iqualfragile wrote:is ruberwardys script avivable to the public?

and try using optipng, it can reduce the filesize to one third