[Mod] Real Terrain [0.1.0] [realterrain]

[bobomb]

0.0.5 pushed to github. many new features and recently incorporated the development of raster analysis functions formerly part of my GIS mod: viewtopic.php?f=9&t=12928

[bobomb]

https://github.com/bobombolo/realterrain/tree/usgs

a recent branch of the mod that uses Lunatic-Python to run python inside lua... nothing major but something python users might be interested in for mod-making...

[Fixerol]

I'm very excited about this mod, it will be cool to play minetest on real world terrain. I myself made a map of Ukraine for OpenTTD from some of the DEM files... (greyscale ones). Is it possible to make this mod as separate program too?

[bobomb]

this mod is only a mod to minetest and in fact requires some configuration of external libraries which can be a challenge. please watch the video I have prepared to see how it works. you will need to install imagemagick and magickwand and have mod security disabled. https://www.youtube.com/watch?v=66pehrH6Bh0

[bobomb]

for those asking for trees, and to see the biome editor results:

[bobomb]

introducing image overlay and band overlay mode which allows you to map a color image onto an elevation map or even map separate bands as red green and blue onto an elevation map (this allows for the display of false-color landsat7 and other satellite imagery)



the color output is limited to 216 "web-safe" colors at this time

[bobomb]

use realterrain as a mandelbrot exporer or a polynomial explorer:

[bobomb]

also there is now a default mode that requires no libraries and uses 8-bit color BMP files (only the red channel is used where grayscale rasters are expected such as for elevation and biomes, but color is used for color overlays -- unlike more advanced modes which can handle grayscale images with only one channel, this bmp mode crashes on grayscale images, but at least those wishing to test the system out without installing any libraries now have a way to use this feature-rich mod)

[bobomb]

some new features: supports graphicsmagick in CLI or Lua lib mode (this is the best lib to use actually). Automatic bit-depth detection implemented wherever possible. Command-line ImageMagick and GraphicsMagick mode added so that no lua libraries are required (for RGB BMPs and imperfectly for grayscale TIFs, and for very small PNGs), improvements to scaling, surface-only modes, surface detection, and distance options implemented.



anyone familiar with the tiff format could contribute by getting the native tiff processor working... or anyone that knows of pure lua libraries that handle various image formats. image libraries that handle 16 bit pixel values, and lua libraries that grant access to individual pixels and pixel groups without loading the entire image into memory are needed.

[Gael de Sailly]

I've downloaded a tile from SRTM Data Search (tile #38-03) and finally managed to get it working with realterrain. I use the magick processor.
Scale: 1/93 horizontal and 1/40 vertical.
Some screenshots with their real location:

The Vosges in France

Lac d'Annecy in the Alps (please imagine the water)

Chamonix and the Mont Blanc

[twoelk]

wow
Heh nice to see pics of places I know :-)

I once was caught in a massive thunderstorm on the shores of Lac d'Annecy and travelled the at some places rather dramatic approach to the Mont Blanc Tunnel near Chamonix many a times.

So, do the valleys of the Ardeche and Verdon look as impressive at this scale as they do in reallity?

edit: oops, guess they are too far south

[Gael de Sailly]

So, do the valleys of the Ardeche and Verdon look as impressive at this scale as they do in reallity?

edit: oops, guess they are too far south

Yes, that's in the tile below, but I have also tried this one and found the Verdon. The SRTM data aren't accurate enough to generate correctly the canyon (maybe the radar used by SRTM can't scan the depths of the gorges).

That may be better to change the scale to have 40-50 real meters per block, but realterrain can't draw it properly because the position is rounded to the nearest pixel instead of calculating the weighted average of the 4 nearest pixels.

[bobomb]

smoothing when scaling may be in a future release. what is the horizontal resolution of the SRTM original?

creating a cover file that sets the lake surface to pixel value to 55 should be relatively easy. roads to pixel value 15 might be nice to show where the highway runs. how old is your version of realterrain?

[Gael de Sailly]

smoothing when scaling may be in a future release. what is the horizontal resolution of the SRTM original?

There is 6000 px for 5°, for latitude and longitude.
1 px = 93 m (latitude)
For the longitude, it's distorted on high latitudes, so it's 93 m at the equator and 61 m at 49°N.
simply 93×cos(lat)

The Verdon Canyon

[emperor_genshin]

+1 This is indeed interesting, great job bobomb :)

[twoelk]

...
The Verdon Canyon
... some pics ...

hmm looking at the pics makes me wonder why I spent two days crossing them tiny hills of a few blocks hight from the coast via Draguignan to the Verdon Canyon on bicycle. A tour of really spectactuIar sights I will never forget. Just as the endless fighting the mistral as I followed the Durance to Avignon.

Actually recreating the Artuby bridge in minetest might be fun - or the stone arc in the Ardeche valley. Hmm might need some painted blocks for the Chauvet cave though ;-P

[jp]

Just as the endless fighting the mistral as I followed the Durance to Avignon.

Haha, I'm glad to see someone is knowning such details of my region, I didn't expect that here. I often walk along the Luberon, and I fighted a lot with the mistral in bicycle as well. If you return some day in this corner, I'd be glad to meet another Minetester ;)

[Fixerol]

Nice! How big are those (in nodes)?

[Gael de Sailly]

I can't load the libraries.
Native processor works.

+ Logs

py processor:

Your phone or window isn't wide enough to display the code box. If it's a phone, try rotating it to landscape mode.

Code: Select all

2016-02-09 14:18:14: [Main]: Automatically selecting world at [/home/gael/.minetest/worlds/bidule]
2016-02-09 14:18:15: WARNING[Main]: BanManager: creating /home/gael/.minetest/worlds/bidule/ipban.txt
2016-02-09 14:18:15: WARNING[Main]: NodeDefManager: Ignoring CONTENT_IGNORE redefinition
2016-02-09 14:18:15: ERROR[Main]: ModError: Failed to load and run script from /home/gael/.minetest/mods/realterrain/init.lua:
2016-02-09 14:18:15: ERROR[Main]: ...s/realterrain/lib/lunatic-python-bugfix-1.1.1/python.lua:3: /usr/lib/python2.7/site-packages/lua-python.so: Ne peut ouvrir le fichier d'objet partagé: Aucun fichier ou dossier de ce type
2016-02-09 14:18:15: ERROR[Main]: stack traceback:
2016-02-09 14:18:15: ERROR[Main]:    [C]: in function 'assert'
2016-02-09 14:18:15: ERROR[Main]:    ...s/realterrain/lib/lunatic-python-bugfix-1.1.1/python.lua:3: in main chunk
2016-02-09 14:18:15: ERROR[Main]:    [C]: in function 'require'
2016-02-09 14:18:15: ERROR[Main]:    /home/gael/.minetest/mods/realterrain/init.lua:28: in main chunk
2016-02-09 14:18:15: ERROR[Main]: Voir debug.txt pour plus d'informations.

gm processor:

Your phone or window isn't wide enough to display the code box. If it's a phone, try rotating it to landscape mode.

Code: Select all

2016-02-09 14:18:35: [Main]: Automatically selecting world at [/home/gael/.minetest/worlds/bidule]
2016-02-09 14:18:37: WARNING[Main]: BanManager: creating /home/gael/.minetest/worlds/bidule/ipban.txt
2016-02-09 14:18:37: WARNING[Main]: NodeDefManager: Ignoring CONTENT_IGNORE redefinition
2016-02-09 14:18:37: ERROR[Main]: ModError: Failed to load and run script from /home/gael/.minetest/mods/realterrain/init.lua:
2016-02-09 14:18:37: ERROR[Main]: .../.minetest/mods/realterrain/lib/graphicsmagick/Image.lua:304: libGraphicsMagickWand.so: Ne peut ouvrir le fichier d'objet partagé: Aucun fichier ou dossier de ce type
2016-02-09 14:18:37: ERROR[Main]: stack traceback:
2016-02-09 14:18:37: ERROR[Main]:    [C]: in function 'load'
2016-02-09 14:18:37: ERROR[Main]:    .../.minetest/mods/realterrain/lib/graphicsmagick/Image.lua:304: in main chunk
2016-02-09 14:18:37: ERROR[Main]:    [C]: in function 'require'
2016-02-09 14:18:37: ERROR[Main]:    ...l/.minetest/mods/realterrain/lib/graphicsmagick/init.lua:2: in main chunk
2016-02-09 14:18:37: ERROR[Main]:    [C]: in function 'require'
2016-02-09 14:18:37: ERROR[Main]:    /home/gael/.minetest/mods/realterrain/init.lua:43: in main chunk
2016-02-09 14:18:37: ERROR[Main]: Voir debug.txt pour plus d'informations.

magick processor:

Your phone or window isn't wide enough to display the code box. If it's a phone, try rotating it to landscape mode.

Code: Select all

2016-02-09 14:18:58: [Main]: Automatically selecting world at [/home/gael/.minetest/worlds/bidule]
2016-02-09 14:19:04: WARNING[Main]: BanManager: creating /home/gael/.minetest/worlds/bidule/ipban.txt
2016-02-09 14:19:04: WARNING[Main]: NodeDefManager: Ignoring CONTENT_IGNORE redefinition
2016-02-09 14:19:04: ERROR[Main]: ModError: Failed to load and run script from /home/gael/.minetest/mods/realterrain/init.lua:
2016-02-09 14:19:04: ERROR[Main]: ...el/.minetest/mods/realterrain/lib/magick/magick/init.lua:163: Failed to load ImageMagick (MagickWand)
2016-02-09 14:19:04: ERROR[Main]: stack traceback:
2016-02-09 14:19:04: ERROR[Main]:    [C]: in function 'try_to_load'
2016-02-09 14:19:04: ERROR[Main]:    ...el/.minetest/mods/realterrain/lib/magick/magick/init.lua:163: in main chunk
2016-02-09 14:19:04: ERROR[Main]:    [C]: at 0x7f992b68c9e0
2016-02-09 14:19:04: ERROR[Main]:    [C]: in function 'require'
2016-02-09 14:19:04: ERROR[Main]:    /home/gael/.minetest/mods/realterrain/init.lua:36: in main chunk
2016-02-09 14:19:04: ERROR[Main]: Voir debug.txt pour plus d'informations.

imlib2 processor:

Your phone or window isn't wide enough to display the code box. If it's a phone, try rotating it to landscape mode.

Code: Select all

2016-02-09 14:19:18: [Main]: Automatically selecting world at [/home/gael/.minetest/worlds/bidule]
2016-02-09 14:19:20: WARNING[Main]: BanManager: creating /home/gael/.minetest/worlds/bidule/ipban.txt
2016-02-09 14:19:20: WARNING[Main]: NodeDefManager: Ignoring CONTENT_IGNORE redefinition
2016-02-09 14:19:20: ERROR[Main]: ModError: Failed to load and run script from /home/gael/.minetest/mods/realterrain/init.lua:
2016-02-09 14:19:20: ERROR[Main]: ...ael/.minetest/mods/realterrain/lib/lua-imlib2/imlib2.lua:1: module 'limlib2' not found:
2016-02-09 14:19:20: ERROR[Main]:    no field package.preload['limlib2']
2016-02-09 14:19:20: ERROR[Main]:    no file '/home/gael/.minetest/mods/realterrain/lib/lua-imlib2/limlib2.lua'
2016-02-09 14:19:20: ERROR[Main]:    no file '/home/gael/.minetest/mods/realterrain/lib/lua-imagesize-1.2/limlib2.lua'
2016-02-09 14:19:20: ERROR[Main]:    no file './limlib2.lua'
2016-02-09 14:19:20: ERROR[Main]:    no file '/usr/share/luajit-2.0.2/limlib2.lua'
2016-02-09 14:19:20: ERROR[Main]:    no file '/usr/local/share/lua/5.1/limlib2.lua'
2016-02-09 14:19:20: ERROR[Main]:    no file '/usr/local/share/lua/5.1/limlib2/init.lua'
2016-02-09 14:19:20: ERROR[Main]:    no file '/usr/share/lua/5.1/limlib2.lua'
2016-02-09 14:19:20: ERROR[Main]:    no file '/usr/share/lua/5.1/limlib2/init.lua'
2016-02-09 14:19:20: ERROR[Main]:    no file './limlib2.so'
2016-02-09 14:19:20: ERROR[Main]:    no file '/usr/local/lib/lua/5.1/limlib2.so'
2016-02-09 14:19:20: ERROR[Main]:    no file '/usr/lib/x86_64-linux-gnu/lua/5.1/limlib2.so'
2016-02-09 14:19:20: ERROR[Main]:    no file '/usr/lib/lua/5.1/limlib2.so'
2016-02-09 14:19:20: ERROR[Main]:    no file '/usr/local/lib/lua/5.1/loadall.so'
2016-02-09 14:19:20: ERROR[Main]: stack traceback:
2016-02-09 14:19:20: ERROR[Main]:    [C]: in function 'require'
2016-02-09 14:19:20: ERROR[Main]:    ...ael/.minetest/mods/realterrain/lib/lua-imlib2/imlib2.lua:1: in main chunk
2016-02-09 14:19:20: ERROR[Main]:    [C]: in function 'require'
2016-02-09 14:19:20: ERROR[Main]:    /home/gael/.minetest/mods/realterrain/init.lua:40: in main chunk
2016-02-09 14:19:20: ERROR[Main]: Voir debug.txt pour plus d'informations.

The libraries works on my other computer: I use the magick processor. Both with Ubuntu 14.04, Minetest 0.4.13-dev.

Including libraries this way is very unclean. Libs should be provided by MT's core itself. The core uses an image manipulation library, it may be possible to allow Lua accessing these functions.

[bobomb]

yes it would change a lot of things for the better to not have to include any libraries for basic pixel-level access to images... including full 16 bit support and all common filetypes. i have discussed this in the irc but there is not much interest in including it in core.

[Gael de Sailly]

That's especially for the image library. The core uses an image library. It could simply (I know that technically, it may not be that simple) provide the functions to the Lua interface, no need to add another library to the core.

[bobomb]

yes that's what I mean. should we ask the core developers to include pixel-access to images as a lua method?

[Gael de Sailly]

I've sent a pull request to realterrain some weeks ago : PR 97.

For the image library, yes we should ask the devs. That's the moment : they're very active and make very good things these days. We should open an issue on minetest/minetest.

[Gael de Sailly]

I have another idea:

making a separate python script that converts the image into raw data that Lua can easily use. Run the script before starting Minetest.
With this method, the raw data can even been split up into many different tiles, each of them matching a… maybe 200x200 pixels, hugely reducing the amount of used RAM (unload a tile when it has not been used during 5 consecutive chunk generations for example). We can even use zlib compression, performed by python, and decompressed by the Minetest API:

Your phone or window isn't wide enough to display the code box. If it's a phone, try rotating it to landscape mode.

Code: Select all

minetest.decompress(data)

I mean, in brief, what has been done for the Moon Mapgen mod by arpruss.

[bobomb]

there are issues with most of the image processor modes under windows, but if you had imagemagick working in the past then changing the MAGICK_AS_CONVERT = false on init.lua line 40 should get you working again.

[bobomb]

as for image functions in core, this is the answer (no):

viewtopic.php?f=7&t=14164

[bobomb]

actually celeron55 says yes: viewtopic.php?f=7&t=14164#p210309

anybody want to take a crack at it?

[bobomb]

I have another idea:

making a separate python script that converts the image into raw data that Lua can easily use. Run the script before starting Minetest.
With this method, the raw data can even been split up into many different tiles, each of them matching a… maybe 200x200 pixels, hugely reducing the amount of used RAM (unload a tile when it has not been used during 5 consecutive chunk generations for example). We can even use zlib compression, performed by python, and decompressed by the Minetest API:

Your phone or window isn't wide enough to display the code box. If it's a phone, try rotating it to landscape mode.

Code: Select all

minetest.decompress(data)

I mean, in brief, what has been done for the Moon Mapgen mod by arpruss.

I have considered this approach but am not, personally, interested in it. The current method, with most of the (working) libraries does not load the entire image into memory anyway (the native png processor lifted from imageloader mod is an exception and crashes on even mid-sized images -- lua image libraries that load the entire image into memory were not considered for this mod. the imagemagick and graphicsmagic libs read the entire file and create a disk cache of it but at least do not load it into memory).
Mainly though I don't want to create any temp files because many of the source files of DEMs are huge and because I want them to be usable without any extra processing steps, otherwise we might as well convert DEM formats to ESRI's GRID (text-based) format. However, I envision a tiling system for version 0.2.0 of the mod which would allow any kind of tiles to be used, whether they be multiple contiguous tiles from a DEM set or from a python script as you say.
the mod already performs "caching" of sorts and only looks up pixels for map chunk "footprints" once in a vertical column -- it knows when a sky or sub-surface chunk is being rendered and does no pixel lookups...
I also experimented with cropping the image into smaller images for efficiency (not 200x200 but mapchunk size of 80x80 and multiples thereof) but with large source DEMs (which are the ones we'd want to do this with) re-opening the source file for cropping took several seconds each time. certain processors actually do this out of necessity. gm and convert for example -- which then translate the image into and parse a text pixel enumeration, which isn't working in windows...)
I think the best way forward is to use core's image handling functions to create lua methods that will open an image and get pixel values. anyone willing to help with this task please jump in ASAP. we could eliminate all outside libraries, command line (python) calls, and remove the mod security settings for this mod, making it much more portable and easy to use by a wider audience.

[Gael de Sailly]

I've made a script that produces a rivermap from a DEM heightmap. Might be useful in the future.

It's here: rivermapper on GitHub.

Requires python3 and Wand library 0.4.2
Works only for 16-bit depth image for now.

RAM taken: ~ 20 Mo per million pixels, around 700 Mo for 6001*6001 SRTM tiles.
Speed: on my computer, it took 7 minutes to generate the rivers for this 6001*6001 SRTM tile.

Edit: Wand library version, and optimization.

[Gael de Sailly]

Sorry, I forgot to say that the code is only working for wand library 0.4.2.