[Mod][Merged] Valleys Mapgen [valleys_mapgen]

[Ivà]

C++ version, I've found default:stone and default:dirt. I have a server with the latest git version, so you can see it for yourself if you want: mtdig.zapto.org:3000

That's probably because I used a new node for the river bottoms -- river mud. It's there because I couldn't think of a clever way to keep non-water plants (trees) from spawning in the rivers. One or two might be ok, but they were everywhere.

It now runs the latest git version with the nodes.lua from your git repo and it seems ok (I'm on my mobile and can not test it properly). I used the default mgdjr_spflags to generate the map.

Thanks for your work!

[duane]

If this is the C++ version this may be because the biome generation code needs editing in a river mapgen, see my changes to 'generate biomes' in my own core river mapgen https://github.com/paramat/minetest/commit/ffcd316744ad0ce1434cfecfa1844af951c6936b#diff-0316b3962f6ac1c7125de4e659ae3e43R445 lines 464 481 505 526 duane you will need these changes to trigger biome calculation and to work with river water.

Ah! Sand. That solves the problem much more neatly, and I put your changes in as well, thanks.

[duane]

These trees were generated entirely from four schematics. I think they look as good as any created with voxelmanip, and they're faster (though not as fast as the stock trees). Plus the decoration system does all the work of planting them.

+ Spoiler

[Don]

The trees look nice.

[Amaz]

This is awesome! I've added support for it to Lord of the Test, and here are some screenshots of the result: https://imgrush.com/c8680d1c6ef5 and https://imgrush.com/1OF2M95rfjWa.png

[kaadmy]

This mapgen should be merged into the main engine as the fifth option, really. It provides a realistic terrain that is a lot more interesting than v6 and less fantasious than v5 or v7, for those who prefer that style of land.

Keep up the good work!

Agreed! The mapgen in Lua is too slow to play normally, and making it C++ would be great.

[kaadmy]

Wow. I downloaded/compiled and it looks great!
One thing I noticed though, is that it won't start a singleplayer(Or multiplayer?) world if the setting keymap_autorun is not set. Is that required in default, or just this branch>

[duane]

Wow. I downloaded/compiled and it looks great!
One thing I noticed though, is that it won't start a singleplayer(Or multiplayer?) world if the setting keymap_autorun is not set. Is that required in default, or just this branch>

Whoops. That's my autorun code, ala world of warcraft. I hate having to hold the forward key all the time. I'll have to fix that since it's causing problems.

Edit: Done. It has a default now -- the F key.

[duane]

I've uploaded my working support files to github. This project is intended to add the features of vmg that don't belong in C++ to the C++ project.

Here you'll find a file called traditional.lua, which is basically a skeleton for doing voxelmanip changes after the c++ mapgen has created the world. It loads the heightmap, heatmap, and humiditymap created in C++, verifies the correct ground level (heightmap seems to change randomly, possibly because it hasn't actually been completed before control is handed to lua -- I'm not sure), and calls a function which could handle decorations.

I don't like this file, because my goal for this side project was to be compatible, and going back to placing everything directly is counter to that aim. It would be better if we use the biome and decoration managers whenever possible. Of course, there are drawbacks to that.

I intend to drop all of the vmg tree generation in the final product. I've already proved to my satisfaction that schematics can do the job as well or better. In the deco.lua file, I've put code to create schematics programmatically. All the lua processing is done before the first voxel of the world is created, then the C code in the game does the actual handling of the trees, more quickly than you could manage with lua.

I've copied a lot of the decoration and biome data from the default mods, because I had trouble making changes to them without everything spawning everywhere at once, as though there weren't any biomes. I suspect there's a way around that, and we'll be able to remove that code eventually.

[duane]

For those of you who are looking for speed in the C++ version of valleys, I have some good news. The short of it is, use the v7 caves (v7caves flag).

I did some simple profiling. Almost all of the mapgen's functions are called from makeChunk, the heart of the code which actually generates the terrain. I put some cpu time checks through it and got results like this:

T1: 2279078.0181818 calculate_noise (Yike!)
T2: 24672.563636364
T3: 3223.7909090909
T4: 36025.409090909
T5: 51817.645454545
T6: 6307.8272727273
T7: 5776.9636363636
T8: 27691.436363636
T9: 8536.3727272727
T10: 759692.98181818 update liquids and light maps

Notice that calculate_noise is using two orders of magnitude more time than anything else except liquid and light handling (which I can't do much about). So that's our problem area. I immediately did the same thing for that function:

T1: 1524947.6870229 actual perlinMap calls
T2: 310.32061068702
T3: 10171.938931298 all the fancy math

That opened my eyes. I thought the math was going to be a problem, but it's all the perlin noise maps that are killing us. Here's the entirety of the code involved.

Your phone or window isn't wide enough to display the code box. If it's a phone, try rotating it to landscape mode.

Code: Select all

   noise_filler_depth->perlinMap2D(x, z);
   noise_heat->perlinMap2D(x, z);
   noise_humidity->perlinMap2D(x, z);
   noise_heat_blend->perlinMap2D(x, z);
   noise_humidity_blend->perlinMap2D(x, z);

   // vmg noises
   noise_terrain_height->perlinMap2D(x, z);
   noise_rivers->perlinMap2D(x, z);
   noise_valley_depth->perlinMap2D(x, z);
   noise_valley_profile->perlinMap2D(x, z);
   noise_inter_valley_slope->perlinMap2D(x, z);

   if (flags & MG_CAVES) {
      if (spflags & MGDJR_V7_CAVES) {
         noise_v7_caves_1->perlinMap3D(x, y, z);
         noise_v7_caves_2->perlinMap3D(x, y, z);
      } else {
         noise_caves_1->perlinMap3D(x, y, z);
         noise_caves_2->perlinMap3D(x, y, z);
         noise_caves_3->perlinMap3D(x, y, z);
         noise_caves_4->perlinMap3D(x, y, z);
         noise_stala->perlinMap3D(x, y, z);
      }

      if (spflags & MGDJR_LAVA) {
         noise_lava_1->perlinMap2D(x, z);
         noise_lava_2->perlinMap3D(x, y, z);
      }
      if (spflags & MGDJR_GROUND_WATER) {
         noise_water_1->perlinMap2D(x, z);
         noise_water_2->perlinMap3D(x, y, z);
      }
   }

The 3D maps are taking the bulk of the time since all the maps have the same dimensions, and those are exclusively involved in cave generation. So I ran a quick check with mg_flags = nocaves set. Time spent generating map chunks dropped by a factor of more than a hundred. Setting mgdjr_spflags = v7caves was just about as fast. Maybe verson 7's noise maps are more efficient.

The bad news is that liquid and light handling are taking more than ten times as much time as anything else after that. So, there's probably not much to be done to speed everything up past the cave issue.

[duane]

After a spate of fixes and adjustments, here's another screenshot.

+ Spoiler

[Dragonop]

I think the plains/hills need more grass or some other decoration, like small rocks and stuff.

[duane]

I think the plains/hills need more grass or some other decoration, like small rocks and stuff.

I think you're right. I've just started on the lua portion, so there's a lot of room for improvement. It looks really good with Plantlife, especially the ferns.

Speaking of which, VanessaE has outdone herself with textures to support vmg in her HDX high-resolution packs. They look amazing. Now we need higher resolution textures for the new flowers and trees.

https://forum.minetest.net/viewtopic.php?pid=19702

+ Spoiler

+ Spoiler

+ Spoiler

[duane]

This looks better. A little color, to keep them from merging into a dull mass.

+ Spoiler

+ Spoiler

+ Spoiler

[Gael de Sailly]

I'll release version 2.2 this weekend.

EDIT : done.

[duane]

The latest github version of lua vmg has a new option, vmg_caves, that lets you turn off cave creation. It would be a lot for me to give up, since I like caves, but the trade-off is that the mapgen runs about twice as fast as normal (up to four times as fast on my old linux laptop). This could be worthwhile for those who aren't as concerned about the underworld. (And you won't have to worry about falling into a hole while you're mining.)

Edit: This is the original, lua vmg, not the C++ version.

[Ivà]

The latest github version of lua vmg has a new option, vmg_caves, that lets you turn off cave creation. It would be a lot for me to give up, since I like caves, but the trade-off is that the mapgen runs about twice as fast as normal (up to four times as fast on my old linux laptop). This could be worthwhile for those who aren't as concerned about the underworld. (And you won't have to worry about falling into a hole while you're mining.)

But this option turns off v7cavegen too or only the vmg cavegen?

[duane]

But this option turns off v7cavegen too or only the vmg cavegen?

This time, I'm referring to the original, lua version of Valleys. The v7 caves option only applies to the C++ version.

The original Valleys Mapgen's cave creation algorithm (which I also ported to C++) uses several 3D noise maps, which can take a lot of CPU time to create. Both the lua and C++ versions have this issue, but in the latter case, the v7 mapgen's cave generation is available. We don't have a more efficient option for the lua version of Valleys, but now you can at least turn caves off to speed everything else up.

It makes a huge difference in speed.

[Sokomine]

This mapgen should be merged into the main engine as the fifth option, really. It provides a realistic terrain that is a lot more interesting than v6 and less fantasious than v5 or v7, for those who prefer that style of land.

Yes, that's an issue for me as well. I don't like the fancy, unrealistic terrain of v5 or v7 much. It's good for some special situations and sometimes looks great - but it's not a place where it's likely that people would choose to live.

Whoops. That's my autorun code, ala world of warcraft. I hate having to hold the forward key all the time. I'll have to fix that since it's causing problems.

Edit: Done. It has a default now -- the F key.

I want that code as well, please! It annoys me to always have to press w to run forward, sometimes in combination with e. It's particulary annoying on Android devices, but also on the desktop. Please get it merged into the main branch of MT. My fingers would thank you!

As to the speed gain you encountered: Most of it seems to be related to the noise calculation/handling if I got you right? Of course I'm intrested in any speed gains. The village code runs pretty fast by now, but it can't be as fast as a native mapgen. Importing and exporting the voxelmanip data structure to and from lua takes a significant amount of time. But then there are other manipulations of the terrain needed apart from flattening and placing the actual houses. So perhaps voxelmanip will always be needed for the villages.

[duane]

I want that code as well, please! It annoys me to always have to press w to run forward, sometimes in combination with e. It's particulary annoying on Android devices, but also on the desktop. Please get it merged into the main branch of MT. My fingers would thank you!

As you know, I posted this earlier as a patch. I haven't looked into what's required for merging into the game. However, I still find it suspicious that all the autorun code is already there, but not tied to a key. I think someone must have decided that it was a bad idea at some point, so I wonder if it would be accepted anyway.

As to the speed gain you encountered: Most of it seems to be related to the noise calculation/handling if I got you right? Of course I'm intrested in any speed gains. The village code runs pretty fast by now, but it can't be as fast as a native mapgen.

The biggest cpu load I found in profiling vmg-c was the generation of 3D noise tables. Everything else together took about a tenth of that time. After that came lighting and liquid handling, and again, everything else together took about a tenth of that time. I was always taught that optimizing for a 10% performance gain (or less) is usually a waste of time.

I wonder if people aren't overstating the speed of C++ mapgens. There's certainly a performance cost for calling to lua and back, but I don't think it's nearly as great as a lot of other aspects of the game. And lua is designed to be very fast for a scripting language. I've heard it's about half as fast as C at number crunching. There are probably a few compiled languages that can't manage that.

The version 7 mapgen code shows a lot of thought about how to avoid wasted time. The terrain (and cave) generation may be a little strange, but it's efficient. That may have more of an impact than the C compiler.

Edit: I put in a pull request. We'll see what happens.

Edit: Since the two ports are now handling about the same features, I did a set of average chunk times on each. I set vmg_caves to false in the lua version, since we already know that's an issue. The times I got on my rickety, old laptop were 745ms for the C++ version (with its lua companion code) and 1482ms for the original, so the C++ is running about twice as fast. Ok, that's fairly significant.

[duane]

The lua code that I've tinkered together in the valleys_c branch of Valleys Mapgen now adds all of the capabilities from the main branch that weren't included in the C++ code. It features generic trees that replace the default trees (and look much better, for my money), cave decorations, and the flowers we have that are not in the default game. I've also included small and large rocks that can be used to create stone, in case you want to skip wooden tool making (but they're mostly for decoration). I made the rock models myself.

Remember that you must use the C++ vmg to get the full effect (though the lua should work perfectly well with the stock mapgen).

+ Spoiler

+ Spoiler

Edit: What I said about the lua working with earlier mapgens is no longer true. I had to add a lua function for working with the biomemap, so this code will now only work with the C++ version of Valleys Mapgen. However, I now have arrow arum growing in the water, where it belongs.

[duane]

I put together some higher resolution textures for the flowers that aren't in hdx textures, in case anyone wants them.

+ Spoiler

[prof-turbo]

Duane : create your repository for your own hdx textures, and post them in the texture packs section of the forum :)

[duane]

Valleys Mapgen is now using a tweak thought up by Hybrid Dog, which allows it to cooperate with other mods that use voxelmanip. Mods like Caverealms and Intersecting used to have serious issues running with VMG, because the latter builds the world from scratch. When other mods loaded first, they'd look around and find no world to work with. Thanks to a one-line fix, that shouldn't happen anymore.

This solves one of the main problems that started me on the C++ mapgen project. The only catch is, it could get hairy if another mod uses the same trick.

[kaadmy]

What are the name of the noise param settings in the minetest.conf?
Are mgv7 noise params used?

[duane]

What are the name of the noise param settings in the minetest.conf?
Are mgv7 noise params used?

In the lua version the noise parameters are:

Your phone or window isn't wide enough to display the code box. If it's a phone, try rotating it to landscape mode.

Code: Select all

vmg_noise_1 = -10, 50, (1024,1024,1024), 5202, 6, 0.4, 2
vmg_noise_2 = 0, 1, (256,256,256), -6050, 5, 0.6, 2
vmg_noise_3 = 5, 4, (512,512,512), -1914, 1, 1, 2
vmg_noise_4 = 0.6, 0.5, (512,512,512), 777, 1, 1, 2
vmg_noise_5 = 0.5, 0.5, (128,128,128), 746, 1, 1, 2
vmg_noise_6 = 0, 1, (256,512,256), 1993, 6, 0.8, 2
vmg_noise_7 = 3, 1.75, (256,256,256), 1605, 3, 0.5, 2
vmg_noise_8 = 0, 1, (32,32,32), -4640, 4, 0.5, 2
vmg_noise_9 = 0, 1, (32,32,32), 8804, 4, 0.5, 2
vmg_noise_10 = 0, 1, (32,32,32), -4780, 4, 0.5, 2
vmg_noise_11 = 0, 1, (32,32,32), -9969, 4, 0.5, 2
vmg_noise_12 = 0, 1, (64,64,64), 3314, 4, 0.5, 2
vmg_noise_13 = 0, 1, (256,256,256), 2835, 5, 0.5, 4
vmg_noise_14 = 0, 1, (256,256,256), 6674, 5, 0.5, 4
vmg_noise_15 = 0, 1, (256,256,256), 6940, 5, 0.5, 4
vmg_noise_16 = 2, 8, (256,256,256), 2349, 3, 0.5, 2
vmg_noise_17 = 0, 1, (768,256,768), -1805, 4, 0.5, 4
vmg_noise_18 = 0, 1, (243,243,243), -5787, 4, 0.5, 3

In the C++ version, these are copied from V7:

Your phone or window isn't wide enough to display the code box. If it's a phone, try rotating it to landscape mode.

Code: Select all

mgdjr_np_filler_depth = 0, 1.2, (150, 150, 150), 261, 3, 0.7, 2.0
mgdjr_np_v7_caves_1 = 0, 12, (100, 100, 100), 52534, 4, 0.5, 2.0
mgdjr_np_v7_caves_2 = 0, 12, (100, 100, 100), 10325, 4, 0.5, 2.0
mgdjr_np_biome_heat = 50, 50, (750, 750, 750), 5349, 3, 0.5, 2.0
mgdjr_np_biome_heat_blend = 0, 1.5, (8, 8, 8), 13, 2, 1.0, 2.0
mgdjr_np_biome_humidity = 50, 50, (750, 750, 750), 842, 3, 0.5, 2.0
mgdjr_np_biome_humidity_blend = 0, 1.5, (8, 8, 8), 90003, 2, 1.0, 2.0

These are copied from VMG lua:

Your phone or window isn't wide enough to display the code box. If it's a phone, try rotating it to landscape mode.

Code: Select all

mgdjr_np_terrain_height = -10, 50, (1024, 1024, 1024), 5202, 6, 0.4, 2.0
mgdjr_np_rivers = 0, 1, (256, 256, 256), -6050, 5, 0.6, 2.0
mgdjr_np_valley_depth = 5, 4, (512, 512, 512), -1914, 1, 1.0, 2.0
mgdjr_np_valley_profile = 0.6, 0.5, (512, 512, 512), 777, 1, 1.0, 2.0
mgdjr_np_inter_valley_slope = 0.5, 0.5, (128, 128, 128), 746, 1, 1.0, 2.0
mgdjr_np_inter_valley_fill = 0, 1, (256, 256, 256), 1993, 6, 0.8, 2.0
mgdjr_np_caves_1 = 0, 1, (32, 32, 32), -4640, 4, 0.5, 2.0
mgdjr_np_caves_2 = 0, 1, (32, 32, 32), 8804, 4, 0.5, 2.0
mgdjr_np_caves_3 = 0, 1, (32, 32, 32), -4780, 4, 0.5, 2.0
mgdjr_np_caves_4 = 0, 1, (32, 32, 32), -9969, 4, 0.5, 2.0
mgdjr_np_lava_1 = 0, 1, (32, 32, 32), 75266, 4, 0.5, 2.0
mgdjr_np_lava_2 = 0, 1, (32, 32, 32), -89113, 4, 0.5, 2.0
mgdjr_np_water_1 = 0, 1, (32, 32, 32), 82329, 4, 0.5, 2.0
mgdjr_np_water_2 = 0, 1, (32, 32, 32), -59107, 4, 0.5, 2.0

Edit: I've moved the Valleys C++ code to a branch of the fork I'm using, to make coordinating pulls easier. In the process, I changed all occurrences of "djr" to "valleys", so the "mgdjr" entries above will be "mg_valleys" the next time you pull from github. Sorry for any confusion.

[kaadmy]

All right, because terrain looked much more spread out with djr mapgen.
Just wanted to see why :)

Edit: Maybe it would be a good idea to put the parameters and the defaults in the minetest.conf.example?

[duane]

All right, because terrain looked much more spread out with djr mapgen.
Just wanted to see why :)

Edit: Maybe it would be a good idea to put the parameters and the defaults in the minetest.conf.example?

Yep, that's on my todo list.

I should note that if you're using the "fast" flag, the terrain is going to be flatter, since the mapgen is skipping the last step of generation. There are a lot more sharp peaks without it. (Too many for my taste. I'd like to find a setting somewhere between the two, at some point.)

Edit: I've pushed the changes to minetest.conf.example and a better-commented version in doc/valleys_c_minetest.conf.example.

[duane]

I think I've eliminated the nerd poles spawning in the mountains. Here's two shots of the same spot in the same game, before...

+ Spoiler

and after.

+ Spoiler

I can do the same with the lua mapgen, if anyone's interested. Does the broken terrain bother anyone?

Edit: It turns out that this isn't a problem with the lua after all.

[Don]

Is there still cliffs?