Optimizing the networking code

[MirceaKitsune]

Followup to a discussion on the dev IRC. After several tests on multiple servers, and observations longer than an year now, I wanted to bring up a serious discussion on this matter... which I genuinely consider the #1 issue in the Minetest engine.

The networking code of Minetest is extremely unoptimized, compared to other engines with multiplayer support. This causes lag on most servers to be so great, it's impossible to play at all; Chat messages and commands take nearly a minute to reach the server, doors and furnaces / chests can take up to two minutes to open (see the result after right clicking), and chunks can take over three minutes to reach the client. If I want to do as little as exploring a world, I have to wait in front of an invisible wall for minutes, until the chunk there loads up... allowing me to walk just a few more blocks before I hit another unloaded area and the process repeats. Other players reported that lag on servers is so great, it's impossible to follow another player as they show you to their house. Connecting to a server also takes minutes, even with curl.

To be honest, I'd call multiplayer support experimental at this stage, because playing online is in practice impossible... unless the server is very close and ping is minimal. Of course this isn't a complaint, but explaining why the problem is bad and we need a solution. To me it looks like we urgently require a new networking architecture, and a faster way to send packages between the server and clients. The purpose is achieving a normal level of lag, as experienced in every sane engine at the same ping.

So far there's a glimpse of hope: Freeminer is said to have fixed this, by switching to the Enet networking library... I haven't tried it yet since Freeminer has issues of its own. I suggested porting Enet support back into mainstream Minetest... but from what I remember proller saying, there's a compatibility problem with the networking protocol, which would make new clients and old servers unable to connect. We could of course make Enet optional in cmake, and only use it when the client is compiled with it and the server it's connecting to is as well.

In either case, can this be added as a top priority on the dev list please? And do we have programmers experienced with networking in the Minetest development team? Also are the exact problem and solution known by anyone yet?

[Krock]

Enet uses the MIT license which should be compatible with Minetest.
I would like to keep the compability with older Minetest versions (until 0.4.7 at least)
With this, the downloads wouldn't be faster at all. The data amount is still the same.


TL;DR everything.

[Calinou]

The networking code of Minetest is extremely unoptimized, compared to other engines with multiplayer support. This causes lag on most servers to be so great, it's impossible to play at all; Chat messages and commands take nearly a minute to reach the server, doors and furnaces / chests can take up to two minutes to open (see the result after right clicking), and chunks can take over three minutes to reach the client. If I want to do as little as exploring a world, I have to wait in front of an invisible wall for minutes, until the chunk there loads up... allowing me to walk just a few more blocks before I hit another unloaded area and the process repeats. Other players reported that lag on servers is so great, it's impossible to follow another player as they show you to their house. Connecting to a server also takes minutes, even with curl.

Whoa, no. I've been playing just fine on heavily modded servers, with less than a second of lag. Some servers run on slow hardware or run way too many mods, nothing to do against that.

Anyway, ENet can be implemented, it has been done in Freeminer (but after the license change, so we can't just merge the support that was written for it).

[rubenwardy]

I only experience problems on some servers. Old Coder's and Redcrab's work fine for me.

[MirceaKitsune]

Whoa, no. I've been playing just fine on heavily modded servers, with less than a second of lag. Some servers run on slow hardware or run way too many mods, nothing to do against that.

Anyway, ENet can be implemented, it has been done in Freeminer (but after the license change, so we can't just merge the support that was written for it).

I only experience problems on some servers. Old Coder's and Redcrab's work fine for me.

I tried several servers from the server list every few weeks. There are a few which don't have much lag... likely because they're in Europe where I live, and hosted on a good connection. Most however had issues in this regard.

I didn't check what mods the slow servers run. I can't say I noticed really heavy ones by eye, which could justify such slow networking. If there is a way to see the information publicly, it would help to know what mods the problematic servers run. If custom scripts are indeed the culprit, the engine might not be at fault.

Perhaps someone can help me test this? I could try connecting to a server without any mods, which is hosted on a modest connection but not from nearby Europe (an American server would be best). It needs to have a moderately high ping... just enough so it wouldn't cause noticeable lag in an engine like Quake or Minecraft. It would help to bust the myth that it's the engine's fault... which to me feels true, but I don't have actual proof and I could be wrong.

[Hiradur]

When I picked up MT early this year (0.4.9) for the first time I was turned off by low performance and lag as well. I suffered from lag several seconds long (even in single player). I'm pretty sensitive to low FPS and lag which is why I turned the game down. I consider them to be the #1 issue as well. However, I just recently tried the latest dev-build and I gotta say that lag has been improved by a large margin. I'd consider it playable for me if only the FPS wouldn't go down so much on a typical server environment. I got a high-end pc and it's a shame that some servers manage to bog me down to 3 FPS with far view enabled. No other game, even in 4k resolution, manages to do that.

[rubenwardy]

By far view, do you mean unlimited view range? That will always be slow. It renders every loaded block.

I get 60fps with no lag on latest versions, on a pentium series 2.8GB dual core integrated graphics 4GB ram.

[Krock]

By far view, do you mean unlimited view range? That will always be slow. It renders every loaded block.

I get 60fps with no lag on latest versions, on a pentium series 2.8GB dual core integrated graphics 4GB ram.

I doupt networking code has to do with the FPS count.

[MirceaKitsune]

I doupt networking code has to do with the FPS count.

It doesn't. FPS lag is determined by how efficient the code and your graphics card is, while networking lag is determined by connection speed and how fast packages get from the server to clients... totally different things.

[Hiradur]

By far view, do you mean unlimited view range? That will always be slow. It renders every loaded block.

Yes I did. I found out what you said some hours ago. Increasing the view distance to a level that is comparable to Minecraft's far distance (where I get 150-400 FPS) still drops FPS below 60 (and btw: vsync and frame limiter are disabled).

I get 60fps with no lag on latest versions, on a pentium series 2.8GB dual core integrated graphics 4GB ram.

Do you get constant 60 FPS in multiplayer as well? On servers I'm usually <30 FPS on an i5 4570 and a GTX 760 if I'm not looking into the sky.

I doupt networking code has to do with the FPS count.

I would usually agree but what happens if the networking code is blocking so that execution is halted until a certain paket is received? I guess this would decrease FPS. I don't know how much this is a problem here though.
I didn't want to hijack this thread and I know that network lag and FPS are two different things but I just wanted to make clear that I consider both to be one of the top issues with MT.

[Jordach]

Do you get constant 60 FPS in multiplayer as well? On servers I'm usually <30 FPS on an i5 4570 and a GTX 760 if I'm not looking into the sky.

WAT THE.

I get 90fps using 1080p, 4x AA, all the settings turned on and with shaders enabled in multiplayer on a AMD APU (A10-6800K) + GTX 750ti.

You're doing it wrong or you don't have the nVidia drivers installed.

[lansney]

when swimming arround the web for C++ lib, I founded some of these interessting, but later I dont know how to integrat these...

http://www.jenkinssoftware.com/index.html
http://pocoproject.org/index.html
http://think-async.com/
http://www.zoidcom.com/

raknet seems a very good oppurtinity IMO...

[sfan5]

You do realize we won't even consider RakNet because it is not FOSS?

Edit: rubenwardy pointed out that RakNet is open-src.

[lansney]

i dunno cause i'm not a really Developper, but i just give my opinion when I can, and thus, they arent only raknet it's just one on my list...

[rubenwardy]

Raknet isnt FOSS, but you are given the source.

[lansney]

ok, ya Honnstly I didnt se this... but for the others what did you thing about these?

[Sokomine]

The record time for a round-trip message was 20 *minutes* - at least in my experience and on a particular server. That server had serious problems (multiple mods installed that conflicted or just did the same). Yet it took 10 minutes for my chat message to arrive at another player's console - and 10 minutes for the trip back. We timed it. It was really horrible.

Normal servers are not so bad. Digging and building usually works fine on most servers. VanessaEs and OldCoders servers usually respond fast - even though they're half a world away. There's no noticable diffrence to european servers. Singleplayer does not feel noticeably faster.

Version of game and clients, amount of players online, mods installed and perhaps some config settings seem to be decisive factors regarding lag experienced on servers. Most servers do seem to get slower if more players are running around. This cannot be explained away by more players needing more ressources on a server - certainly they do, but not *that* much. Simple servers may run well on a Raspberry Pi - and if you add more players, a good dedicated server may get slow and unusable.

There are other factors as well. Spawn on VanessaEs survival server just kills my FPS. There are too many signs there. And players got a bit too enthusiastic about pipeworks and technic, so there's constantly something floating through tubes, and half of the blocks change their status every second or so. Walking away from spawn helps a lot.

In order to find out how much lag there really is and what's causing it, we need something to measure it more reliably than the feeling that it's too laggy on a particular server right now. At least some basic setup - maybe a simple, reproducable, flat lua mapgen plus only the default mod could be a good start. Strip away all abms as well. Join the world with enough players.

[Krock]

Most servers do seem to get slower if more players are running around.

Spawn on VanessaEs survival server just kills my FPS. There are too many signs there.

My server (as example) could handle up to 40 players at once but running it over a regular router is just crazy. The more data sending intervals the server has, the higher is the chance to mess up the router.. somehow.


About the FPS-killer: It's not directly releated to the networking code - An earlier discussion