LuaJIT

[BlindBanana]

Are there any efforts in evaluating possible gains by using LuaJIT instead of Lua within Minetest?

Please correct me, but I think LuaJIT is still only a 32 bit library, so Minetest would be limited to 32 bit builds.


Anyhow, a somewhat hacky way to build Minetest with LuaJIT (precompiled static lib) would be to modify
src/CMakeLists.txt for minetest and minetestserver executable:

Your phone or window isn't wide enough to display the code box. If it's a phone, try rotating it to landscape mode.

Code: Select all

    if(LUAJIT)
        target_link_libraries(
            ${LUAJIT}
        )
    else(LUAJIT)
        target_link_libraries(
            ${LUA_LIBRARY}
        )
    endif(LUAJIT)

Please notice that the bundled liblua will be built but not used if -DLUAJIT=../LuaJIT-1.1.8/liblua.a is used.

configuring on linux (64bit system) with:

Your phone or window isn't wide enough to display the code box. If it's a phone, try rotating it to landscape mode.

Code: Select all

export LDFLAGS=-m32
cmake . -DBUILD_CLIENT=1 -DBUILD_SERVER=1 -DCMAKE_BUILD_TYPE=Release \
-DENABLE_CURL=1 -DENABLE_FREETYPE=1 -DENABLE_GETTEXT=1 -DENABLE_SOUND=1 \
-DRUN_IN_PLACE=1 -DWARN_ALL=1 \
-DLUAJIT=../LuaJIT-1.1.8/liblua.a \
-DCMAKE_C_FLAGS=-m32 -DCMAKE_CXX_FLAGS=-m32

and finaly build with:

Your phone or window isn't wide enough to display the code box. If it's a phone, try rotating it to landscape mode.

Code: Select all

gmake -j5

But this is untested, since I have no 32 bit libs on my system.

[Calinou]

There are gains, see this: http://forum.minetest.net/viewtopic.php?pid=75924#p75924

configuring on linux (64bit system) with:
export LDFLAGS=-m32
cmake . -DBUILD_CLIENT=1 -DBUILD_SERVER=1 -DCMAKE_BUILD_TYPE=Release \
-DENABLE_CURL=1 -DENABLE_FREETYPE=1 -DENABLE_GETTEXT=1 -DENABLE_SOUND=1 \
-DRUN_IN_PLACE=1 -DWARN_ALL=1 \
-DLUAJIT=../LuaJIT-1.1.8/liblua.a \
-DCMAKE_C_FLAGS=-m32 -DCMAKE_CXX_FLAGS=-m32

You only need the -DLUAJIT=../LuaJIT-1.1.8/liblua.a and optionally -DRUN_IN_PLACE=1, -DENABLE_GETTEXT=1 and -DENABLE_FREETYPE=1, the other flags are defaults. :)

[jojoa1997]

what is the difference between luajit code wise.

[rarkenin]

what is the difference between luajit code wise.

LuaJit allows for slightly faster LUA by precompiling.

[VanessaE]

It isn't just "slightly" faster, it's about 2x as fast as plain interpreted Lua on some stuff, and in some cases 10x or more.

I ran it on my server for several hours as a stability test, and it definitely improves things. I look forward to seeing it properly integrated into Minetest.

You can't just blindly link to it though, as vanilla Minetest has a couple of minor bugs that don't show up with regular Lua (or regular Lua doesn't enforce some syntax as strictly as LuaJIT does, I'm not sure which).

[BlindBanana]

[...] I ran it on my server for several hours as a stability test, and it definitely improves things. [...]
You can't just blindly link to it though, as vanilla Minetest has a couple of minor bugs that don't show up with regular Lua (or regular Lua doesn't enforce some syntax as strictly as LuaJIT does, I'm not sure which).

So you have a branch where it is integrated? Maybe as 64 bit lib, since LuaJIT 2.0 is 64 bit capable? Is it on github?

[PilzAdam]

It isn't just "slightly" faster, it's about 2x as fast as plain interpreted Lua on some stuff, and in some cases 10x or more.

IIRC sapier has tested it in mobf and it was the same speed and even slower in some points.

[BlindBanana]

Since all my servers run x64, I tried LuaJIT 2.0.1-fixed (Freebsd 8.3, system gcc - like 4.2) and i throws many errors in:

Your phone or window isn't wide enough to display the code box. If it's a phone, try rotating it to landscape mode.

Code: Select all

    script.cpp / scriptapi.cpp / scriptapi_inventory.cpp / scriptapi_nodemeta.cpp /
    scriptapi_object.cpp / scriptapi_entity.cpp / scriptapi_noise.cpp /
    scriptapi_nodetimer.cpp / scriptapi_env.cpp / scriptapi_item.cpp /
    scriptapi_node.cpp / scriptapi_craft.cpp / scriptapi_content.cpp /
    scriptapi_common.cpp / scriptapi_types.cpp

like "undefined reference to `lua_whatever'.

I'll look into it, but this may take some time.

http://luajit.org/performance_x86.html has some pretty numbers for perfomance gains with LuaJIT 2.
Which Version did sapier test?

[VanessaE]

To build with LuaJIT, do the following, using the latest git of Minetest (from directions given to me by RealBadAngel):

1. Get the latest LuaJIT source code.
2. Compile it in a work folder but do not install it.
3. `git clone` or `git pull` the latest git sources for the Minetest engine.
4. Go into the minetest folder, run `make clean` if you need to, then follow the standard Minetest build procedure -- but do not install it.
5. Delete the binaries you just built - we're interested in the intermediate results at this stage, not the final binaries. That is, `rm bin/minetest*`.
6. Copy libluajit.a from the LuaJIT source folder over to minetest/src/lua/build and name it as "liblua.a".
7. Edit minetest/builtin/serialize.lua: At line 200, find "unittest_input = {a bunch of stuff here}" and edit out the \[\] from the end of the "escapechars" field, so that you have (all on one line, perhaps):

Your phone or window isn't wide enough to display the code box. If it's a phone, try rotating it to landscape mode.

Code: Select all

unittest_input = {escapechars="\n\r\t\v\\"\'", noneuropean="θשׁ٩∂"}

8. Go back to the root of the Minetest folder and run `make` a second time, without any parameters. Do not do anything that would clean up from the last compile - we want the cached results to remain. This will force it to re-link the server and client against the newly-copied-in-and renamed libluajit.a
9. Install the resultant Minetest binaries, if necessary (this should work fine for run-in-place builds too).
10. Get minetest_game if you want it, and install it normally.

[prestidigitator]

Hmm. Is there a shared library rather than a static archive? Static archives are almost always a bad idea these days, especially if you are ever going to allow for dynamic loading (e.g. plugins providing more Lua functionality).

[kaeza]

Hmm. Is there a shared library rather than a static archive? Static archives are almost always a bad idea these days, especially if you are ever going to allow for dynamic loading (e.g. plugins providing more Lua functionality).

You can load other libs from Lua without requiring Lua itself to be a dylib. This is what I did here.

[prestidigitator]

You can load other libs from Lua without requiring Lua itself to be a dylib. This is what I did here.

We discussed this a little more in IRC, but I'll clarify a little here for other people too. Here are some of the problems that static libraries can cause:

• Symbols from the library that the main program doesn't use also cannot be used from libraries loaded dynamically at run time. So if program P links with library A but doesn't use symbol S (e.g. a function) from it, and at run time P dynamically loads library B, then B will fail to load if it needs symbol S. Also, there can be conflicts if the dynamic linker attempts to link a dynamic copy of A to the process again because of B. If P links to A as a shared library, on the other hand, all of its symbols are available to B.
• Similarly, you do NOT want to link a static library dependency into a shared library. If libraries A and B both linked symbols from a static library C in, there could very well be conflicts at run time.
• I'm not sure if anything has changed in recent compiler/linker/loader/language versions, but in older implementations there were issues initializing static objects (calling their constructors) if they were in a static library. On the other hand, shared libraries do have well specified behavior regarding the construction of static objects, and give reasonable guarantees about order too (something like all trivial initializations from a compilation unit occurring before ANY code from that module is executed, and non-trivial constructors run before any other (non-constructor) code is executed so long as you haven't violated the guarantee in the constructors themselves).

Static libraries CAN help when you are running in a very restrictive environment (especially in terms of memory), and you use only a small portion of the functionality of a library. However, if any OTHER applications use the same library, shared libraries can again be better (since the code portion of the shared library can be cached and potentially loaded only once, whereas if two applications link the same parts of a static library, it has to be resident in memory twice).

All in all it's a good rule to ALWAYS use shared libraries unless you have a very, very specific reason for using static libraries and are running in a very restricted environment (I'm talking for example embedded devices here, not just older PCs).

[BlindBanana]

I oppose!

Shared linking breaks stuff badly if more programs depend on different versions of a library.
So old versions are kept on systems even if they have serios security issues...

[Mito551]

Sorry, no 32-bit builds (don't know how)

do you use 64bit machine?

[kaeza]

To build with LuaJIT, do the following, using the latest git of Minetest (from directions given to me by RealBadAngel):

1. Get the latest LuaJIT source code.
2. Compile it in a work folder but do not install it.
3. `git clone` or `git pull` the latest git sources for the Minetest engine.
4. Go into the minetest folder, run `make clean` if you need to, then follow the standard Minetest build procedure -- but do not install it.
5. Delete the binaries you just built - we're interested in the intermediate results at this stage, not the final binaries. That is, `rm bin/minetest*`.
6. Copy libluajit.a from the LuaJIT source folder over to minetest/src/lua/build and name it as "liblua.a".
7. Edit minetest/builtin/serialize.lua: At line 200, find "unittest_input = {a bunch of stuff here}" and edit out the \[\] from the end of the "escapechars" field, so that you have (all on one line, perhaps):

Your phone or window isn't wide enough to display the code box. If it's a phone, try rotating it to landscape mode.

Code: Select all

unittest_input = {escapechars="\n\r\t\v\"\'", noneuropean="θשׁ٩∂"}

8. Go back to the root of the Minetest folder and run `make` a second time, without any parameters. Do not do anything that would clean up from the last compile - we want the cached results to remain. This will force it to re-link the server and client against the newly-copied-in-and renamed libluajit.a
9. Install the resultant Minetest binaries, if necessary (this should work fine for run-in-place builds too).
10. Get minetest_game if you want it, and install it normally.

Compiled following instructions and now game is waaaay faster. I tried ramming at the map borders (unloaded chunks) at full speed (fly+fast) but could not reach them because terrain was generating very fast. Tested with moreores+gloopores (Lua terrain generation). I have to run some more tests using CPU intensive mods (MOBF, Mesecons, Technic, and some of paramat's mods), but overall this thing is much faster.

Thank you VanessaE and RealBadAngel for the quick (and hacky :P) way to use LuaJit.

[prestidigitator]

I oppose!

Shared linking breaks stuff badly if more programs depend on different versions of a library.
So old versions are kept on systems even if they have serios security issues...

This is why the (modern) convention is to install libraries using symlinks and multiple names. A program that doesn't care the version links to 'libmylib.so' (when provided), a program that requires version 1 links to 'libmylib-1.so', and a program that needs very specifically a version 1.2.3 links to 'libmylib-1.2.3.so'. This actually works beautifully when developers and release managers handle their dependencies properly, and allows multiple versions to coexist seamlessly.

This is a GOOD thing for version management, not a problem. If you think about it, all those programs that link to static libraries are even WORSE for keeping old versions of the libraries around; every instance of a program linked with the old version of the library at compile time has a copy of the old version hidden inside it! Compare this to shared libraries, where if a program uses a loose version dependency it can actually use a newer version of the shared library if it happens to be installed on the system. If you needed to replace an old insecure version of the shared library, any program that used it (and has specific enough dependencies to keep it from linking with newer versions of the library) would fail to run and tell you about the dependency problem, instead of silently and happily running using the old (hidden) insecure code!

[Calinou]

It isn't just "slightly" faster, it's about 2x as fast as plain interpreted Lua on some stuff, and in some cases 10x or more.

IIRC sapier has tested it in mobf and it was the same speed and even slower in some points.

There is profiling and looking at performance.

When you profile, you just look at some random abstract parameters like CPU time; this is generally not how you should look for performance improvements.

When you look at performance, you look at client frames per second when running a server at the same time, or server CPU usage (which has an impact on power usage and temperature), which is what the users will see. And you should do that on several ranges of hardware (old, new, low-end, high-end).

[BlindBanana]

I oppose!

Shared linking breaks stuff badly if more programs depend on different versions of a library.
So old versions are kept on systems even if they have serios security issues...

This is why the (modern) convention is to install libraries using symlinks and multiple names. A program that doesn't care the version links to 'libmylib.so' (when provided), a program that requires version 1 links to 'libmylib-1.so', and a program that needs very specifically a version 1.2.3 links to 'libmylib-1.2.3.so'. This actually works beautifully when developers and release managers handle their dependencies properly, and allows multiple versions to coexist seamlessly.

This is a GOOD thing for version management, not a problem. If you think about it, all those programs that link to static libraries are even WORSE for keeping old versions of the libraries around; every instance of a program linked with the old version of the library at compile time has a copy of the old version hidden inside it! Compare this to shared libraries, where if a program uses a loose version dependency it can actually use a newer version of the shared library if it happens to be installed on the system. If you needed to replace an old insecure version of the shared library, any program that used it (and has specific enough dependencies to keep it from linking with newer versions of the library) would fail to run and tell you about the dependency problem, instead of silently and happily running using the old (hidden) insecure code!

No, please not!

Let me throw in some quotes from http://harmful.cat-v.org/software/dynamic-linking/:

“I tend to think the drawbacks of dynamic linking outweigh the advantages for many (most?) applications.” — John Carmack

All the purported benefits of dynamic linking (aka., ‘shared libraries’, which is a misnomer as static linking also shares libraries) are myths while it creates great (and often ignored) problems.

Both performance and security are seriously harmed by dynamic linking, but the damage caused by the huge complexity created by dynamic linking is extensive in almost all areas (the term ‘dll hell’ is just one example of the many hells created in dynamic linking environments).

There isn't a copy of the entire C library in every binary. There is a copy of each library routine called, directly or indirectly, by the program in question.

Sharing of instructions is done at the granularity of process text segments, as in V6 or V7 Unix. The text segment of a process that forks is shared between parent and child by page mapping. Also, running (via exec) a program multiple times concurrently causes the (pure) text segment to be shared by page mapping across those processes. So all copies of rc and on a machine should share a text segment.

Also consider the "Links section" at the end of the linked site.

[BlindBanana]

[...]When you look at performance, you look at client frames per second when running a server at the same time, or server CPU usage (which has an impact on power usage and temperature), which is what the users will see. And you should do that on several ranges of hardware (old, new, low-end, high-end).

With an compiletime option to link either against liblua.a or against an external libluajit.a some MT users could run performance tests on their hardware and report back to this forum.

Maybe with an separate "performance" game like "minimal"?

[VanessaE]

Sorry, no 32-bit builds (don't know how)

do you use 64bit machine?

Indeed I do.

[Mito551]

Sorry, no 32-bit builds (don't know how)

do you use 64bit machine?

Indeed I do.

so... how about a 32-bit virtual machine?

[BlindBanana]

Here is a Release Build for Ubuntu Precise AMD64, with LuaJIT v2.0.1 linked in. Also includes cURL support. Includes both the server and the client, and minetest_game. Both LuaJIT and Minetest were compiled with -O3; this build does not contain debug symbols. builtin/serialize.lua has already been patched.

minetest commit ID: d6026a5fee
minetest_game commit ID: 3a617bcb6e

http://digitalaudioconcepts.com/vanessa/hobbies/minetest/builds/minetest-git_20130318-1_amd64.deb

Sorry, no 32-bit builds (don't know how)

If you have the needed 32 bit libs on your system you could build minetest with:

Your phone or window isn't wide enough to display the code box. If it's a phone, try rotating it to landscape mode.

Code: Select all

export LDFLAGS=-m32
cmake . -DCMAKE_C_FLAGS=-m32 -DCMAKE_CXX_FLAGS=-m32
gmake -j2

The "-m32" switch tells the compiler and linker that you want a 32bit build.

[Mito551]

Here is a Release Build for Ubuntu Precise AMD64, with LuaJIT v2.0.1 linked in. Also includes cURL support. Includes both the server and the client, and minetest_game. Both LuaJIT and Minetest were compiled with -O3; this build does not contain debug symbols. builtin/serialize.lua has already been patched.

minetest commit ID: d6026a5fee
minetest_game commit ID: 3a617bcb6e

http://digitalaudioconcepts.com/vanessa/hobbies/minetest/builds/minetest-git_20130318-1_amd64.deb

Sorry, no 32-bit builds (don't know how)

If you have the needed 32 bit libs on your system you could build minetest with:

Your phone or window isn't wide enough to display the code box. If it's a phone, try rotating it to landscape mode.

Code: Select all

export LDFLAGS=-m32
cmake . -DCMAKE_C_FLAGS=-m32 -DCMAKE_CXX_FLAGS=-m32
gmake -j2

The "-m32" switch tells the compiler and linker that you want a 32bit build.

oh well.

[Menche]

If you have the needed 32 bit libs on your system you could build minetest with:

Your phone or window isn't wide enough to display the code box. If it's a phone, try rotating it to landscape mode.

Code: Select all

export LDFLAGS=-m32
cmake . -DCMAKE_C_FLAGS=-m32 -DCMAKE_CXX_FLAGS=-m32
gmake -j2

The "-m32" switch tells the compiler and linker that you want a 32bit build.

Oh, the -m32 has to be in the linker flags too? That makes sense.

CMake has it's own variables for linker flags: CMAKE_MODULE_LINKER_FLAGS, CMAKE_EXE_LINKER_FLAGS, and CMAKE_SHARED_LINKER_FLAGS.

[VanessaE]

Here's an excerpt from my build script that should help a few people out.

This is made for bash, and is primarily directed at Ubuntu and perhaps other Debian-based Linux distributions. It uses checkinstall to prepare the actual package file for minetest/minetest_game, and is what I used to produce the one seen above this post.

Set MINETESTDIR to the root of your minetest source tree.
Set MINETEST_GAMEDIR to root of your minetest_game tree.
Set LUAJITDIR to the root of your LuaJIT source tree.

Use absolute paths (including "~" for your homedir if needed) for all three, no trailing slashes.

You may also wish to adjust the "make -j7" to a lower number (perhaps set it to number of CPUs*cores in your machine, plus 1).

Your phone or window isn't wide enough to display the code box. If it's a phone, try rotating it to landscape mode.

Code: Select all

#!/bin/bash

set -x

MINETESTDIR="/path/to/minetest"  #  That is, the main minetest engine.
MINETEST_GAMEDIR="/path/to/minetest_game"
LUAJITDIR="/path/to/LuaJIT-2.0.1"

cd $LUAJITDIR
make clean
make -j7

cd $MINETESTDIR

export CXXFLAGS=-O3
make clean
rm CMakeCache.txt
cmake . -DRUN_IN_PLACE=0 -DENABLE_FREETYPE=0

make -j7

# skip the first 195 lines of serialize.lua, then find and alter the first
# match to remove \[\] from it.  Then remove the compiled binaries and force a
# re-link of the source against LuaJIT

sed -i '195,$s/\\\[\\\]//' $MINETESTDIR/builtin/serialize.lua
cp $LUAJITDIR/src/libluajit.a $MINETESTDIR/src/lua/build/liblua.a
rm bin/*
make

# Copy over default, unmodified game content

cp -a $MINETEST_GAMEDIR $MINETESTDIR/games

# Install the completed build

cd $MINETESTDIR

sudo checkinstall --fstrans=no -y --backup=no --pkgname=minetest-git



[BlindBanana]

Impressive!

Just a short and therefore limited test, but luajit (v2.0.1 with hotfix) brings real speed improvement:

• Before luajit using nuke means waiting for the explosion sound and disconnect/reconnet to see the result. With luajit the nodes/blocks are instantly blown away!
• Also the generation of new areas ist faster, I could not outrun the mapgen anymore!

Server:

• FreeBSD 8.3
• MT 0.4.5 with fixed builtin/misc.lua, builtin/serialize.lua
• ~ 300 MiB ram used

Client:

• openSUSE (homegrown Version :-} )
• MT 0.4.5 with fixed builtin/misc.lua, builtin/serialize.lua
• ~ 1.1 GiB system ram used (265px texturepack)
• ~ 420 MiB gpu ram used

[prestidigitator]

No, please not!

Let me throw in some quotes from http://harmful.cat-v.org/software/dynamic-linking/:
...
Also consider the "Links section" at the end of the linked site.

Sorry, man, but what you linked to is a load of bunk. Show me a system that doesn't rely heavily on shared libraries and I'll show you a system that is completely unmaintainable and broken. I'm surprised that was written in 2008, because the authors seem to be stuck in a 1980's mindset. The issues that have been mentioned about versions are versioning and release management problems, not problems with using dynamic libraries. Relying on static linking is shooting yourself in the foot.

[BlindBanana]

No, please not!

Let me throw in some quotes from http://harmful.cat-v.org/software/dynamic-linking/:
...
Also consider the "Links section" at the end of the linked site.

Sorry, man, but what you linked to is a load of bunk. Show me a system that doesn't rely heavily on shared libraries and I'll show you a system that is completely unmaintainable and broken. I'm surprised that was written in 2008, because the authors seem to be stuck in a 1980's mindset. The issues that have been mentioned about versions are versioning and release management problems, not problems with using dynamic libraries. Relying on static linking is shooting yourself in the foot.

Every system shares libraries, this has nothing to do with static or dynamic linking!
So it's impossible to show you a system without shared libraries.

But you could look into Plan 9.

When a library is statically linked to an executable only a copy of each library routine called is included.
Which btw makes deployment a lot easier.

[Traxie21]

If someone could maintain a version of Minetest for Linux that does not use any system libraries but keeps them in the executable, I'd love it.

I have two linux computers, one of which can probably run MT, but neither can connect to the internet. So I cant install the needed libraries for MT. (No friggin idea why, they were second hand, I bought them that way)

[prestidigitator]

Every system shares libraries, this has nothing to do with static or dynamic linking!

Actually it has everything to do with static and dynamic linking. What you might be getting confused with is run-time loading of dynamic libraries, where the application requests a library that was not specified at compile/link time. This is incredibly useful for plugin frameworks. It must be used with care, because you want to make absolutely sure nothing can be loaded that was not intended by the application design, but it is still invaluable.

Ironically you are using a game that essentially uses unchecked, completely open run-time loading of dynamic libraries for just about everything, so arguing that it is a bad thing is a little hypocritical. Every mod we use is a library loaded at run-time. Nevermind that it is written in Lua and not C/C++.

When a library is statically linked to an executable only a copy of each library routine called is included.
Which btw makes deployment a lot easier.

This is where I think you are getting very confused between shared libraries and run-time loading. If the library is a shared library, your statement is false. The WHOLE shared library is absolutely loaded into memory. Period. However, the code and read-only data portions of the library can be shared between running processes (and not just ones started from the same parent process that happened to use the library; I mean ALL applications on the system that use the shared library).

I also disagree that statically linking using an archive library helps with deployment. If you deploy by simply providing a downloadable folder and don't want to bother thinking about library paths or launch scripts or anything, you might be right. However, anyone who has any experience deploying real applications or maintaining packages can just as easily package a shared library with an application, whether the application is installed or run in place from a download folder. You can even package a shared library with the application and manage the application execution in such a way that it is used only if the library is not already included on the system. This is whether you dynamically load the library at run-time or link to it at compile time.