Database interface & map,env,player metadata and more

jachoo
Member
 
Posts: 45
Joined: Mon Sep 12, 2011 08:29

Database interface & map,env,player metadata and more

by jachoo » Thu Feb 16, 2012 16:43

I'm working on a completely new database interface.

The idea is that all map-, env-, player- (etc.) data should be stored in a database rather than in files. Also, using that databases should be possible from lua.

[h]Why it is beneficial:[/h]

With such database interface (available in lua too) mods could easily and effectively store their own data or metadata for players, map, env etc.

If you are mod developer, you can add metadata to almost everything (map, players, etc.). You don't have to save data in files and do other boring things like that.
If you are c++ dev, you can easily create new databases and use existing ones. No more need to creating new files like ipban.txt. Just open a table in db and use it.

[h]What is done:[/h]

- new database interface (on templates, very easy to use without writing SQL queries)
- two databases created:
- map.sqlite contains: map, map metadata, sector meta
- env.sqlite contains: env metadata, auth data, ban manager data, players data and [new!] players metadata
- new lua functions for metadata:
- set_player_meta(string player_name, string meta_name, string type, value)
- get_player_meta(string player_name, string meta_name, string type)
- set_map_meta(string meta_name, string type, value)
- get_map_meta(string meta_name, string type)
- new lua functions for custom databases:
- get_database(string name) -> int (unique database ID)
- get_db_table(int database, string name) -> int (unique table ID)
- get_table_data(int table, string key_type, key, string data_type) -> nil / data of proper type
- set_table_data(int table, string key_type, key, string data_type, data)
- remove_table_data(int table, string key_type, key)
- NOTE: in all new lua functions, possible type names are: string, int, double, bool, v3s16, v3f, v3fpos. Example use in data/mods/jachoo/init.lua
- reading & writing map from/to files discontinued (anyone uses that?)
- writing data to files discontinued (loading works, but saving only to new databases)

[h]What is planned:[/h]

- add metadata to map blocks
- speed up database access

[h]Sources and code examples[/h]

https://github.com/jachoo/minetest-jachoo2/

(not tested on linux, may need some fixes)

Create/init the database and tables:
Your phone or window isn't wide enough to display the code box. If it's a phone, try rotating it to landscape mode.
Code: Select all
Database* m_database;            //database
Table<v3s16,binary_t>& m_blocks; //table for map blocks. key=v3s16, data=binary
Table<std::string>& m_map_meta;  //table for map metadata. key=string, data=automatic
[...]
m_database( new Database("map.sqlite") ),
m_blocks( m_database->getTable<v3s16,binary_t>("blocks") ),
m_map_meta( m_database->getTable<std::string>("map_meta") )


Writing binary data (ie map block) to database:
Your phone or window isn't wide enough to display the code box. If it's a phone, try rotating it to landscape mode.
Code: Select all
std::ostringstream o(std::ios_base::binary);
o.write((char*)&version, 1);
block->serialize(o, version, true);

// Write block to database   
m_blocks.put(block_pos,o.str());


Interface for getting player metadata in C++:
Your phone or window isn't wide enough to display the code box. If it's a phone, try rotating it to landscape mode.
Code: Select all
template<class Data> Data getPlayerMeta(const Player& player, const std::string& name)
{
    return m_players_meta.get<Data>(player.getName()+":"+name);
}


Lua simple example of using player metadata: command /lastmessage:
Your phone or window isn't wide enough to display the code box. If it's a phone, try rotating it to landscape mode.
Code: Select all
minetest.register_on_chat_message( function(name, message)
    local meta_name = "lastmessage"
    if message:sub(1,1) == '/' then
        local cmd = "/lastmessage"
        local s
        if message:sub(0, #cmd) == cmd then
            local v = minetest.get_player_meta(name,meta_name,"string")
            if v then
                minetest.chat_send_player(name, 'Your last message: '..v)
            end
            return true
        end
    else
        minetest.set_player_meta(name,meta_name,"string",message)
    end
end)


Using custom databases in lua mods:
Your phone or window isn't wide enough to display the code box. If it's a phone, try rotating it to landscape mode.
Code: Select all
db = minetest.get_database("my_database") -- creates/loads database from mod's directory
tab = minetest.get_db_table(db,"my_table") -- creates/loads table from that database
local n = minetest.get_table_data(tab,"string","some key","int") -- returns nil or number from table with key "some key"
local pos = minetest.get_table_data(tab,"int",666,"v3s16") -- returns nil or v3s16 from table with key 666
minetest.set_table_data(tab,"int",666,"v3s16",{x=0,y=0,z=1}) -- inserts/overrides record in a table
Last edited by jachoo on Mon Feb 20, 2012 15:54, edited 1 time in total.
 

User avatar
Jordach
Member
 
Posts: 4412
Joined: Mon Oct 03, 2011 17:58
GitHub: Jordach
IRC: Jordach
In-game: Jordach

by Jordach » Thu Feb 16, 2012 17:33

So, will this be your new client?

( ͡° ͜ʖ ͡°) ( ͡o ͜ʖ ͡o) [$ ( ͡° ͜ʖ ͡°) $] ( ͡$ ͜ʖ ͡$) ヽ༼ຈل͜ຈ༽ノ



My image and media server is back online and is functioning as normal.
 

jachoo
Member
 
Posts: 45
Joined: Mon Sep 12, 2011 08:29

by jachoo » Fri Feb 17, 2012 10:59

It's completely server-side. Client is not affected.
 

User avatar
Jordach
Member
 
Posts: 4412
Joined: Mon Oct 03, 2011 17:58
GitHub: Jordach
IRC: Jordach
In-game: Jordach

by Jordach » Fri Feb 17, 2012 11:37

Ah so when you are Single player, then it should not lag?

(The console actually types out the test on some pcs.)

( ͡° ͜ʖ ͡°) ( ͡o ͜ʖ ͡o) [$ ( ͡° ͜ʖ ͡°) $] ( ͡$ ͜ʖ ͡$) ヽ༼ຈل͜ຈ༽ノ



My image and media server is back online and is functioning as normal.
 

Gatharoth
Member
 
Posts: 196
Joined: Thu Dec 22, 2011 02:54

by Gatharoth » Fri Feb 17, 2012 19:17

I personally can't wait for this to be complete. :3 It will open up a whole new era for mods.
 

User avatar
Death Dealer
Member
 
Posts: 1379
Joined: Wed Feb 15, 2012 18:46

by Death Dealer » Sat Feb 18, 2012 17:36

Wow sounds great!
Keep calm and code python^_^
 

Nemo08
Member
 
Posts: 132
Joined: Mon Dec 26, 2011 04:59

by Nemo08 » Sat Feb 18, 2012 22:38

Death Dealer wrote:Wow sounds great!
+1
You can make a delete feature data from the database if the mod that created them removed? I also foresee a problem with changing the format of the data in different versions of mods, can be sure to write in the base version of the mod and use the data only for this version?
 

jachoo
Member
 
Posts: 45
Joined: Mon Sep 12, 2011 08:29

by jachoo » Sat Feb 18, 2012 23:16

Nemo08 wrote:
Death Dealer wrote:Wow sounds great!
+1
You can make a delete feature data from the database if the mod that created them removed? I also foresee a problem with changing the format of the data in different versions of mods, can be sure to write in the base version of the mod and use the data only for this version?


It would be problematic. Already there is no "uninstall mod" feature, and it's not likely to be planned soon... If modder wants to store lots of data I would recommend to create a new separate database (in mod's dir, so it could be removed with a mod). Creating a new database should be just one Lua command :)

Versioning data in DB sounds, well... quite crazy. Instead, I'm thinking about creating multi-keys db records. If you're a modder - you would use one of the keys as a version number, if you want.
 

Gatharoth
Member
 
Posts: 196
Joined: Thu Dec 22, 2011 02:54

by Gatharoth » Sun Feb 19, 2012 01:01

jachoo wrote:
Nemo08 wrote:
Death Dealer wrote:Wow sounds great!
+1
You can make a delete feature data from the database if the mod that created them removed? I also foresee a problem with changing the format of the data in different versions of mods, can be sure to write in the base version of the mod and use the data only for this version?


It would be problematic. Already there is no "uninstall mod" feature, and it's not likely to be planned soon... If modder wants to store lots of data I would recommend to create a new separate database (in mod's dir, so it could be removed with a mod). Creating a new database should be just one Lua command :)

Versioning data in DB sounds, well... quite crazy. Instead, I'm thinking about creating multi-keys db records. If you're a modder - you would use one of the keys as a version number, if you want.


I see a small work-around for the "uninstall" problem. Add a small requirement to allow the use of the database. For an example, have the mods send a heartbeat when they are loaded. If heartbeat fails to be heard X times, remove all data from the mod.

So, from the mod aspect, something like
Your phone or window isn't wide enough to display the code box. If it's a phone, try rotating it to landscape mode.
Code: Select all
minetest.database_heartbeat("secret_passages")


The called function from above, would send the heartbeat for the mod "secret_passages". Thus informing the database that it is still active.

Also since it's a database, it can store its own needed information, such at Number of heartbeats failed for the mods.

At least the way I see it, it will never be used except on the start-up of the server. So it shouldn't require anything more than that.
Last edited by Gatharoth on Sun Feb 19, 2012 01:03, edited 1 time in total.
 

Nemo08
Member
 
Posts: 132
Joined: Mon Dec 26, 2011 04:59

by Nemo08 » Sun Feb 19, 2012 10:53

jachoo wrote:...
If modder wants to store lots of data I would recommend to create a new separate database (in mod's dir, so it could be removed with a mod).
...

Well, it removes all the problems with the versions
 

jachoo
Member
 
Posts: 45
Joined: Mon Sep 12, 2011 08:29

by jachoo » Mon Feb 20, 2012 00:37

I've just added Lua interface for creating and accessing custom databases. Who will test it?

Explanation in first post. Code -> check github.

Example code using new functions:

Your phone or window isn't wide enough to display the code box. If it's a phone, try rotating it to landscape mode.
Code: Select all
-----------------------------------------------------------
-- Custom database example:
-- /send <player> <message>
-- /mailbox
-----------------------------------------------------------

jachoo = {}
jachoo.db = minetest.get_database("testdb")
jachoo.mailbox = minetest.get_db_table(jachoo.db,"mailbox")

minetest.register_on_chat_message( function(name, message)
   
    local cmd = "/send"
   
    if message:sub(0, #cmd) == cmd then
   
        -- /send <player> <message>
        local data = message:split(' ',2)
       
        local player = data[2]
        local msg = data[3]
       
        if not player or not msg then
            minetest.chat_send_player(name,'Wrong params: /send <player> <message>')
            return true
        end
       
        local receiverref = minetest.env:get_player_by_name(player)
        if receiverref == nil then
            minetest.chat_send_player(name, player..' is not a known player')
            return true
        end
       
        local v = minetest.get_table_data(jachoo.mailbox,"string",player,"string")
        if v then
            v = v.."\n"..name..": "..msg
        else
            v = name..": "..msg
        end
       
        minetest.set_table_data(jachoo.mailbox,"string",player,"string",v)
        minetest.chat_send_player(name, "Message sent to "..player)
   
        return true
    end
   
    local cmd = "/mailbox"
   
    if message:sub(0, #cmd) == cmd then
           
        local v = minetest.get_table_data(jachoo.mailbox,"string",name,"string")
        if v then
            minetest.chat_send_player(name, "Mailbox: \n"..v)
            minetest.remove_table_data(jachoo.mailbox,"string",name)
        else
            minetest.chat_send_player(name, "Your mailbox is empty")
        end
   
        return true
    end
   
end)
Last edited by jachoo on Mon Feb 20, 2012 15:56, edited 1 time in total.
 

jachoo
Member
 
Posts: 45
Joined: Mon Sep 12, 2011 08:29

by jachoo » Mon Feb 20, 2012 15:58

+ important bugfix (env database sometimes did not save)
+ new Lua function: remove_table_data(int table, string key_type, key)

check out github
https://github.com/jachoo/minetest-jachoo2/
 

zoot686
Member
 
Posts: 11
Joined: Tue Jan 24, 2012 22:37

by zoot686 » Wed Feb 22, 2012 21:42

sqlite is nothing but a flat file(ie: plain text), character delimited database anyways.
 

celeron55
Member
 
Posts: 430
Joined: Tue Apr 19, 2011 10:10

by celeron55 » Tue Feb 28, 2012 17:36

Here are a few points I discovered by researching what this actually does.

It stores eg. auth and ban data as binary blobs in a database. I do not understand at all why you would want to do this. It does not make any sense and is just more difficult to handle by hand and external tools.

For clarification: The original reason why map data is stored in sqlite is because common file systems are not able to store as many files as was required by the original format. For that to be fixed, it was enough to just throw the blocks as blobs in sqlite.

As for commit messages, do you really think "asd" or "porzadki" would do? Nope.

Then there are random things like adding unrelated stuff to .gitignore. These kinds of things should be put onto a separate branch.

If you want to code things in a way that can be merged upstream, you *need* to discuss things first. If you want, you can come on #minetest-delta and, assuming there are people around, we can try to figure out what would be preferred in minetest of this and what is not.

I don't know what will come out of such discussion, but without it, large things like this *always* end up not good. With it, they have a chance to end up being good.
 

jachoo
Member
 
Posts: 45
Joined: Mon Sep 12, 2011 08:29

by jachoo » Tue Feb 28, 2012 20:03

Sqlite can (in general) modify strings for such things as encoding. There are no other differences between strings and blobs. That's why I used blobs. But they are still just plain strings.

Auth and ban data is stored in exactly the same way as it was in files. Lines in files = records in tables. Any sqlite tool should handle that easily.

About commit messages... My fault. I forgot to merge them into one commit. It's just some initial work.
 

Nemo08
Member
 
Posts: 132
Joined: Mon Dec 26, 2011 04:59

by Nemo08 » Mon Mar 12, 2012 11:35

any news?
 

User avatar
redcrab
Member
 
Posts: 831
Joined: Tue Dec 13, 2011 13:45

by redcrab » Mon Mar 12, 2012 13:51

nice initiative...
I like the idea to have one data model for the whole project.
but what I like less .. it is the idea to have the world to much tied to player .... but if management tools may be develop to have agile feature between players, privilege , protection, world ... that will be great ..
Last edited by redcrab on Mon Mar 12, 2012 13:54, edited 1 time in total.
0.4 for serious builder click here
Dedicated Minetest redcrab server forum at http://minetestbb.suret.net

It's nice to be important but it is more important to be nice.
 


Return to Minetest Engine

Who is online

Users browsing this forum: No registered users and 2 guests

cron