Players disconnected if max_objects_per_block exceeded?!

[sorcerykid]

The past several weeks my server has been crashing with the following series of error messages:

Your phone or window isn't wide enough to display the code box. If it's a phone, try rotating it to landscape mode.

Code: Select all

2017-01-22 07:50:02: ERROR[Server]: ServerEnv: Trying to store id=2863 statically but block (-61,-2,-56) already contains 90 objects. Forcing delete.
2017-01-22 07:50:02: ERROR[Server]: ServerEnv: Trying to store id=2864 statically but block (-61,-2,-56) already contains 90 objects. Forcing delete.
2017-01-22 07:50:02: ERROR[Server]: ServerEnv: Trying to store id=2865 statically but block (-61,-2,-56) already contains 90 objects. Forcing delete.
2017-01-22 07:50:02: ERROR[Server]: ServerEnv: Trying to store id=2866 statically but block (-61,-2,-56) already contains 90 objects. Forcing delete.
2017-01-22 07:50:02: ERROR[Server]: ServerEnv: Trying to store id=2867 statically but block (-61,-2,-56) already contains 90 objects. Forcing delete.
2017-01-22 07:50:02: ERROR[Server]: ServerEnv: Trying to store id=2868 statically but block (-61,-2,-56) already contains 90 objects. Forcing delete.
2017-01-22 07:50:02: ERROR[Server]: ServerEnv: Trying to store id=2869 statically but block (-61,-2,-56) already contains 90 objects. Forcing delete.
2017-01-22 07:50:02: ERROR[Server]: ServerEnv: Trying to store id=2870 statically but block (-61,-2,-56) already contains 90 objects. Forcing delete.
2017-01-22 07:50:02: ERROR[Server]: Server::ProcessData(): Canceling: No player for peer_id=2954 disconnecting peer!
2017-01-22 07:50:02: ERROR[Server]: Server::ProcessData(): Canceling: No player for peer_id=2789 disconnecting peer!
2017-01-22 07:50:02: ERROR[Server]: Server::ProcessData(): Canceling: No player for peer_id=2947 disconnecting peer!

Evidently there a correlation between the static object storage errors and the invalid peer-id's, as they always occur simultaneously. So for purposes of debugging, I set max_objects_per_block to 5 and edited the function deactivateFarObjects( ) in src/environment.cpp as follows:

Your phone or window isn't wide enough to display the code box. If it's a phone, try rotating it to landscape mode.

Code: Select all

if( obj->getType() == ACTIVEOBJECT_TYPE_PLAYER )
{
        warningstream << "Attempting to delete Player SAO id=" << obj->getId( ) << " from block " << PP(blockpos_o) << std::endl;
}
else if( m_active_blocks.contains(blockpos_o) )
{
        warningstream << "Attempting to delete object id=" << obj->getId( ) << " from active block " << PP(blockpos_o) << std::endl;
}


Sure enough, after dropping my entire inventory at spawn and logging off, thereby deactivating the associated map block (-1,0,0), then objects within the active map block (-4,0,1) were also deleted:

Your phone or window isn't wide enough to display the code box. If it's a phone, try rotating it to landscape mode.

Code: Select all

2017-01-22 18:00:33: ACTION[Server]: sorcerykid leaves game. List of players: publicworks, parkdistrict
2017-01-22 18:00:33: ERROR[Server]: ServerEnv: Trying to store id=15 statically but block (-1,0,0) already contains 5 objects. Forcing delete.
2017-01-22 18:00:33: ERROR[Server]: ServerEnv: Trying to store id=17 statically but block (-1,0,0) already contains 5 objects. Forcing delete.
2017-01-22 18:00:33: ERROR[Server]: ServerEnv: Trying to store id=18 statically but block (-1,0,0) already contains 5 objects. Forcing delete.
2017-01-22 18:00:33: WARNING[Server]: Attempting to delete Player SAO id=19 from block (-4,0,1)
2017-01-22 18:00:33: WARNING[Server]: Attempting to delete object id=20 from active block (-4,0,1)
2017-01-22 18:00:33: WARNING[Server]: Attempting to delete object id=21 from active block (-4,0,1)

It would appear as soon as the max_objects_per_block threshold is exceeded in any single map block, then all remaining objects within the entire map are deleted as well. This affects non-static objects including PlayerSAOs. No warnings, no callbacks, no cleanup. This is of particular concern for TNT explosions, which can spawn hundreds of objects in a single map block.

The issue is complicated by several factors. First and foremost, once force_delete is set, then every object is queued for deletion even if the associated map block is still active. This is due to a flaw in the following conditional

Your phone or window isn't wide enough to display the code box. If it's a phone, try rotating it to landscape mode.

Code: Select all

// If block is active, don't remove
if(!force_delete && m_active_blocks.contains(blockpos_o))
     continue;


Clearly this should NOT be an "and" operation, otherwise force_delete from any prior iteration can override the active map block status in all successive iterations, resulting in unintended side-affects.

Secondly, even after obj->isStaticAllowed() is validated, the object can still be deleted regardless of whether it is static object or not. Perhaps this is intentional? I'm not sure. But in the very least, it would be prudent to short circuit whenever encountering a Player SAO. This should afford a reasonable sanity check, I would think:

Your phone or window isn't wide enough to display the code box. If it's a phone, try rotating it to landscape mode.

Code: Select all

// Ignore active map blocks and ignore PlayerSAOs
if(m_active_blocks.contains(blockpos_o) || obj->getType() == ACTIVEOBJECT_TYPE_PLAYER)
     continue;


This is a critical bug in Minetest 0.4.14, and is likely impacting many other servers.

I hope that an appropriate patch can be made available very soon. In the meantime, the workaround proposed above should suffice.

[paramat]

Just fixed https://github.com/minetest/minetest/commit/2ea60156437962d7d29d20606bf5d9189059f76b
https://github.com/minetest/minetest/issues/4067

[843jdc]

It seems that text written on signs are entities. I wonder if the text on some signs are deleted if the limit is exceeded?
I have not tested that. Or heard any complaints. I just raised max_objects_per_block to a high value as a workaround

[sorcerykid]

It shouldn't impact ordinary signs or even multi-colored signs (since those are just texture overlays). But it will definitely affect the objects attached to itemframes.

This is why you may have noticed on some servers that vending machines will randomly have blank itemframes for seemingly no reason.

[843jdc]

I'm about to hopefully find out. Signs & signs_lib are next on my list to examine closely to get working in my custom 'mod'. I have text hanging in the air as unknown objects haha. Out of everything that I have converted so far, only signs are left to work on. Once it works I will apply my changes, let my players do a quick check that everything works, then recompile the server for both my Linux host and the Fedora virtual machine using the latest git code and your patches.

I want to be running Windows 10 tomorrow night and using the virtual machine. Redis helps a lot for speed. 3 minute idle time then 2 minutes writing the map database to the slow HDD. I just noticed that someone applied a change to the MT code allowing unix sockets to be used with MT & Redis xD. IDK how much that will help but anything that speeds up the server and auth.txt lookups will help.

Too bad the player files are not in a database. CSV format would be great. Hint. Hint. Then I will have to convert 77,000+ player money files of less than 5 bytes each to a database. Copying 150,000+ tiny files between the virtual machine and my SSD takes forever :(

Yes. Getting off topic, I think. But IDK where to put this so....sorry.

[TenPlus1]

Entities like dropped items, mobs, sign text and itemframes add up and sometimes can exceed the default limit of 49 entities per block which can cause issue... Lately signs_lib mod has been changed so that empty signs do not add an entity, also TPS_signs mod (a fork) counts how many items are inside the block before updating the sign which also helps, and Mobs Redo API also counts entities before adding anymore mobs.

[843jdc]

TenPlus1 - I set "max_objects_per_block = 400" per your suggestion some time back to resolve "Trying to store id=# statically but block (x,y,z) already contains X objects". I haven't seen that error message since I made that change.

I moved your signs_lib mod and renamed the signs. That caused almost all, if not all, signs to have pink Unknown Object on them. Aliasing signs:text to the new name did not prevent the unknown objects from appearing. I then executed /clearobjects on a copy of my map (10+ million blocks) last night. After an hour, the unknown objects were gone. The signs and text on them seemed fine. Every cat was gone too haha I'm guessing that entities can not be aliased :(

Signs_lib is complicated. I tip my hat to you for writing it. After a few hours of looking at the code, I have a general idea of what it does but not any more than that.

I am about to backup the main server's map, apply my changes, and run /clearobjects on it. I'll let my players tell me if anything is wrong. If the kiddies can quit squabbling long enough to do that as I see them doing yet again :/

I'd like to find a way to tie in lag's anti-curse mod to signs. Too often young kids are finding offensive signs. Not good

[sorcerykid]

Thanks for the clarification TenPlus1. I wasn't sure how the signs_lib mod actually worked. It makes sense that the text must overlay the node, and that's only possible with an entity. Those could add up very quickly.