There are no "formal" mob api's in Minetest - that's left to the modding community rather than an engine or core-game part.
There are currently (as far as I am aware) 3 popular mob engines:
I think the current popularity of mobs_redo is in part because if you want to just focus on modelling and texturing, but no coding, it's the easiest choice; a lot of the basic AI for movement and hunting down the player is dealt with. I tried having a quick look at MOBF and the documentation is quite complex; I have not looked at CME in a while.
That said, it doesn't necessarily have the features you describe. Hunting in packs and establishing settlements is outside of its scope.
If you were to define custom routines of your own, you could feed them to mobs_redo in the do_custom, or append them to mobs_redo in a forked version of the code.
For the other engines, I cannot say I am sufficiently familiar with them to tell what you could or could not do with them.
Arguably, there is little you could not-do if you coded the AI yourself, but that would then be a separate engine of your own.