FOSS gamedev and creative worlds

Mod Search engine


Why?
It's a pain to search mods on this forum, therefore I added a fast way to get a topic link for a specific mod.
Also, it allows to filter the results or to look up all mods by an user.

Special
You can search for two mods at once by separating the keywords with a "|". (shown as in the image)
Also, there's a simple feedback system (Like / Dislike) which requires JavaScript.

Link: http://krock-works.16mb.com/MTstuff/modSearch.php

Suggestions and bug reports are appreciated.

That's a very good thing! It ought to be linked somewhere. Else this mod will get lost like all the other ones.

Do you execute a live search or do you parse the forum pages from time to time?

Sokomine wrote:That's a very good thing! It ought to be linked somewhere. Else this mod will get lost like all the other ones.

Do you execute a live search or do you parse the forum pages from time to time?

Good question. I wrote a bot to do this job, using an open source html parser library and some MySQL queries to add and/or update the mods.
I will run it sometimes a week for the 1st and 2nd page of Mod Releases and Modding General - maybe once a month for a complete update.

It will not have a DDoS effect because it only reads from the forum pages and not from every single topic.

Good work ! :)

Krock wrote:Good question. I wrote a bot to do this job, using an open source html parser library and some MySQL queries to add and/or update the mods.

That's good. I've tried to sort mods into categories on the Wiki, but doing so manually was so much work (tedious cut&paste) that I gave up. With a script that parses the forum automaticly, introducing categories becomes far more doable.

+1 for PHP-friends!

I used to use php a lot, and now I have resolved to never use it again.

rubenwardy wrote:I used to use php a lot, and now I have resolved to never use it again.

Why not?

It is a terrible language with lots of inconsistencies. Badly designed. There a lot of better frameworks that are faster to develop for and the result is less messy. Django (python) and Rails (ruby) being examples.

Yes for python. Very good language.

+1

Very good, i like it.

Update!
- Added AND and OR support in the keywords

How it works: (creative mode = true)

I think a list of indexed mods would be a useful addition to your page. Nice work BTW

Update!
- Bugfixes
- New keyword marker


Any suggestions?

I don't even try to use the forum search anymore. It should be on minetest.net

One bug: When you select Author and then go to the next page, it switches back to Title.

Casimir wrote:One bug: When you select Author and then go to the next page, it switches back to Title.

Thanks for reporting. A small typo caused this fail - Fixed.

It would be great to add a link to this in the forum at a prominent place so that players do get a chance to find mods.

Suggestion: don't make tiny annotations like this, it's very hard to read.
http://i.imgur.com/AMTBgP9.png

Update!
- Added games to the page
- Added a filter for games

Can you add a "Minetest" to the <title> of the page? It makes it easier to find the bookmark in your browser. "Mod search" is not as great as "Minetest Mod Search" if you do a search for Minetest in your bookmarks. :)

I just suggested a button for the Python gitsync tool could very easily be added to the lua forming the front page of minetest https://forum.minetest.net/viewtopic.php?f=14&t=8749 and wrote a scraper to extract all the git links from the forum and now find this.
Would it be posible to merge this with gitsync and add user voted tags for sorting. You'd have a far better 'modstore' then...
If not could you also use your search to produce a list of links to save from multiple scrapers working...

PS Also could one of the admin create a pinned post on the forum you could just keep updated with your list?

PPS could you also do the same for worlds/maps?

Well, I only index forum topics. I don't extract download links from every post.
I don't see how this could be merged with gitsync since the search engine uses PHP, MySQL and a handcrafted update script which has remote access to the MySQL database. Adding tags is a good idea but I must be aware of intentional spammers.

dgm5555 wrote:[..] list of links to save from multiple scrapers [..]

Could you please explain what you mean with this?

I could extend my program/script to fetch the data of a list if there are enough informations.
Yes, I could do the same for worlds and maps but there it's a bit complicated to ignore talk-only topics.

This is a great addition! thank you for making this!! Now we need to get all the modders to have a similar name convention and post in this forum!

dgm5555 wrote:[..] list of links to save from multiple scrapers [..]

I meant I have just run a python bot/scraper over the forum and extracted html links which point to git addresses ('list of links'). However if you're also doing (albeit for a slightly different purpose) the same it doubles the server load (thus the reference to 'multiple scrapers'). Combining this function into your tool would be much friendlier on the forum (and yours sounds better than mine as I searched every post in the forum).
Here's what my list looks like: https://forum.minetest.net/viewtopic.php?p=170199#p170199

The mod store is at http://forum.minetest.net/mmdb but it is currently broken due to issues with the forum.minetest.net database. The main develeper, iqualfragile, will do it soon now he knows what the problem is. (The problem doesn't affect viewing, only the adding or editing or commenting.)

It would be very useful to have a bot to extract the mod name, title, licence, description, GitHub or repo and download from the forum.

Experimental function added: (Only available for mods in the forum "Mod Releases")

http://irc.minetest.ru/minetest/2015-02-16#i_4157203

This is not a proper JSON file but bots should be able to read it.
If wanted, I can include a "view source" and "download" link if available.

If you search "Random" with a blank search feild you get this:

Napiophelios wrote:<snip>

Thanks for the report. The random function works correctly again.