Added text search ordered by relevance score for UtopiansteemCreated with Sketch.

in #utopian-io7 years ago (edited)

This is a development contribution report about the newly developed multiple-word text search for utopian.

I am very grateful to have been able to take part in this action made possible by @elear and the Utopian community.


contribution3.png

When I offered to take part in this, Elear suggested that it would be cool if I would do it together with @codingdefined. So I started to talk on Discord with @codingdefined and then started the development yesterday, each on his fork of the api.utopian.io repo, but we were in contact over Discord to discuss the project in the meantime.

I had some experience with Mongodb before this, but only for basic finds, edits, deletes and such, nothing fancy.

Yesterday I researched for many hours and read quite a few articles about the different types of search available for Mongodb/Mongoose. I tested some possible solutions locally and eventually realized that the text search with added indexes and ordering by relevance ($meta) score was the best option for this task.

I researched the repo to understand how to integrate the search better and do away with the regex search as @elear suggested.

I managed to keep most of the existing structure intact. I stayed very late yesterday and had made the search work as I wanted, but I didn't want to make a PR so late.

Today after I did the PR, me and @codingdefined started to test and see that it behaves as expected.
While testing @codingdefined found an unrelated bug on the live utopian site and he reported it.

pull-18-1.png

Cool, so what does the new search do?

Well, with this we can search multiple words at a time (regex was bad at this), and the results are ordered after a $meta match score (relevance). This relevance score is also broadcasted by the api.utopian.io when making a search, so we could highlight the ones with a really high score.

Notice the score being broadcasted below:


utopian-search-score.png


pull-18-2.png

We can set weights for the body of the text and the title. Right now title has 5 and body has 2. So a post that has a searched word in the title is more relevant(has more weight) than a post that has the word in the body.

pull-18-3.png

Also we can search for exact phrase if we put the desired text between quotes, or we can exclude words if we put a minus in front of a word.

pull-18-4.png

Here is the pull request:
https://github.com/utopian-io/api.utopian.io/pull/18

Going ahead I am investigating how I can add aggregated pipeline text search in order to limit the results broadcasted by a $meta score threshold. So that we only broadcast relevant entries, and not somewhat relevant or barely relevant.



Open Source Contribution posted via Utopian.io

Sort:  

Hey @sirrius I am @utopian-io. I have just super-voted you at 37% Power!

Suggestions https://utopian.io/rules

  • Your contribution is less informative than others in this category.

Achievements

  • I am a bot...I love developers... <3
  • You have less than 500 followers. Just gave you a gift ;)
  • Seems like you contribute quite often. AMAZING!
    Up-vote this comment to grow my power and help Open Source contributions like this one. Want to chat? Join me on Discord https://discord.gg/Pc8HG9x

Approved in Utopian. Thanks @sirrius

[utopian-moderator]

Thank you also for everything you are doing for the community.

Sirrus, will utopian.io allow more precise and detailed searches in steemit posts? I need to read everything about, but so far is a little dificult for me!! I sendi a big Hug and always the best vibes!! :)

Hey @leveuf . Utopian is build on top of the STEEM Blockchain like Steemit, but it is limited to only one category utopian-io.

Doing this it can set some rules about the posts and the open-source contributions it supports. Steemit is not limited by category. So the utopian search is only relevant to the accepted open-source contributions posted through utopian.io website.

Steemit has a search function powered by Google. Maybe you can use that in your searches. Have a great day ahead :)

I've used it but with just random results, and many low quality posts!! I'm talking with other utopian about wikiversity, that sounds very interesting, have you heard about it?? Great day for you too!!! :)