Utopian.info - Autocompletion and searching

Note: the changes in this contribution won't be live on https://utopian.info as I am completely reworking the application structure and it would break the website if I added it. If you want to test it out locally then make sure to clone the develop branch and not the master branch: https://github.com/amosbastian/utopian/tree/develop (instructions are in the README).

What's new?

The ability to search for stuff on Utopian.info has long been missing, so I added it! It uses typeahead.js for autocompletion, flask_wtf and wtforms for the forms and PyMongo and Flask for the actual searching and showing the results themselves.

Some of you may remember seeing search forms for each section on the home page in my last contribution. The basic functionality of these is now working, and I have also added another form to the header which allows you to search anywhere on Utopian.info (see image above). I will go over the changes below and explain how they were implemented!

Autocompletion

On the home page

To implement the autocompletion I used typeahead.js which made it very easy. Most of the code I used is exactly the same as the examples they give here. The only problem I faced was getting the data that is actually used for the autocompletion. To do this I added a context processor in the application factory that injects the data into the base.html template so it can be used by typeahead.js.

@app.context_processor
def inject_autocompletion():
    moderators = DB.moderators
    posts = DB.posts
    manager_list = [moderator["account"] for moderator in
                    moderators.find({"supermoderator": True})]
    moderator_list = [moderator["account"] for moderator in
                        moderators.find({"supermoderator": False})]
    contributor_list = posts.find().distinct("author")
    project_list = posts.find().distinct("repository.full_name")
    return dict(
        manager_list=manager_list,
        moderator_list=moderator_list,
        contributor_list=contributor_list,
        project_list=project_list,
    )

These lists each contain all the names of all contributors or projects on Utopian for example, and since they are injected into the base.html template it means they are available for use everywhere. To make them available to typeahead.js I simply added the following to base.html (there is probably a better way)

<script>
  var managers = {{manager_list|tojson|safe}};
  var moderators = {{moderator_list|tojson|safe}};
  var contributors = {{contributor_list|tojson|safe}};
  var projects = {{project_list|tojson|safe}};
</script>

which is then used in js/typeahead.js like so (just like the example)

$('#projects .typeahead').typeahead({
  hint: true,
  highlight: true,
  minLength: 1
},
{
  name: 'project',
  source: substringMatcher(projects)
});

Of course this has also been styled, which results in the following when searching for contributors on the home page for example

In the header

Of course you should be able to search everywhere on Utopian.info so I added a search form to the header as well. I really like the look of GitHub, so I tried making it similar to theirs. Anyway, since this allows you to search anywhere I made it so it autocompletes for managers, moderators, contributors and projects using another example they give that uses multiple datasets. As you can see in the GIF below it works really well

FlaskForm

For each search form I have created a class in forms.py using flask_wtf. To add this I followed this tutorial where everything is described in great detail. An example of one of the classes can be seen below

class ContributorForm(FlaskForm):
    """
    Form for handling the contributor search on the home page.
    """
    contributor = StringField(
        "Username",
        validators=[DataRequired()],
        render_kw={
            "placeholder": "Contributor",
            "id": "contributor"
        }
    )

    def __init__(self, *args, **kwargs):
        if "formdata" not in kwargs:
            kwargs["formdata"] = request.args
        if "csrf_enabled" not in kwargs:
            kwargs["csrf_enabled"] = False
        super(ContributorForm, self).__init__(*args, **kwargs)

Once a form like this has been passed to the template you can simply render it using the following code

<form action="{{ url_for('search.index') }}" method="get" class="header-search">
  <div class="header-search__group" id="search">
    {{ search_form.q(class="header-search__input typeahead") }}
    <input type="submit" style="position: absolute; left: -9999px">
  </div>
</form>

After submitting the form the data in it is sent to a Python script where the data can be used to return the search results.

Getting the search results

Of course once a user searches for something you should also return the results. When I first started implementing this I was thinking of using text indexes since MongoDB provides them to support text search queries on string content. In the end I implemented this with PyMongo's find() method by simply matching the given data with a regex.

def contributor_search(data):
    """
    Returns a list of distinct contributors that match the given data.
    """
    posts = DB.posts
    search_result = posts.find({"author": {"$regex": data}})
    return list(search_result.distinct("author"))


@BP.route("/contributor", methods=["GET"])
def contributor():
    """
    Handles the contributor form on the home page.
    """
    contributor_form = ContributorForm()
    search_result = contributor_search(contributor_form.contributor.data)
    if len(search_result) == 1:
        # TODO: Redirect to contributor's page
        return "{}'s page...".format(search_result[0])
    return jsonify(search_result)

Each form on the home page has its own route in search.py which then in turn uses another function to get a unique list of the names of e.g. contributors as seen in the code above. Currently it is very basic and checks if the query only matches one contributor or more than one. If only one contributor is returned then (to be implemented) it will redirect the user to that contributor's page, otherwise it will show a list of the contributors that matched the given data.

Since the form in the header allows the user to find managers, moderators, contributors and projects it simply uses the functions that were implemented for the specific forms to retrieve four lists containing the results. The results are then shown on a page, as you can see in the GIF below

Pull request

• Autocompletion + search

What are my plans?

The next feature I want to complete is fleshing out the search results page. I would like it to be something like GitHub's one where you would be able to choose between managers, moderators, contributors and projects, all with sorting and pagination of course! Unfortunately I have a pretty busy period coming up at university, so I will have to see how much work I can get done.

Contributing

If you want to contribute, then please read CONTRIBUTING.md for more information.

---

Posted on Utopian.io - Rewarding Open Source Contributors

---