Fetching Steemmonsters' game data into the MongoDB

I use a similar approach at Steem Monster Explorer where I fetch custom JSON operations broadcasted by @steemmonsters constantly. (small sleep cycles) MongoDB works well for such small cases where you need to store and query dictionary (document) based data in a quick way.

Workflow

---

1. Fetch account_history and filter custom_json transactions
2. Ignore the follow plugin on custom jsons. (follow, unfollow, mute, reblog)
3. Insert the json data into the local mongodb instance

Script

---

import json
import pymongo
from steem.account import Account
import time

def import_sm_actions(mongo_collection, op_count=None):

        counter = 0
        print("Fetching @steemmonsters' history")
        account = Account('steemmonsters')
        for custom_json in account.history_reverse(filter_by=["custom_json"]):

            # stop at the max. limit
            if op_count and counter == op_count:
                break

            if custom_json["id"] == "follow":
                # do not sync follow, unfollow, mute, reblog ops.
                continue

            action_data = json.loads(custom_json["json"])
            action_data.update({
                "action_id": custom_json["id"],
                "timestamp": custom_json["timestamp"],
                "trx_id": custom_json["trx_id"],
            })

            mongo_collection.update(
                {'trx_id': custom_json["trx_id"]},
                {'$set': action_data},
                upsert=True
            )
            counter += 1

        print(f"Total of {counter} operations indexed.")


def main():
    mongo_collection = pymongo.MongoClient()["steem_monsters_db"]["actions"]
    while True:
        import_sm_actions(mongo_collection, op_count=100)
        time.sleep(3)


if __name__ == '__main__':
    main()

Notes

---

• Depending on your query behaviors you might need to add some indexes. (Otherwise full collection scan might hurt query times.)
• This code is POC. I have an enhanced version at Steem Monster Explorer. However, that's a good start if you're into playing with these kind of data.