RE: Is Custom JSON the Future of Steem? | The one thing that has been going up on Steem in the bear market!
One thing that's missing from making custom_json a true killer feature is indexing. If you broadcast (write) a custom_json, that's all well and good. But how do you retrieve (read) it later?
One way would be to buy a subscription to SteemSQL, but that's 40 SBD per month.
Another way would be to stream the blockchain from a certain block height (the genesis block of your app). The problem is, if you ever need to rebuild the data after a few years of uptime, you're going to have to replay all of those blocks again.
Steem Engine NFTs can be used for this instead. You create an NFT, then declare all of the properties your app needs. Then you create instances of the NFT (objects) and assign values to those properties. Then you can query them from the Steem Engine API.
If you goal is just pure data storage/retrieval, it might be a good solution. Plus, if you can figure out how to make the NFT instances useful as tradable assets, that's even better.
I use the sql database :)
It is great but as you said a bit costly. Not much of an expert of the topic of indexing, but if it improves the data searching and retrieving that is great! Maybe we should look to implement this in some future hardfork, since jsons are now so wildly used and probably will grow even bigger.
The NFT options is something interesting, I was just exposed to NFTs with gerbers test game .... wasn't aware that you can do an unlimited objects with them. Looks like the NFT will open yet another door on the Steem blockchain.
I wouldn’t say NFT objects are unlimited. The number of NFTs is limited by having to lock another token, so that’s basically a lateral cost. Also, after defining three properties, there’s a cost to add more. So there’s a width cost.
In terms of having the blockchain itself do the index functionality, that’s a matter of scalability. If they define a dedicated plugin to index these ops, it will impact replay times, if that plugin is enabled on the node.
The cost of data indexing doesn't go away just because a different bit of code is doing the database calls.
Correct, so instead of your app indexing just your app’s data, the plugin is indexing everything. I’d imagine that kind of solution would benefit from a setting to tell it at least which ID you’re interested in, so that if you’re running a private node, the impact is less than someone running a public node that indexes all IDs.
Sure, that seems like a reasonable approach. Or maybe a plug in that allows push or pull of relevant IDs to support indexing externally without needing to stream the entire blockchain.
Like a dedicated p2p network? Yeah. We basically get that with Steem Engine, only it doesn't rely on a C++ dev to implement. Not that there's anything wrong with that.
I'm not sure what you mean by a dedicated p2p network. Steem Engine streams the whole blockchain right? I'm saying if all you want is a subset of data from a subset of custom_jsons, the right plug in would let you do that without the whole chain. The plug in would only need to be implemented once, then anyone (i.e. non-C++ devs) could use it. In fact Steem Engine could use it too.
Pushed to only registered IP addresses that are interested in those IDs. One way to implement this would be something like a p2p network. Although this comes with its own set of problems to solve (security, authentic inventory).
Is there documentation somewhere on how NFTs on Steem Engine work?
You can go trough @aggroed posts on NFT where he is explaining things..
https://steemit.com/steem-engine/@aggroed/nfts-going-live
https://steemit.com/steem-engine/@aggroed/the-first-nfts-have-been-created
Also check Steem Engine discord for question...
Thanks for the links.
https://github.com/harpagon210/steemsmartcontracts-wiki/blob/master/NFT-Contracts.md
Thank you!
I don't understand the problem with indexing. Why couldn't you store transaction id and block num into the application database and retrieve them fast? This is the way how taverngames works
I believe because the indexing is not decentralized. If you ever lose "the application database", you need to replay blocks since the application genesis to recover the index.
That in itself wouldn't be that bad. At least you have a recovery process. It should also be possible to checkpoint the database to a backup periodically and then you only have to replay since the checkpoint.
This is far more ideal.
I don't see a problem here. Why would I lose application database?
Yeah, I don't see the problem either. You got to be very sloppy to lose the entire database especially without backup checkpoints.
It happens or what is more likely to happen is corruption where you have to rebuild from source.
Yes. @smooth has right. This is what I had on mine mind. You can always restore the backup.
Yes, if your app broadcasts the op, on your server, that’s a solution. But if a user broadcasts it from their browser, it starts to get tricky. Does your app then send the block number to you? Did they even use your app? Or did they use SteemConnect/Steem Keychain? Did they use a third party app that doesn’t know it needs to report the block number to you?
In a sense, the blockchain is the method of communication, so it’s redundant to also use another protocol to report the block number. Ideally, you just monitor the blockchain for new data or just some other indexing strategy, not both.
Users use my app and steemconnect as well. When they broadcast the transaction they get the transaction id and block num, then they send it to app endpoints.
Users don't use a third party apps.
If you said it's redundant to use also another protocol and blockchain is the method of communication why do you need SteemSQL then? It is also redundant.
Btw storing the data in the app application makes you independent, you don't need to count on someone's app like SteemSQL (what if they will disable it?)
You do the indexing on your side. I don't see there any disadvantages.
Also you don't need to stream all transactions from the blockchain only those that you really need.
If you accept requests where your users report block numbers, you’re bypassing the need to index independently, which is a great solution. If your app is designed to only accept requests this way, you’re golden.
The problem with this method is if you miss a block number being reported from your users, you need some strategy to find the stragglers. Whatever process you use, assuming it isn’t a manual process, why don’t you always use that strategy instead?
Well, technically, if you’re streaming the blockchain, you are getting all transactions. You’re ignoring most of them, but that’s at least 500 MB of bandwidth, daily, last I checked.
Streaming works. But you need a strategy for knowing what the last processes block number was, so you can ensure you didn’t miss a block if your stream is interrupted.
By the way, if you have several apps all streaming independently, you should consider:
https://github.com/inertia186/meeseeker
Yes you are right.
Btw my app works in two way. Sometimes accept the requests from users (during gaming) but I also need to stream the blockchain (for shop purchases, yes they are not tracked as you mentioned because steemconnect doesn't return with the trx id and block num).
I have the complex solution for streaming blocks, which also detects missed blocks based on previous block, also It retries to preprocess the block when some block is not yet available through the API.
I can agree that all the things depends on the application specification.
I'll take a look at this github repo! Thanks :)
I also invite you to checkout how it works in the https://taverngames.io :)
What happens if the user's client crashes in between the time they broadcast the transaction and the time they send the id and block number to the app?
What happens if there is a chain reorg and the transaction ends up in a different block?
For custom jsons it doesn't matter, user doesn't lose his STEEM only some RC amount so it can be retried.
I've never noticed the chain reorg during custom json processing. I've processed many of transactions and no issues reported.
Transfer transactions are streamed from blockchain in the background process. So I never lose them.
Come and checkout : https://taverngames.io
Chain reorgs on Steem are usually rare, and usually only one or two blocks (so resolved within seconds), so you may not see them much. But sometimes they do still happen and sometimes take much longer to resolve, so you need some strategy for handling it.
If your app expects the user to retry or accept lost transactions occasionally, that may be a perfectly acceptable strategy.
Yes it is. Everything works as expected.