Donating 5000$ and raising money with this post for covering some of SteemData costs + server upgrade. (by @Furion)
I commit to personally give 5000$ worth of STEEM or BTC at his convenience and 100% of the reward from this post and below comments to @Furion to cover server costs for hosting SteemData. Will also request from him some kind of proof of payments to verify costs.
What is SteemData?
SteemData helps developers and researchers build better STEEM applications. We parse the STEEM blockchain for you, and provide the data as a fast and convenient MongoDB service.
Here is the last update he made about SteemData
SteemData is currently available in a limited scope.
The following features are not available at the moment:
AccountOperations(Account History /w virtual ops)- Additional
AccountsData
Issues
Hardware
A full steemd node /w high-throughput is required for SteemData to function properly. It needs all plugins and all operations enabled, so that it can construct account history and store all the virtual operations alongside the operations stored on the blockchain.
Further, the node has to be in the same datacenter as SD to handle the required throughput. This is because the usage of Steem blockchain is growing, and re-syncing all the affected state requires over 100 requests per second. The private network latency within a data center is typically below 1ms, while a public network latency is usually 10 fold or more. This would decrease SD's throughput significantly.
Unfortunately, the datacenter where SteemData currently resides only offers servers with up to 256GB of RAM, and a steemd node configured to SD's requirements needs more than that.
Also, the SteemData MongoDB server is running out of disk space :(
Rusty code
I have patched things up in a quick-and-dirty fashion for a couple of months now, and the SteemData codebase has gotten a bit messy. I see this as an opportunity to clean things up, and improve reliability/performance.
Further, this is an opportunity to add infrastructure support and documentation, such that anyone can spin up their own SteemData cluster.
What needs to be done?
I am currently speccing out the new infrastructure. A new cluster will be setup in a different datacenter, capable of provisioning servers with NVMe SSD's in soft & hard RAID configs, and up to 512 GB RAM.
I should to test various steemd configurations, to achieve desired performance and provision the appropriate hardware for the next 6 months of operations.
The new DB server will also benefit from faster SSD's and larger in-RAM cache (currently 30GB, will be 60GB or 120GB).
Creating a replica set would add resilience and decouple BC processing from the database, freeing additional CPU cycles for queries. Further, adding replicas that are geo-distributed would allow for low latency in-app integrations worldwide.
As mentioned before, this is an opportunity to upgrade the operational side as well, with automated provisioning and monitoring/remediation improvements.
TL;DR: Wishlist
- Clean up the codebase
- Infrastructure as code
- Documentation
- New Servers
- Multi-Replica DB Cluster
Thank you
SteemData is a popular choice for indie developers and power users.
Steem is growing at a rapid pace, and its daily on-chain state throughput is making Bitcoin and Ethereum look pale in comparison. To scale SteemData I'm looking at the approx. recurring server costs of $3,000/mo. This would not have been possible without witness pay. Thank you for supporting my work, and thank you for using SteemData
-- @Furion
Thank You.
Contribution comment #1
Contribution comment #4
@transisto and @furion : what about data sharding the blocks chronologically and load balance from there? Of course I have no insights in steemdata's exact loads, but aren't the latest blocks queried most? If so, a relatively small chunk of data consumes most CPU / RAM but only a fraction of SSD, so by sharding "new blocks" (MongoDB can shard by default) on a separate node splits the SSD vs CPU/RAM/bandwidth problems.
Just an idea! ;-)
@scipio
EDIT: small self-upvote for visibility, 100% upvoted "contribution comment #4"
Scaling MongoDB is currently not a problem. The database would need to grow by another 1000% before sharding becomes relevant.
I think this is a very good in Amazon forest Amit user community don't forget to Upvote
Good Work Follow x Follow please <3
hola aca saludos desde venezuela me sigues te sigo :$
czechglobalhosts
Contribution comment #7
give me vote and follow i also give you vote and follow
Thank you !
Contribution comment #8
https://steemit.com/free/@safetony/free-crypto-kitty-giveaway
contribution comment #9
#Wow
Thats a great post,,
i am a big #fan of you,,
keep it up,,
i always #support you
Thank you @transisto
wow so sexy to see women involved in actual programming and other stuff like this, blockchain development you could call it, it is just so nice to know there are women who wont get bored of this stuff and who can find value in men who do stuff like this all day, and the amount of income you can generate from having these sorts of skills is becoming greater and greater, we are becoming wealthy by helping to build, or support the building of these new blockchain tools, which help everyone and become a great service to humanity. Like how Google has been paid billions for its work but the contributions of Google might measure into the Trillions if one tried to measure all of the economic activity Google has allowed for. Its almost like when we have women who support men who work on steemit, morally, and sexually, we then end up having GREAT motivation for men to work harder! if a woman rewards her boyfriend with sex everytime he gets a lot of upvotes or maybe does some development work, makes a post showing his progress on his new project, and his girlfriend can reward him with a sex upvote, it is a fumy concept we could turn into a comedy sketch and I bet @fyrtsikken would enjoy the idea, a woman who only has sex with her boyfriend if he gets enough upvotes on steemit, as a way to encourage him to work harder)
VERY exciting to see attractive women on steem wyen it is all so new! Women arebadly needed in crypto to encourage and motivate men to work harder! There is a LOt of slackers in crypto who just sit on their bitcoin and dont do any productive work besides bag holding , which is arguably productive but honestly we can still do work WHILE we make money on Bitcoin. If someone is just only making money off their bitcoin, and not doing any otehr work, that is just as lazy as a bum, theyre just a bum ust a wealthy bum! peopel should use their free time that Bitcoin has given them and PRODUCE things like at LEAST make some art if you cant make anything tangibly useful! There is ALWAYS a need for more entertainment content but your art has o be GOOD like @elgeko and his video cart, it must become a product people cal easily sell all ovr tje world!
Ok it was inspiring seeing your sexy picture! keep the selfies coming we need MORE women on steem so they can get upvotd and atract more guys and otehr girls to the platform!
i checked your youtube channel but i think the link is wrong coz theres a dude talking
haha nah it will be right. It is surely a dude trying to make some coin.
Feeling you bro. It's the way out though. No time for dulling
Let me follow you up ASAP
why are you voting for urself? this platform is fakeland
If you like the book, it is a request for reading
https://steemit.com/book/@saifuk/dan-brown-origin
Contribution comment #5
Contribution Comment
Contribution comment #6
what is this???
genau das frag ich mich auch. :-)
Auch ein deutsch-sprechender Mensch hier!
haha ich auch...
hallo! ich spreche deutsch sort of
hi.... Iam beginner in steemet follow me @rijal10 i want to share many things here and please follow me! please participate
and upvote this @sweetsssj
https://steemit.com/steem/@rijal10/how-to-register-vip-acount-bitcoin-co-id
great , follow me please
Stop asking for followers, read the post and actually contribute. Just look what happened to your rep!
Good game. Top comment. I'm so confused by this platform, but that can be a good thing...
hi.... Iam beginner in steemet follow me @rijal10 i want to share many things here and please follow me! please participate
and upvote this @sweetsssj
https://steemit.com/steem/@rijal10/how-to-register-vip-acount-bitcoin-co-id
Did you even read my comment?
Dear friend
steam world you have successfully opened the edge. And you have successfully top level reached, this is my wish. And I am glad this is my practice in your development.
My request to you, I am like a small child in the steam world unknown and withut benefited.
You must know
"People for people"
So went my way in the world, steam your collaboration service.
Please do follow me and help and thanks to me
Dear friend
steam world you have successfully opened the edge. And you have successfully top level reached, this is my wish. And I am glad this is my practice in your development.
My request to you, I am like a small child in the steam world unknown and withut benefited.
You must know
"People for people"
So went my way in the world, steam your collaboration service.
Please do follow me and help and thanks to me
Dear friend
steam world you have successfully opened the edge. And you have successfully top level reached, this is my wish. And I am glad this is my practice in your development.
My request to you, I am like a small child in the steam world unknown and withut benefited.
You must know
"People for people"
So went my way in the world, steam your collaboration service.
Please do follow me and help and thanks to me
SteemData looks interesting might check it out on some future project. Not been 100% sure of current APIs.
@transisto and @furion So many great developments on the steem blockchain, and this is a great one to get more smart and creative developers on the block. I just upvoted both the post, and this comment.
By the way @transisto , I made a reply in your previous post of the size of a normal post, and your feedback is very important to me.
Here is a link to your previous post and my comment.
https://steemit.com/steem/@transisto/my-version-of-steem-is-not-content-based-it-is-contribution-based#@gold84/re-transisto-my-version-of-steem-is-not-content-based-it-is-contribution-based-20171230t142131356z
Looking forward to hear from you!
Regards, @gold84
yeah man there is so much demand for new deve;opers and great upvpyte potential for them that we should make a special front end just to post steem development projects and it will have a list of supporters, peopel who delegate steempower or curate themselves to peopel who post godo proejcts, kinda like @utopian-io for dvelopers, it would be really nice, it can allow steem to have a WHOLE github like system with slack as well, we should REALLY make the STEEM BROWSER and have a whole Steem Giuthub type website where tseem posts temselves hold the code, and just replace github repositories with steemit pots and comments
@ackza I totally agree with you! Thanks for adding this great idea to the message. Hope to hear from you more oftern on my post ideas/chapters. You have a lot to share there.
I am not sure if you know I am completing a full 11 posts/chapters steemit guide for new visitors, new steemit users, minnows, for them to learn as much as possible and as easier they can about steemit and the steem blockchain, to make their way here. In my already first 2 chapters published I got lots of questions, comments, ideas, and I believe it is helping them.
Looking forward to your thoughts when you get a chance.
Regards, @gold84
with this guys supporting steemit, steemit can only get huge!! we believe it
What is this contribution thing ? Guys
Explain me please
He wrote that all post rewards goes to funding the project including rewards from comments . So if I want to support them with more than 1 full upvote, I can both upvote his post and one or more of his comments. That's why.
Next time I'll stack them all like this so no random comment can fuck up the order.
Explanation for high upvote
Even if you stack them like this, this does not guarantee that others will not comment and break the order of your comments. And I guess the last comments won't be visible in the same page so there are sone disadvantages...
#1
#2 ...
I'm a bit confused too I guess you just have to upvote it. Nice to see people investing and growing the community as this is what I'm here for :)
Good innovation upvoted
contribution comment #1.5
contribution comment #1.55
great , follow me please
Okie lets do it...
My Contribution Comment :)
please up vote my posts as u have strong steemit power
Bro, how are you able to get this huge upvote. I am new on steem can you please show me the way?
Just for upvotes you can use @booster bot
How do I know it worked. Is it @boosterbot
Or
@booster bot
If you like the book, it is a request for reading
https://steemit.com/book/@saifuk/dan-brown-origin
Using the same comments that has nothing to do with the post over and over again may give you some reports. I can see that you mostly write good comments, so I wont report you this time, but please dont spam
Contribution comment #21
Off😂
@transisto It's such a great pleasure to be part of this contribution. I strongly believe the turnup is gonna be massive.
@tomsnoob
Contribution, contribution... Let`s go working
my last post contains a free gift Happy New year everyone!
Thanks...!
great
Comentario de contribución desde Venezuela, Excelente (Y)
I didn't understood what is the work but still if you agree to do something good for steemit community .. this is worth commenting 👍
My last post contains a gift, Happy New year everyone!
Great job.
give me vote and follow i also give you vote and follow
voted and followed
Dont follow people that ask for followers without contributing to anything. He wont be a loyal follower, and he is spamming. The comment is copy paste
Contribution comment #22
give me vote and follow i also give you vote and follow
Contribution comment #2
Contribution Comment #2
I love you !
Contribution comment #2
Keep up the good works..power!
Dear friends, as my friend had taken the world steemit be happy
contribution comment #2.5
Contribution comment #3
contribution comment #3.5
Your writing is really a beautiful and clean mind telling you, your mind is like a great mountain and sea as a liberal.
Man's love for the people you spell cost whether you may be long live.
DEAR FRIEND THAK SO MUCH
Great initiative!
I'm planning (as already mentioned in my last post) to provide some benchmark results for different node configuration to help make better decisions regarding infrastructure needed to run various Steem related services (so it would be for both consensus and API nodes and later whole infrastructure supporting it)
Server(s) costs itself even over-scaled and powerful enough to handle all expected load are not that big compared to infrastructure features such as HA, (redundancy, load balancing, self healing) etc etc and of course human labor.
Currently I would suggest using smaller servers with specialized steemds (consensus node for handling subset of calls, then account history (ops can be whitelisted differently for different nodes and different use cases) etc etc) Don't throw requests than can be handled by consensus nodes at your full nodes.
nice topic
Maybe this has been asked somewhere already...
Why does Furion actually need more help with server costs? He has been one of the top witnesses for a while now and as one in the top-20, he is earning 7000-8000 SP per month. At current prices (~$2.85 right now), that's $20,000+ per month vs. ~$3000/mo. renting the equipment. Break-even on the new projected costs would be a $0.50 STEEM price.
For a few thousand dollars, you can buy your own 512 GB RAM server. Cash out a few thousand STEEM from witness rewards just from this month and he can buy two.
Not too long ago, Steem was at 80c.
You're right, I should just power down some Steem to cover the costs. I do however feel bad whenever I do that.
SteemData is well funded given the recent price increase and my witness position. I would like to forward the funds raised in this campaign to other Steem projects.
There’s no need to feel bad for powering down, especially since the witness rewards are received as SP. It’s the only option.
And just to clarify - I get the costs of your project. I just didn’t understand why ~$1000 of post rewards were needed when witness rewards are 20+ times more than that...per month (at current prices). If you cashed some out tonight, you could potentially get about $5 per STEEM. Seems like a great deal to me!
Good luck with your new servers and upgrades! I have a feeling that most of us will be upgrading witness specs again soon with STEEM prices continuing upward.
Your writing is really a beautiful and clean mind telling you, your mind is like a great mountain and sea as a liberal.
Man's love for the people you spell cost whether you may be long live.
DEAR FRIEND THAK SO MUCH
That's why I think, that steem does not scale.
512GB Node ..that is crazy. No one, except rich or already SP rich witnesses could afford running this
STEEM goes further and further away from decentralization. heading into a 20 Witness "diversivied" centralized system
Currently my most powerful nodes has 64GB of RAM.
Yes, steemd is very demanding. No, it does not need that much RAM. Optimizations are on their way (easy to spot while following github). Eventually, what we need from steemd itself works fine with 16GB of RAM.
What we are talking here is the infrastructure to provide SteemData service.
Servers are cheap. Smart humans are expensive, but worth their costs.
Your writing is really a beautiful and clean mind telling you, your mind is like a great mountain and sea as a liberal.
Man's love for the people you spell cost whether you may be long live.
DEAR FRIEND THAK SO MUCH
Steemdata is a backend that powers down stream applications. It will have processing and throughput needs over and above that needed to maintain a blockchain node. I’ve used the api and written big queries... it needs an upgrade
like getting a good post
Your writing is really a beautiful and clean mind telling you, your mind is like a great mountain and sea as a liberal.
Man's love for the people you spell cost whether you may be long live.
DEAR FRIEND THAK SO MUCH
I know..I am good!
Chers!
Hasn't he already crowd-funded some equipment before/
I’m not sure about equipment, but there was quite a few thousand allocated via posting rewards for SteemQ, which then turned into Viewly. And last I saw (3 months ago), there was to be an ICO for Viewly. Haven’t seen or heard anything about it since then. Furion’s social activity on the blockchain has only been one comment in the past two months...and that comment was worth $190+ when I saw it earlier today.
Your writing is really a beautiful and clean mind telling you, your mind is like a great mountain and sea as a liberal.
Man's love for the people you spell cost whether you may be long live.
DEAR FRIEND THAK SO MUCH
Your writing is really a beautiful and clean mind telling you, your mind is like a great mountain and sea as a liberal.
Man's love for the people you spell cost whether you may be long live.
DEAR FRIEND THAK SO MUCH
Your writing is really a beautiful and clean mind telling you, your mind is like a great mountain and sea as a liberal.
Man's love for the people you spell cost whether you may be long live.
DEAR FRIEND THAK SO MUCH
Dear friend
Like getting a good graphics
I announced you
you are a The world famous grapher
Dear friend
Like getting a good graphics
I announced you
you are a The world famous grapher
A single person running SteemData handles System Administration, Database Administration, Network Administration, Development, Quality Assurance, Testing, Planning, and Customer Service.
As a systems administrator and senior developer I was making over $100k/year. At the prices you detailed above, that puts the income around $250k/year. If you don't feel like you are worth $250k/year, or if you think anyone making that much should always spend their own funds and never get help from outside sources, then thats your opinion.
Maybe you just don't realize the amount of time, work, and effort this type of business requires. The upkeep, the research, the optimizations, the idiot users that fuck things up, etc.
Furion isn't just hanging around https://steemit.chat all day. I think he deserves well over $250-300k+/year
Then pay him the money you think he deserves. The difference between this blockchain and a job as various tech administrators is that those jobs are typically paid by a company with revenue streams from their products and services, or from a third party who needs the particular administration services.
If furion needs or deserves $300,000 a year, then perhaps he should work on monetizing his services rather than relying on charity from the limited daily reward pool. From a business perspective, that is what would make sense when it comes to expansion and covering additional costs, not a bunch of “contribution comments” on a post.
And since you’re up-to-date on what furion is doing on the daily, maybe you can tell us what has been happening with Viewly and its ICO? Or maybe let furion know that he can always drop an update on us when he’s not working at his eight SteemData jobs.
He is getting what he deserves. You are claiming he shouldn't. Just salty, I think.
But hey, it is quite alright that we have users posting 10-20 times a day getting $300 on each while not giving back to the community so let's just attack developers that are actually hosting servers and developing and maintaining them.
If you feel this post is overvalued, or any other post, you are welcome to use that downvote button and make that clear.
Now ... perhaps you have some kind of personal issue with furion. To that, I can say or add nothing to. In fact, I am unaware of anything about Viewly ... however, I will likely look into it a bit since you bring it up here.
Hi, our Telegram is always open for everyone to join. https://t.me/viewly . Thank you for your interest and kind words
wow excellent post
Your writing is really a beautiful and clean mind telling you, your mind is like a great mountain and sea as a liberal.
Man's love for the people you spell cost whether you may be long live.
DEAR FRIEND THAK SO MUCH
Why not support SteemSQL in a similar way?
I don't think this is an either or type thing.
Both services are great and I personally have used both.
That being said, SteemSQL is a fucking beast and thank you for the service.
Your writing is really a beautiful and clean mind telling you, your mind is like a great mountain and sea as a liberal.
Man's love for the people you spell cost whether you may be long live.
DEAR FRIEND THAK SO MUCH
Your writing is really a beautiful and clean mind telling you, your mind is like a great mountain and sea as a liberal.
Man's love for the people you spell cost whether you may be long live.
DEAR FRIEND THAK SO MUCH
SteemData is really great and I am using it for some features for dMania.
Having more options to query the Steem blockchain makes a lot more features and apps on Steem possible. That is why a service like this is very important.
Lately it was lagging behind and the data wasn't up to date sometimes. I guess it was because of the limited hardware resources.
If SteemData is reliable and all the data is up to date, I think it will become crucial for the growth of Steem and we will see a lot of new great apps and features.
@Furion don't forget to add more indices for comments 😅 I really need them.
I agree with you we cultivate
Hi there @zombee I have been trying to post more on dmania so does that mean like I m doing something wrong ? I feel I can make people laugh. So I try to make more post ... Then today I realised not many people are looking into it .. n my post promotions went worthless ... So I m not sure should j make more post or just a few
dMania is still growing. Every post and contribution helps us grow. Thanks for your support.
& truth be told I m having zero knowledge of coding .. I can barely understand what to do with these kind of post ... But I try to searc keywords mentioned in these .. n m reading everything line 12 hours a day
Your writing is really a beautiful and clean mind telling you, your mind is like a great mountain and sea as a liberal.
Man's love for the people you spell cost whether you may be long live.
DEAR FRIEND THAK SO MUCH
Contribution comment
This is some good stuff. I'm glad to hear that we are improving the infrastructure used by the tools used by the community.
Awesome. Voted!
interesting! I want to use SteeemData later