Explaining the STEEM Blockchain Freeze of July 2nd, 2018
Here's a quick video update of what happened last night when the STEEM blockchain froze.
Getting three text messages all at once was certainly a freak out moment!
Finding new error messages in the logs I've never seen before was also quite interesting:
I was quick to jump into the Steemdevs Slack:
As well as the #witness steem chat:
I'm very thankful @andrarchy was available to send out this tweet to our community:
As I mentioned in the video, the witnesses and Steemit, inc did a great job working hard to get this chain back up and running as quickly as possible. Improvements I'd still like to see from Steemit, inc include a professional site down page that is immediately in place (instead of a generic 503 Error Response page) along with a status page people can go to with regular updates in real time as to what is going on. Some of this was mentioned 8 months ago. That said, they did Tweet this time and stay in close contact with the Witnesses as things were happening in real time. That's a big improvement.
Thank you all for watching/reading and for your support of my witness account!
As of this post, I'm currently position 10!
If you want to know more about what witnesses do check out:
You can read my witness announcement post here:
My previous witness updates:
- EOS Is Alive! Object Computing + STEEM? (June 14th, 2018)
- Updated lukestokes.mhth Witness Account Delegations (January 6th, 2018)
- Witness Delegation Plans for lukestokes.mhth (December 6th, 2017)
- Witness Update - lukestokes.mhth - December 1st, 2017
- Witness Update - lukestokes.mhth - October 26th, 2017
- Witness Update - lukestokes.mhth - September 2nd, 2017
- Witness Update - lukestokes.mhth - July 19th, 2017
- Witness Update - lukestokes.mhth - July 14th, 2017
- Witness Update - lukestokes.mhth - June 19th, 2017
- Witness Update - 6/6/2017 - More Human Than Human
I only post witness updates when I have something worth sharing, but I'm always available if you have a question or concern. I can be reached on Discord or Rocketchat.
What an intense and draining night it was. So many witnesses had personal problems, traveling, in hospitals. I had a damn leaky pipe at 3am (when I discovered it), luckily it didn't burst. I had to cut off the main water valve for the house, then went back at my computer to follow the situation. Shitty times. Yet, despite everything, we did our best to be around to keep the blockchain alive. Teamwork at its finest.
Thanks for the information. I hope the platform is secure and stable over the long term. Questions are being raised about EOS, but I hope that's just a case of EOS being new with the initial kinks needing to be worked out.
So, the blockchain froze as the consensus was lost. This is a good thing then, which means that it did what it was supposed to do, right? Except the fact that buggy part of the software did allow the input of wrong value the rest went as it is supposed to……………. that’s good to know. Maybe you are one of the few witnesses that I follow so this the only post in my feed that gave me a level of assurance as to what went down wasn’t anything lethal.
Although apart from the general gist, I have really no idea about the technical differences between a replay and a restart. Except maybe that replay lasts for hours while a restart can be a matter of minutes. I have heard that the unlucky ones had to do a replay 😊. I never knew that missing the blocks maybe out of your hands then (I actually thought that it had a lot to with guys being lazy)
Thanks for sharing all this man. I spend quite some time and energy on Steemit so had a bit of scare, not much but it was compounded by the events of my personal life. I hope you drop by my latest post about my experience of the scare.
PS: Nice green hair 😊. Just a few days I ago I gave my first witness vote to you. Second was for @dragosroua. I’ll take a little more time with rest of the twenty-eight.
Thank you for being available all the time everywhere and helping out with information flow. Yes, response from the community in a crisis mode was absolutely amazing.
Thanks for the update. Communication timing is great.
Green is often spill-over from the background being too close or too brightly lit. If you can push it back, that is easiest. Otherwise, lowering the green lighting but keeping the background evenly list is tricky.
Huobi announces suspension of Steem deposit and withdrawal?
Well done, Luke.
You were the first witness I voted for, and I know I made the right decision. Thanks for staying up all night, fixing thing and reporting back promptly.
While your bedroom looks like it just survived nuclear winter, I'm glad the Steem blockchain had a far easier time of it. ;)
I was freaking out... not really but didnt made any post yesterday.
For several days now I cannot upvote any post or comment without getting an error. Have you seen this before? Do you know if anyone else is experiencing this?
Thanks for the explanation of what happened last night. I though maybe it was connected with the issue I'm having with upvote errors, but it was a bigger system issue.
If you can help me with why I can't upvote, I will then be able to upvote this post!
If you're getting the slider before the error: find something you want to upvote 100%. Drag the slider down a ways and then back to 100, then vote. After that you should be able to vote normally.
If you're not getting the slider I don't know that I can help; when it happens to me I glare at it for a while and eventually get the slider.
It's not related to yesterday's issues; @m-sargon has had this off an on for a month, and it hit my own account once.
Also when it doesn't work on Steemit I've been able to vote through Busy.
Perfect! I'm back up and able to upvote again. Thank you!!!
What error? Would help to know what it is in order to know what problem you’re seeing. You might also want to log out and log back in again (maybe you’re using your memo key instead of your posting key)?
I have only ever used my posting key and it's saved in my password keychain on my iMac. Below is a screenshot of the error. Thank you for taking time to respond to me.
See the reply from @tcpolymath which may help. Looks like it may be related to this? https://github.com/steemit/condenser/issues/2926
Yes, that's the bug I'm seeing.
well, thanks for all the hard work
I saw the blockchain down and as part of a witness team my first call was to our tech genius to let him know as well. As we are only around place 100 with @swisswitness we only missed one block but indeed the number of missed blocks is not always a way to see how good one is as a witness. But the pressure of being one of the top 20 is something else but then the rewards are also better...
I guess as witnesses the only thing we have to do is be vigilant and stick together to keep this blockchain up and running.
liking the hulk hair though.