Even if it was @netuoso, there's nothing malicious about trying strange operations out and seeing how the blockchain will be able to handle it or not. Imagine this happened after we already had millions of active users, it would be much worse. I'm glad @nijeah did what they did. After all, a fix was found, though not without lots of stress for all the witnesses/devs involved, I am sure.
Edit: Obviously I realize the seriousness of the situation. I'm also not a developer/witness so I'm somewhat ignorant about the proper procedures. But I expect the STEEM blockchain to be strong enough to handle something of this nature. If it cannot, there is no reason to use it over another coin. I'm sorry, but I just cannot blame the user who initiated this operation. Clearly it would have been wiser to make the attempt on a testnet, so perhaps there was some malicious intent.
There's something weird about it though. @nijeah tried 4 operations, first a -1Vest withdrawal, then a -2Vest, then -10 Billion, and finally -1 Trillion, which is way over the Vesting Fund of 391,231,329,807 Vests.
Not to shamelessly plug my stuff but, I emphasize this very detail here: https://steemit.com/steem/@jerc33/steem-blockchain-down-here-s-what-happened
Also, and not less important. No one just tries stuff like this and at these disastrous amounts (albeit negative amounts, sure) on a production environment. This is a completely irresponsible conduct for someone "just testing the system".
EDIT: The right approach would be. trying this in a testing environment, of course. But still disregarding that one, at a -1Vest withdrawal @nijeah had already all the information he/she needed to report it to @steemit directly. And by doing so, the SteemitDevs would have 7 days to prepare and probably correct the error, instead of having to push all-nighters just because of the incompetence of a, presumably self-entitled "pen-tester".
I have a hard time believing this had other intents than malicious ones. Incompetence doesn't look like this.
Yes, I did notice the absurd increasing quantities. I understand the view that this is irresponsible, but don't know enough about coding to be able to say whether there was a better way to test this than live on-chain. Besides, the operation was started 7 days before, there should have been plenty of time to detect this anomaly and implement a fix before the blockchain froze. I'm sorry, but I expect the STEEM blockchain to be extremely robust. After 2 years of being live it should be able to handle something as basic as negative withdrawals.
That's easy, We're all humans. Every code-base, be it Google's, Microsoft's, Facebook's or wtv, has flaws like this waiting to be discovered. And some of those that have been discovered already are even dumber, like the empty password flaw on macOS, recently.
Of course, if this happened to some software I created the first thing I'd want to do after fixing it would be hide under a rock out of shame. I'm sure SteemitDevs feel the same way already.
About detecting though, that's tricky. You can't implement unit tests on problems you don't foresee. But as someone involved in pen-testing projects I have to say, the lack of communication on nijeah's part raises all kinds of red flags to me.
But, I'm of the opinion that Steemit failed miserably at one very important thing, the fact that it never organized a proper bug-bounty program like, for example EOS did, on hackerone.com . Like @isnochys said, there's even no proper testing environment and that's clearly dumb on their part. (@ned you need a testing-evn and bug-bounties on hackerone or bugcrowd or whatever. utopian doesn't count, it's a joke.)
Correction: Maybe there is a testing environment after all, according to @therealwolf
That's not how coding works. Especially in something as sensitive as a blockchain.
You don't just try something and see if it breaks things. You write a test for it and if you have a suspicion about a bug, you'll let those people know, who are responsible and if you can fix it yourself - you'll do that.
Is there a live testnet for the STEEM blockchain? I agree that would be a more conservative place to try an operation like this. However, I'm a bit astonished that something as simple as prohibiting a negative power down has not been patched in 2 years. Then again, until it happened, it could just seem absurd.
Yes, STEEM has of course a testnet.
https://developers.steem.io/testnet/
https://testnet.steem.vc/
That is strange, then, that the operation wasn't tried there first. Perhaps there was some malicious intent after all.
Actually there are several testnets, and the option to set up an own one: https://developers.steem.io/testnet/
Relevant
Hehe.
But I'm pretty sure nearly every coder had a time where they just wanted to finish an update/feature.
Just. One. Last. Commit ...
Yes, that was not important!
just all of steem nodes stopped working, all applications on the steem blockchain and block production stopped for ~10 hours, and we got thousands of missed blocks
When did I say it was not important? It's one of the biggest crises to happen in the past 2 years of the blockchain being active.
That's why every product should have a valid test net.
Where one can try out things
But I may be repeating myself, steem(it) needs a proper service management.
Testing, Integration and Prod environments.
Steemit Inc has at least 1 testnet (but I think 2). You can even use the testnet easily with dsteem just with a line of code.
https://developers.steem.io/testnet/
Any "testing" of this sort should always be done on a testnet. Anyone experienced enough to do this is experienced enough to know that too.
That makes sense @blocktrades. I suppose it's easy to forget we're still in a beta phase because STEEM is so real for those of us that use it daily.
Fantastic friend