Steemd 0.20.6 bug - memory exhaustion when parsing malicious hello_message
Project Information
- Repository: https://github.com/steemit/steem
- Project Name: Steem
Expected behavior
The Steemd process should handle malformed messages arriving from a peer by logging and error and/or terminating the connection.
Actual behavior
A carefully crafted hello_message sent can cause steemd to attempt allocating all available memory, causing it to crash.
How to reproduce
A fuller writeup of how the bug was found can be read at https://steemit.com/software/@fuzz-ai/a-memory-exhaustion-attack-against-the-steem-blockchain
To exploit the bug, a malicious witness connects to a steemd instance over the network (using the peer-to-peer protocol) and participates in the encryption handshake. When sending the normal hello_message, it populates the variant_object field with an ill-formed variant object. On the wire, this is a repeating sequence of the four bytes
06 -- array
80 c0 02 -- of length 655360
The unparse function allocates this length of vector (the maximum allowed) and then recursively populates the first entry with another maximum-length vector. Then the first element of that entry is also a preallocated-vector, etc., for as many levels of depth as permitted by the maximum message size (and the stack size of the thread.) This results in memory usage far exceeding the 2MB maximum message size, up to about 1.3 terabytes, which will likely cause out-of-memory conditions on all but the very largest servers.
This bug was reported to [email protected] and was patched in release 0.20.7.
Hi @fuzz-ai, welcome to Steem and great to see you contributing to Utopian! Amazing work with AFL to find this issue. I especially like the linked post where you provide more detailed information on how the fuzzing was done. Great to see you reached out to Steemit privately and published this information only after the bug was fixed by Steemit.
You've identified this issue with the hello message, which is typically sent by witness nodes. Could have user operations triggered a similar issue? I'm thinking of
custom_jsonops with only arrays of arrays of arrays... as the payload (but limited at 4k per transaction, not 2MB)? This should go through the same logic, shouldn't it?Your contribution has been evaluated according to Utopian policies and guidelines, as well as a predefined set of questions pertaining to the category.
To view those questions and the relevant answers related to your post, click here.
Need help? Write a ticket on https://support.utopian.io/.
Chat with us on Discord.
[utopian-moderator]
I thought that custom_json ops didn't use the C++ variant type, but just a JSON string. There is a bug in the JSON parser, but not an exploitable one in the way it is used. But I haven't looked at that in detail, it would certainly be good to understand what sort of things somebody could insert using a custom_json operation.
You're right about the custom_json not using the varint type, this is indeed just a string. The
witness_set_properties.propsfield might be a candidate, though? Looking forward to more fuzzing results from you :)Thinking about this a little more, I was worried you might have been right about nested JSON objects, and that deeply-nested JSON objects in the JSON-RPC API could still cause the thread to die because of stack overflow.
The parser does have a check that you can't nest JSON objects or arrays more than 100 deep: https://github.com/steemit/steem/blob/9e83f66c85a2c76bef1a07cef7dd302d2c4be572/libraries/fc/src/io/json.cpp#L442
But I'm not sure it's effective, I can think of one way it might be fooled.
Thank you for your review, @crokkon! Keep up the good work!
Hi @fuzz-ai!
Your post was upvoted by @steem-ua, new Steem dApp, using UserAuthority for algorithmic post curation!
Your post is eligible for our upvote, thanks to our collaboration with @utopian-io!
Feel free to join our @steem-ua Discord server
Congratulations @fuzz-ai! You have completed the following achievement on the Steem blockchain and have been rewarded with new badge(s) :
Click here to view your Board of Honor
If you no longer want to receive notifications, reply to this comment with the word
STOPTo support your work, I also upvoted your post!
Congratulations @fuzz-ai! You have completed the following achievement on the Steem blockchain and have been rewarded with new badge(s) :
Click here to view your Board of Honor
If you no longer want to receive notifications, reply to this comment with the word
STOPHey, @fuzz-ai!
Thanks for contributing on Utopian.
Congratulations! Your contribution was Staff Picked to receive a maximum vote for the bug-hunting category on Utopian for being of significant value to the project and the open source community.
We’re already looking forward to your next contribution!
Get higher incentives and support Utopian.io!
Simply set @utopian.pay as a 5% (or higher) payout beneficiary on your contribution post (via SteemPlus or Steeditor).
Want to chat? Join us on Discord https://discord.gg/h52nFrV.
Vote for Utopian Witness!