Do we have the best witnesses? Questions for the witnesses.
I am @asbear, a standby witness and software engineer with network + security background, the creator of KR Guide Dog, SteemPay, Steemian Health Check, Witness Insignt and many other small tools related to Steemit. I usually work for KR community but occasionally communicate with the other witnesses and the Steemians from all over the world.
We are DPOS, do we have the best witnesses?
Steem is the most successful DPOS supported by the strong community onboard. Every other jealousy blockchain has been trying to diminish Steem, mostly about the imperfection of the witness. I have had a strong faith in DPOS, and trusted that Steem will always have the best people as its top 20 witnesses. But my faith is getting weaker the more I know.
Today I noticed that some witnesses were keeping missing their blocks at the same time, one of them were missing the entire blocks for 2.5 hours. I contacted one of them to let him fix the problem asap, and the reason turned out the network outage in the ISP where their witness nodes are running in. I was quite shocked about this, because this is not professional at all.
How many of the top 20 witness understand what will happen if their ISP is down, and prepared for it? It is not all about missing blocks and losing your profit. It is more about the quality of service of Steem block chain and Steemit of course. Needless to say, this kind of amateurism degrades the reputation of Steem.
Are you concerning geographical redundancy?
DPOS cannot be trusted without the geographical redundancy. Here are the self-check questions.
Where is your witness node and where is your backup witness node?
You should consider running your backup witness node at least in the different ISP, different location or even different continent if possible.
How well are the witness nodes of the top 20 witnesses geographically scattered?
I hope at least one of the top 20 witnesses concerns about it and actually encourages the others to make an effort for it.
How many witnesses are using Privex for their witness nodes?
Privex offers a great deal, and AFAIK many witnesses are using Privex's servers. Their price is much cheaper in Germany so I wouldn't be surprised if many of witness nodes are in Germany. This is ok for standby witnesses but no for top 20 witnesses. I am very interested to see how many witness nodes will be down if there would be network or power outage in Privex's data centre.
How can we avoid using the same ISPs excessively in the similar locations
The best first step is opening up the name og ISP and the region. Then we can easily see what will be the potential problem.
You might not want to open that up. What do you worry about sharing the name of ISP and location your witness node run in to the public? Is the concern theoretically proven? I am open to discuss further about this.
How often are you brushing up your witness vote?
Witness vote makes it all. Things will never go wrong if you vote to the right people. Are you voting to the best people? How often do you confirm they are still the best? Here are the self-check questions.
Are you voting to any inactive witnesses?
Voting to an inactive witness has absolutely no benefit. There cannot be any good reason whatsoever. If you have one, withdraw it.
Are you voting to the witnesses who do not care the security enough?
Some witnesses are still using 0.19.2 that is known to have a security hole. Then voting to the witness who are using 0.19.2 can never be a reasonable choice.
Are you voting to any witnesses with biased feed or not feeding the price frequently?
Price feeding is the most essential work that witness must do with perfection. And it is very easy and cheap.If a witness is not doing his best for feeding price, he does not deserve your witness vote.
The following three websites can help to self-diagnose your witness vote.
https://steeme.com/witnesses
https://steemdb.com/witnesses
https://steemian.info/witnesses
Let us be more professional.
Each top 20 witness earns 7800 Steem Power which is about 24000 USD a month, and they are elected by the community for the best future of Steem. I didn't want to make all the witness my enemy, and also never want to attack anyone. Please do not take this criticism personally.
Amateurism will kill Steem blockchain slowly but surely. Let us discuss as earlier as possible and get stronger together.
I added Utopian tag, as this post includes a very important suggestion for the entire Steemit community.
Update
I blame myself for that my writing could not clearly deliver my point. Sorry if someone felt offended. So I will restate the point especially about Privex.
Somehow at least one person now thinks I am attacking Privex and @someguy123. No definitely I am not. Let me be clear. @someguy123 has been one of my witness votee, as I got a big help from his steam-docker and steemfeed-js projects. And I am also considering moving my witness node to Privex. Thanks to my job, I know security and hardening so I don’t worry about moving out from Google Cloud. Another reason is that they accept STEEM/SBD. I sincerely support the STEEM economy and want to boost it as much as I can. Probably you will not doubt it if you know I am the creator of SteemPay.co.
Here are the key point.
1.If many top 20 witnesses are using Privex without discussing about it (region) each other, we are loosing the geographical redundancy.
- If no one is proactively preventing the witnesses from choosing the cheapest region amongst the location Privex serves, it's natural that witnesses would go for the most economic option, so that we are loosing the geographical redundancy.
- Privex provides only two region, US and Germany, as in the subscription form (https://pay.privex.io/order/). Even if you split your nodes to US and Germany, we are still loosing the chance to improve the geographical redundancy.
This is just one of the most basic principles to consider when you do software architecting. Nothing should be personal and nothing should be even offensive. I really wanted @someguy123 to seriously think about this, and make a strong advice for the witnesses who are deploying their nodes against this principles.
And for those who feel this offensive:
If we are already doing well?
- That a good news, we all are happy. Let’s reassure it and go ahead.
If some of us are doing not the best?
- Still ok! This is a great time to improve that.
If you are doing something wrong, and do not want to change and do not want to be criticised?
- You do not deserve a single witness vote.
If you have any different opinion, let’s talk about it further. Just please don’t take this as an unpleasant criticism and try to blame others.
When I read this I can't help but feel that you are attacking witnesses using Privex and Privex itself. I suppose thats because I read when Ned stated a few months ago those witnesses that use Privex should be voted out. And implied that Privex was a threat to the network.
I read this statement and hear a tone that somehow Privex and Privex users have judgement that is wrong or misguided. That these top 20 witnesses that missed blocks should be cast out.
Well I don't know for sure but @roelandp, @good-karma, and @blocktrades seemed to have missed the most blocks this week. I'm guessing they were the ones that had a server in Germany with Privex. You seem to be implying that those witnesses aren't good for the platform or are acting out of turn. The primary job of the witness is to assemble the chain and run a steady box. Yep they missed some blocks, but if you look at percent of blocks missed someone like @roelandp is still under a fraction of 1% missed. Is he doing a terrible job? Does he not give a fuck? Is he bad for Steem?
Witnessing is more than the box too. @roelandp founded Steemfest one of the largest events this community hosts all year. @good-karma made an app that gets three times the volume of tx as cryptokitties. Blocktrades has been here since day one and runs a business that allows folks in other chains access to steem easily. Which of these three witnesses do you think we should cast out for a problem at a common data center?
I've been using Privex for more than a year. Someguy helped me get my witness up and running when i had zero idea of what i was doing. Someguy provides a docker image that makes getting witnesses started a breeze. Someguy has a ton of experience and can help troubleshoot. Someguy has been updating the docker image with each hardfork change that comes through. Someguy has figured out a way to deliver inexpensive servers to help this community in multiple continents. Someguy has has figured out access to 512gig ram servers to run full rpc nodes. Someguy accepts Steem and SBD for his business. Someguy helped me a ton when I lost my uncle. I stand and will continue to stand with someguy123 and privex.
What does this look like for me? Well, I have 2 servers in two regions with Privex and a third with another company in a third continent. No missed blocks...
If you think more needs to be done to diversify servers then setup a data center and compete with Privex and do a better job. Don't simply complain. Getting rid of Privex would narrow witness choices to mega corps many folks may not actually want to support with their business rather than somehow make this issue magically go away. You'll concentrate the problem in fewer locations... how exactly is that good for Steem?
I get as a new witness you want attention, and there is something to be said regarding making sure that failover scripts are working, blocks are going smoothly, and servers are in multiple regions. You can say that without implying bad shit about Privex or witnesses that use the service.
As a clarification, we don't use privex.
I am work having busy day but walked out to leave a short comment for you. I have no intension to attack anyone. Furthermore, Blocktrades said it is not Privex. You know what that mean. If you are guessing around, please dont.
Also never known Ned said that. Privex has their data centre in two regions. You can distribute without leaving Privex. Not sure what sentences or phrases in my posting made you think in this way? Feels like you are just trying to disagree. I never said we should judge the top witnesses only by their block producing and feeding. Their other contributions are out of tis debation.
Will come back after work for leaving replies to the other guys. Lets do more discussion using the time if you want.
I as well am a big believer in voting for people that do awesome and unique services for the community so I'm with you on that one.
I also think that specially if they hit the top 20 and become very integral for our platform they should also run an extremely professional service to make sure Steem/Steemit is running at TOP performance. I don't think they should be removed but just maybe not top 20 if they're not ready for what that entails, they now have a job.
I know little about the specific witnesses on the top 20 so this is all just generic theory.
I for one am intrigued if they hire someone to do that job for them? Seems they can keep up the awesome projects that put them into the limelight to get to that position and a really experienced person can work part time making sure it's set up and running perfectly and available dependably if needed. That seems like a great trade-off and helps avoid "amateurism" and then we all are happy. haha
I think the point is that there will always be those who favor witnesses who run their own boxes and do not outsource this important aspect of the job to a company that also hosts other witness boxes, because the mere attitude towards this being okay makes the blockchain vulnerable and overly centralized in case there are any malicious hacks, ddos attacks, or state-level action taken against the company. It's not so much that privex or any other hosting service are bad in and of themselves, but rather that a main selling point of dpos and multiple witnesses is to at least maintain some level of decentralization and resistance to attacks on the network that are only made easier by putting all of your eggs in one basket.
If all 20 top witnesses use one company that runs boxes out of seven continents, it still won't matter how geographically dispersed that company's hosting services are if, for example, a nation has the ability and authority to confiscate all of that company's servers in one fell swoop.
I am of the opinion that witnesses be changed regularly by voters. Power corrupts and longer it stays with same set of people kr chances are they may collude and undermine the honest decentralisation.
So voters have to everymonth change their vote to bottom 30th position.
I have multiple nodes in data centers in different parts of the world. I will not reveal the exact locations though for security reasons.
I do not.
No.
Unfortunately the 19.2 tag is not 100% sufficient in this case to determine if they are running 19.2 or 19.3, because there was an "untagged" version of the security patch that was rolled out to a lot of nodes - so there are some nodes that are running the patch, even though it says
19.2.In general though, I do take into consideration whether witnesses are responding to important security patches in a timely manner as one of my criteria for voting.
I take this into consideration, but using a bias is not necessarily going to be a deal-breaker in of itself. If a witness uses a bias for a good justified reason (even if I disagree with it) and is a good witness in a lot of other areas, I may still vote for them.
Thanks for the questions :)
Hey @timcliff, thanks for responding to my questions!
Not sure why you think revealing the location is risking security, however I respect your decision not to.
Thanks for pointing out the version tag. I was unaware of it. However I still think staying in 0.19.2 is not the best no matter what patch is applied under the bonnet because what the people outside of the circle think is also quite important. You know what I mean. I naively hope someone like you could make a voice about it.
This is good. There is however no way to know it or trust it to be honest.
How about witnesses with more than 40% biased feed? Would it change your mind?
@timcliff, I appreciate your time and openness. "The Reports from the Witnesses" is what I really like and hope you can continue stimulating others like that way. Cheers!
I would rather not give anyone any ideas by explaining how it could be exploited, but it is better to be as private as possible about the specifics of your block producing nodes.
The 19.2 version that a lot of witnesses are running it is literally 19.3 just without the change to update the version number.
It is a valid argument with regards to public perception. It is on individual node operators though to make those types of decisions. It is a political/pr decision (not a security one).
No. I am strongly against using a bias to increase SBD production, but I do not expect every other witness to see things from my point of view.
If they have a legitimate reason behind choosing something against my views - I may still vote for them based on other criteria.
I try to look at the big picture (most of the time) instead of making decisions solely based on one issue.
Good to know your perspective. I really appreciate your sharing your opinion!
@asbear -These are Great questions people should be asking.
I have 3 witness nodes in 3 different data centers on 3 different continents.
1 Full Public RPC Node
1 Seed node
1 of our witness nodes was located in one of the datacenters that went offline lastnight, and our failover switch kicked in and our first backup at different datacenter took over immediately.
It is very very very important to have backups in different DC's for just this type of thing
@followbtcnews, thanks for sharing the information. great job with the firm redundancy.
Thanks for making this post @asbear. It is healthy for a DPoS network to critically look at itself and its block producers. I think Steem is especially suited to DPoS security. I've had experience with other DPoS chains and without an active social nexus for their stakeholders to actively participate in, their decentralization and security tend to be compromised. If not immediately, then over time from disinterest and general entropy.
One of my backup nodes is located on a different continent from my primary and secondary witness nodes.
Price feeds don't need to be too frequent. The actual network feed price ends up being the median of witness feeds over the last 3.5 days. It's supposed to be a bit fuzzy. It would not change much if every witness only updated twice per day. Backup witnesses especially don't need to update it frequently. Also the feed price doesn't necessarily need to be updated if the market price has stayed static for some time.
This is true and not true. The original release of Steem v0.19.2 has some security/DoS issues. There were two updates that patched these. Neither of them updated the Steem version number that gets reported by witness nodes. Later, a release containing both security releases was officially tagged as v0.19.3, but that was the only change between the latest security release and the tagged. So while some higher ranked and top 20 witnesses are reporting 0.19.2, they are likely running the patched version but did not see the need to update just for the version report change.
However I think it is important to broadcast that the node is up to date and that is why I updated my witness server to show it.
Much appreciated @pfunk, you are the one of my favourite witness.
I have a different view on this. And probably this perspective is what I think bothers the top witnesses.
I agree that systemically it does not make a big difference. But eventually it degrades the professionalism of Steem witnesses. We should not expect the Investors and competitors understand this system as much.
Price feed is very simple and nearly cost nothing. I would be still quite disappointed if any top or standby witnesses are not managing one of their very basic duty. Broken window theory fits here. I respect your view of the feed though.
Thanks about the versions. I didn't know that clearly and learned now. I will reuse my reply to @timcliff
Also, no other people can prove the witness has 0.19.2 without the security hole. Should we just trust? I would say no. You know and I know that upgrading from 0.19.2 to 0.19.3 is super easy. Don't even need to replay as I recall. I don't think it is asking too much for top witnesses to spend some time to make it look better for everyone.
I am not a business man, consultant, manager or something similar. I am just an engineer. Even someone like me can see something unprofessional. How would that look to the others is the question.
@pfunk, I believe you would not regard me as a troublemaker, and hopefully other witnesses would not as well. Many thanks for your reply.
Not at all!
There is a big chance your questions will be ignored. There is only one witness that I know will not only share some of your concerns, but also actually enjoys having these somewhat difficult conversations and does his best to educate others on the matter.
@drakos if you don't mind me tagging you, I think you are the man for the job.
This is an opportunity to have a healthy conversation on this matter, as a stake holder (mind you im no whale, but i've worked for what i have) it's in my best interest to help, if I can educate our Steemian Population on this very subject a little bit, I will.
It's always a bit worrying to me that we don't do a good enough job (i say all of us, because we are a self governed community) to put this information out there for the new users to understand and act on. Yes, crypto is complicated. Yes, Steem is just as much if not more due to its political elements.
This is the only question that I'm probably informed enough to answer. Most of the witnesses that vote on dead witnesses do so knowingly. I speculate two reasons as to why.
Blocktrades said something to this effect a while back, It would probably be hard for me to find this comment, but I'm sure if he was asked directly, he would not hesitate to confirm.
Now... there is a special radio show happening this weekend.
@aggroed who is a top 20 witness and a very ethical one (imo of course) is hosting A witness Panel this coming Saturday.
It would not only be beneficial for you to come to the show, it might be beneficial to us as an audience to.
In any case, I take my leave, much respect.
Cheers
@meno I really appreciate your input here. You seem to know much more than I know and I honour and respect that fact. Maybe I am making to much noise here, and might need to step back and think again.
Maybe I overrated Steem blockchain and the witnesses. To me it's all amateurism but nothing. At least the top 20 witnesses should really care how the investors and the competitors would think about themselves as a reflection of Steem blockchain. That's how they are respecting everyone. Top 20 witnesses are not just normal Steemians aren't they. It is becoming clearer that maybe Steem witness is not what I want to be. will think about quitting.
How come this kind of excuses could be acceptable for the top 20 representatives of DPOS.
Checking out the comment from @aggroed on this post, it would be pointless attending the chat. I really do not enjoy any aggressive and emotional discussion.
Thanks again for your comment, although I disagree with some of your words, it is really fruitful discussion with you. I will definitely consider voting you if you become a witness later.
Regards
I appreciate your kind reply @asbear, I would however encourage you to reconsider attending the panel this weekend. I'm willing to stick my neck out there and say that even in disagreements, you won't be attacked and your point of view will be discussed as adults.
The witnesses who participate of MSP/Palnet without exception really are wonderful people, and to be honest with you if you perceived aggression from @aggroed's words. It might be because he is a passionate man and nothing else.
If you ask me, I think the conversation that has sprouted in your comment section has been extremely productive. Yes, it there is some coffee grounds at the bottom of the cup, but in many ways you have effectively made enough noise as to direct at least partially this weekend's conversation.
At the expense of possibly offending someone I will speculate that some of the defensive reactions from some of the witnesses on this post, come as some residual bad taste from a recent post made by Bernie, in which he was in his usual ways, very "unique" very Bernie.
In any case, I sincerely hope you come to the panel. You can listen in, if you don't feel like saying anything, don't, but you will hands down feel the love and passion we all have for this community and the unquestionable commitment to betterment of Steem.
All the best brother
Thanks, you are a really good man. :)
It's a bank holiday weekend in the UK so I HAVE TO go out with my wife all day unfortunately (?) but I will try to listen to the recording later.
Cheers!
Believe it or not, witnesses do discuss these issues. There was a big discussion about privex in particular, at one point, one of the main topics discussed in this post.
I did not mean it like an attack @blocktrades, it it came across like that I do apologize. I'm simply stating that they are not discussed in public, on a post like this and as much as I understand the need for a private slack channel to iron details out, the smaller witnesses and the public are left scratching our head in ignorance.
So I will clarify, not an attack in the slightest, simply an observation. I know @asbear is a newer witness, and as such he is both trying to get his name out there (which I appreciate) and also attempting to understand the dynamics at play.
That is all...
I see, thanks for clarification. In fact, I'd say you're spot on at this point: most of the top 20 witnesses don't think it's a good idea to share security details publicly.
On the other hand, I guess I assumed there was already some public docs that give suggestions for good ways to configure your witness from security, etc, point of view. If not, there should be.
I will be audacious enough to ask on behalf of @aggroed (without his blessing of course, hence the audacious) if you could participate of the upcoming panel this saturday.
I think he will be happy to have your voice added to the pot. As one of the biggest investors on the platform, your contributions, concerns and ideas hold a lot more weight.
With that request, I take my leave... (I won't take it personal if you decline)
Cheers
I've got a standing invitation from aggroed to attend and we agreed he would ping me once a month. I hope I can make it sometime next month, but right now I'm just super busy. I work on weekends too (generally only around 5 or 6 hours on weekend days), so even Saturday isn't a "free day" for me.
I confess I also didn't read all your reply before, only the first part. Since I'm here I'll confirm that I did vote for a few disabled witnesses. I left the votes for quite a while as they were essentially a non-vote plus a reminder to the inactive witness that some people who be happy to have them back (for example, I'd love to see SVK contributing here again). But I did eventually remove them, mostly because I would get bugged by people who wanted me to change my vote from one of the inactive witnesses to them.
Possibly a distinction needs to be made between for voting for a non-active witness (OK in my opinion) and voting for a witness whose server has been down but not marked inactive and therefore missing blocks (this is bad). IMO, there's nothing wrong with voting for a non-active witness: the whole ability to set yourself inactive was to allow you to stop blocks from being missed without requiring you to contact all your voters and get them to unvote you.
On the whole inactive witness or disabled witness I've learn to see this whole thing from a different perspective. I'm attempting to be as pragmatic as possible on the subject, because I know there is probably no effective way from removing politics from the emotional underpinnings but...
Just the other day a new witness showed up on trending, requesting votes of course, as he should and pretty much casting some buckets of blame water on to high stake holders not casting votes, or casting them to inactive accounts.
My current position on the matter is that all new witnesses probably need that time to ferment in slight failure, as to see if they can cut the cheese. Meaning, that they are in it for the right reasons.
We don't want another @zeartul pissing on the fruit punch we all drink from and I'm sure no one will disagree with me there. With that in mind, I find it healthy that a mega whale account would be extremely hesitant to vote on a new witness, simply because the vote is available or the name is catchy.
From where I stand, you, and other whales of tide altering proportions act in the best interest of steem by being "ridiculously"(please note the air quotes) careful.
Hey @blocktrades, I hope and believe you didnt take this personally but if you ever felt being offended by me I apologise. As usual my writing is not well shaped.
Maybe due to my professional, I get easily worried when I find anything flaky and try to check up. I understand I might be misunderstood and might mislead someone else. Will be always open for any discussion for good purpose.
Regards
No worries, I didn't take it personally at all. Mostly just wanted to say that these kind of issues are discussed by most of the top 20 witnesses, it's just that security ones in particular are usually not publicly discussed.
Would be great if someone would write an article how to become a witness.
Running a full node with the latest steem version and keeping the dedicated server save in a country where no other nodes are running is doable.
There are enough documents already. Google it :)
well, I am part of a new witness that was recently founded and we did not choose privex for that same reason.
We are hosting our server in Switzerland. For the moment we only have the one server and once we come closer to break-even we will be getting our second one as a backup on another continent.
We are also updating our price feed multiple times per day
Since we are currently only doing 1 block per day missing hundreds of blocks is not an issue but the backup is our next investment.
My witness voting has some room but is clean.
I just want to thank all the poeple that have voted for us in the last weeks
I also want to thank those that are going to vote for us as well.
Thanks for revealing your status!
I will consider voting you when I clearly see your contribution would help Steem. :)
well we think it is important that people know who they are voting for and that they know we are a growing project. We do not have the budget to invest in 3 servers out of nothing and so we are choosing a gradual approach. Hence the more votes, the more growth the more we are able to put back into support
this is an idea that we want to support the steem blockchain if we get some support back from the community with votes.
hosting 3 servers would cost us roughly 600 dollars per month. and now we are making 90(ish), we have paid our server upfront for 3 months and will continue but without community backing it is a losing prospect.
I can see you have 2 votes left over. We would be grateful to receive yours and if you find someone more worthy, then by all means replace us, It is your vote to do with as you see fit...
I just think that our independence here in Switzerland would be a plus
voted. Still cannot find you from the list though.. :( is @felander your witness account surely?
no, my witness account is @swisswitness
since we are a 3 man team we created a separate account
https://steemconnect.com/sign/account-witness-vote?witness=swisswitness&approve=1
here is an easy link
ok you have my vote. thanks!
Awesome work raising awareness about witness voting!
I just released this tool yesterday that will also help users evaluate their current witness votes.
Great job and great writing! Thanks!
cool post, Just used it to clear some slots in my account, bookmarked
Awesome, glad it was helpful!
you might want to set up something that shows when someone is proxying their votes because now it just shows a black screen
Yup, just thought about that so working on that now!
Very cool tool! I like it. The only downside is it may be a bit misleading if witnesses (like me) post from their main account (@lukestokes) and are very active but have a separate account (@lukestokes.mhth) just for managing their witness duties.
Suggestion: Could you set up an alias option for each witness? So instead of doing a lookup on @lukestokes.mhth, take a look at @lukestokes also. Don't go off of @smooth.witness alone, but check @smooth too. Instead of just @noisy.witness, take a look at @noisy.
Thoughts?
As you might have guessed, this came up earlier today so I'm already working on this! And it shouldn't be too hard to implement so I should have an update out shortly.
I'm happy to say I just released an update that I think handles this pretty well. Would love any feedback you might have.
It's a little confusing to me. Personally, I would just have a single line. People are less interested in which account posted something and more interested that an account post something. That's my thinking, anyway, but maybe for witness accounts managed by multiple people, what you're doing makes more sense.
For me, I'd view it like so:
or just something to group them together more easily instead of spreading them out as separate witnesses (when they are really the same witness)
I struggled with exactly this issue as I was working on this and the witnesses managed by multiple accounts is the part that pushed me to this format. Each row is an asynchronous call so there is no ordering around it right now and I need to fix that.
I'm definitely going to clean up the entire view and will try to find a better way convey these relationships.
Thanks for your feedback, I appreciate your time.