Investigating Witness Server Hiccups!

in #witness-category7 years ago (edited)

Hail all, hope all is well. Just a quick update post to inform of my my witness is disabled.

Earlier this morning the steemd running on main witness server stopped with a bus error. Alerted by the STEEM Monitor app on the cellphone and a witness server sentry in place I'd had a chance to switch over to my backup in time for my next scheduled block (or so I thought..) but seemingly it seems to be having issues as well. In light of both my main and backup having issues I've disabled my witness signing key until the problems can be found out and rectified. My apologies for any missed blocks.

Looks like I've got TERRIBLE latency issues which could account for missing on the backup server as noticed above in the screencap... ffs. Getting the main going again now hopefully.. We shall see. Will update as I figure it out.

Update 2: It MAY be Working!

Seems my main is back online and showing decent latency! Signed the keys back to active and going to go investigate why my backup is still showing terrible latency issues.

Hopefully I can get the backup operating properly as well. Wish me luck!

Update: Well, Restart & Go Again

Getting the main server replayed /restarted in hopes that mends the issue. Increased the shared memory size too as well to give it breathing room so hopefully that nips this problem in the ass.. Interested to see what happens once it gets synced up and operational.. The backup seems to be running but having network latency issues still.. Seems to be from the hosting end but I'm not 100% sure.

Sort:  

I forgive you @klye

Thanks. :D

Get your shit together Kyle. I'm thinking the problem is probably because the servers are frozen up there in Northern Manitoba haha

Lmao. I don't host the servers here thankfully. Far to much ticker tape and having to ward off rogue moose for my liking.

Will that have any affect on Steemit price?

Thanks for the information. Hopefully soon again @klye

No worries. Been a while since I posted, wish it was better news!

Yeah hopefully get better news.

Shit happens. Hakuna Matata.

What a wonderful phrase.

I actually adopted it as a life motto after that movie

acted well and quickly. That is what matters. Don't be sorry for a missed block. This can happen to all.

I missed a few. Enough to warrant disabling the signing key.

yes, but don't need to say sorry. I missed my first blocks some weeks ago.. It was not nice, but happened.

Hey Klye, how much RAM do you have on this machine? Is this VM or bare metal? Please check dmesg output, iostat -x 1 and see the disk utilization, also free -m and verify if you have enough RAM. Run ifconfig and see if you have any TX/RX errors, vmstat 1 and observe the (r, b) columns, b is the number of blocked processes, it should be always 0. What is the loadavg (uptime). I know there are many things to check... I keep my fingers crossed.

It could happen, sometimes :-) But good job so far !!