Witness update

HF16 and post-HF16 cause a lot of trouble for witnesses, so I would like to update my witness as follow, with the hope that it could help other fellows witnesses in some ways.
Disclaimer: It's a collection of wisdom from other witnesses, not my own.

For hardware I think it's overkill and not worth for backup witnesses, but I love overkill ;)

• 1 dedicated Dual Xeon, 32GB RAM, 200GB SSD
• 2 VMs at Azure and Google Cloud, 2-4 Cores, 10-15GB RAM, 60GB SSD
• 1 private server Core i7, 32GB RAM, 250GB SSD, behind a corporate-grade firewall, no direct ssh allowed
• 1 seed node Azure

All five machines were compiled with low memory mode, syncing share_file to shm (thanks to @abit and @gandalf) and zswap trick from @smooth to reduce IO as much as possible.

To use shm, please do following (10-12GB is the recommend size)

sudo mount -o remount,size=12G /dev/shm

Remember to backup your /dev/shm before reboot because it will be clean and you will have to resync from scratch again. On the good side that the syncing time is much faster now, 10mins for reindex and 70mins for resyncing.
Suggestion from @abit:
No need to resync but need a replay, if you have --data-dir set to a location on a disk.

To use zswap, please change this line

GRUB_CMDLINE_LINUX="zswap.enabled=1

in /etc/default/grub, then run update-grub and reboot (credit goes to @smooth, thanks).

The first three machines are running with only database_api and witness plugin (no cli connection allowed).
Recommend config.ini:

shared-file-size = 12G
shared-file-dir = /dev/shm
flush = 0
public-api = database_api #login_api 
enable-plugin = witness #account_by_key

The flush parameter doesn't actually have any effect so disable it now for less logs.

The private server is hosting the price feed and failover script which automatic switching between 4 machines. This server is running with login_api and account_by_key to allow connection from cli_wallet.

If you're interested in my failover script please get it here. Automatic switching between unlimited nodes, no dependency, just a small python script but it requires you to have cli_wallet running as daemon.
Basically for my setup, the script running at the private server, if the current witness node misses blocks for any reason, after a threshold (2 blocks in my case), the script switch the witness signing key for the next node. By design if all the keys were iterated then the script will stop. If it switches, you will get an email notification through mailgun API.

Just drop me a line if you have something to ask.

Thanks for reading.

Aizen