My waves node did not start anymore... and how I fixed this
1. when I started my WAVES node I saw this output:
- INFO [main] c.w.Application$ - Starting...
- INFO [main] kamon.Kamon$Instance - Initializing Kamon...
- INFO [lt-dispatcher-3] a.event.slf4j.Slf4jLogger - Slf4jLogger started
- INFO [main] c.w.Application$ - Waves v0.7.9 Blockchain Id: W
- INFO [main] c.w.s.BlockchainUpdaterImpl - Constructing BlockchainUpdaterImpl, total blocks: 765676, persisted: 765456, topMemDiff: 0, bottomMemDiff: 0
- INFO [main] c.w.s.BlockchainUpdaterImpl - State rebuild started, total blocks: 765676, persisted: 765456, topMemDiff: 0, bottomMemDiff: 0
- ERROR [main] c.w.actor.RootActorSystem$ - Error while initializing actor system wavesplatform
- java.lang.Exception: TransactionValidationError(GenericError(Tx id(exc. for some PaymentTransactions) cannot be duplicated. Current height is: 765457. Tx with such id aready present),{"type":4,"id":"BPrkJXm8EE8eTA3kX7ae39ZfwvrBsYPfy1bstK1KHfs6","sender":"3PE5ZLLSdrryLZ6TSCJZhimY9HxmQT8m4Ty","senderPublicKey":"DPjswXJgPJwepuohsKzkSRvfFxDCZQ7hBUBqksoN2a2M","fee":100000,"timestamp":1511518072240,"signature":"Yf4fQRsknTuqdacz2JyvjKk8cToqjoepDS48EdLC3xPxFUepvb8LRog6oJG3kd8wxfgbiu6xK8WRUPGerDBcsS4","recipient":"3P4kAfTst9EU9mGWRVF6EQ5CSttmEwcHjVo","assetId":null,"amount":5000000,"feeAsset":null,"attachment":""})
2. how I fixed it?
- by deleting the state.dat file and waiting for the "repair" by waves itself
- for the 765456 blocks it took about 17h
How it came to this?
3. scrolling upwards in the LOG file I saw this output in step 1. :
- INFO [nator-handler-1] c.w.s.BlockchainUpdaterImpl - Rollback to h=765672 started, total blocks: 765673, persisted: 765456, topMemDiff: 107, bottomMemDiff: 110
- INFO [nator-handler-1] c.w.s.BlockchainUpdaterImpl - Rollback to h=765672 completed:, total blocks: 765672, persisted: 765456, topMemDiff: 106, bottomMemDiff: 110
- can't fork: Cannot allocate memory
4. and later on also these:
- INFO [shutdownHook1] c.w.Application$$anon$1 - Stopping network services
- ERROR [xt-global-74168] s.utils.LogMVMapBuilder - MVStore put error
- java.lang.IllegalStateException: This map is closed [1.4.196/4]
The "can't fork: Cannot allocate memory" probably means my RAM was not sufficient anymore? And waves had then trouble executing its normal shutdown tasks?
- since using WAVES: I see on my system that my RAM gets filled slowly (about 40%), also when I exit WAVES this consumption (40%) stays...
- when WAVES is running it takes about 25%, so I have 35% free... I will observe this more.
The message from step1 looks like this 7-weeks-old bug report, I have added more details there
and hope someday waves can "heal itself" without user intervention in such cases.