My waves node did not start anymore... and how I fixed this

in #waves7 years ago (edited)

1. when I started my WAVES node I saw this output:

  • INFO  [main] c.w.Application$ - Starting...
  • INFO  [main] kamon.Kamon$Instance - Initializing Kamon...
  • INFO  [lt-dispatcher-3] a.event.slf4j.Slf4jLogger - Slf4jLogger started
  • INFO  [main] c.w.Application$ - Waves v0.7.9 Blockchain Id: W
  • INFO  [main] c.w.s.BlockchainUpdaterImpl - Constructing BlockchainUpdaterImpl, total blocks: 765676, persisted: 765456, topMemDiff: 0, bottomMemDiff: 0
  • INFO  [main] c.w.s.BlockchainUpdaterImpl - State rebuild started, total blocks: 765676, persisted: 765456, topMemDiff: 0, bottomMemDiff: 0
  • ERROR [main] c.w.actor.RootActorSystem$ - Error while initializing actor system wavesplatform
  • java.lang.Exception: TransactionValidationError(GenericError(Tx id(exc. for some PaymentTransactions) cannot be duplicated. Current height is: 765457. Tx with such id aready present),{"type":4,"id":"BPrkJXm8EE8eTA3kX7ae39ZfwvrBsYPfy1bstK1KHfs6","sender":"3PE5ZLLSdrryLZ6TSCJZhimY9HxmQT8m4Ty","senderPublicKey":"DPjswXJgPJwepuohsKzkSRvfFxDCZQ7hBUBqksoN2a2M","fee":100000,"timestamp":1511518072240,"signature":"Yf4fQRsknTuqdacz2JyvjKk8cToqjoepDS48EdLC3xPxFUepvb8LRog6oJG3kd8wxfgbiu6xK8WRUPGerDBcsS4","recipient":"3P4kAfTst9EU9mGWRVF6EQ5CSttmEwcHjVo","assetId":null,"amount":5000000,"feeAsset":null,"attachment":""})


2. how I fixed it? 

  • by deleting the state.dat file and waiting for the "repair" by waves itself
  • for the 765456 blocks it took about 17h


How it came to this?


3. scrolling upwards in the LOG file I saw this output in step 1. :

  • INFO  [nator-handler-1] c.w.s.BlockchainUpdaterImpl - Rollback to h=765672 started, total blocks: 765673, persisted: 765456, topMemDiff: 107, bottomMemDiff: 110
  • INFO  [nator-handler-1] c.w.s.BlockchainUpdaterImpl - Rollback to h=765672 completed:, total blocks: 765672, persisted: 765456, topMemDiff: 106, bottomMemDiff: 110
  • can't fork: Cannot allocate memory

4. and later on also these:

  • INFO  [shutdownHook1] c.w.Application$$anon$1 - Stopping network services
  • ERROR [xt-global-74168] s.utils.LogMVMapBuilder - MVStore put error
  • java.lang.IllegalStateException: This map is closed [1.4.196/4]


The "can't fork: Cannot allocate memory" probably means my RAM was not sufficient anymore? And waves had then trouble executing its normal shutdown tasks?

  • since using WAVES: I see on my system that my RAM gets filled slowly (about 40%), also when I exit WAVES this consumption (40%) stays... 
  • when WAVES is running it takes about 25%, so I have 35% free... I will observe this more.


The message from step1 looks like this 7-weeks-old bug report, I have added more details there 

and hope someday waves can "heal itself" without user intervention in such cases.