Multichain crashing frequently.

Question

Hi team, 

Is there a limitation on multichain on how many transactions it can hold in total? 

My setup is having 2 nodes with mining permissions with all the default parameters. I was sending assets from the admin account (1st main address which was created with issue permission) to a address created on 2nd node with getnewaddress command and it has send and receive permission. 

I 'm doing a sustained load test with 5-10 TPS on a 1 core, 1 GB ram node. Initially for about a million transactions it was working fine with CPU usage on the primary in the range of 5-10%. However on the secondary node the CPU usage kept increasing and eventually the daemon crashed on that node. 

 

2017-11-24 19:55:49 ResendWalletTransactions()
2017-11-24 19:55:53 UpdateTip:            new best=00f21c7e1ad3c513dee9143d3c5831bb83b7e29cb35490720311dac68adb2e59  height=4662  log2_work=20.187043  tx=996159  date=2017-11-24 19:55:51 progress=1.000000  cache=0
2017-11-24 19:55:57 CommitTransaction: de595202aae5efde05a9a6290fdfcce03f27126b9e12e2f7e8581b5eb895151a, vin: 100, vout: 1
2017-11-24 19:55:57 Combine transaction for address 1BcLHRG7Kha6otPQr8rH177kbqmCogqwXUjH8B (100 inputs,1 outputs): de595202aae5efde05a9a6290fdfcce03f27126b9e12e2f7e8581b5eb895151a; Time:    4.153s
2017-11-24 19:55:57 ResendWalletTransactions()
2017-11-24 19:55:57 MultiChainMiner: Block Found - 0048d5320063eb8b7ce79124d343fb2b7b195e68636acd868b74c1011386020a, prev: 00f21c7e1ad3c513dee9143d3c5831bb83b7e29cb35490720311dac68adb2e59, height: 4663, txs: 6
2017-11-24 19:55:57 ERROR: ContextualCheckBlock : block height mismatch in coinbase
2017-11-24 19:55:57 ERROR: ProcessNewBlock : AcceptBlock FAILED
2017-11-24 19:55:57 ERROR: MultiChainMiner : ProcessNewBlock, block not accepted
2017-11-24 19:56:09 MultiChainMiner: Block Found - 00dc456a5c4178819b108263fae684e8a6aafeadae7363f5f4422c39e6a75b94, prev: 00f21c7e1ad3c513dee9143d3c5831bb83b7e29cb35490720311dac68adb2e59, height: 4663, txs: 362
2017-11-24 19:56:09 Pre-allocating up to position 0x400000 in rev00003.dat

 

And I started the daemon back manually and it crashed again with logs similar to above case. 

 and then the node refused to start giving out of memory errors. 

2017-11-26 08:16:23 ResendWalletTransactions()
2017-11-26 08:16:30 MultiChainMiner: Block Found - 000067fe730d075473eceff46910a630e196549331ff7456df22ec38ee9c913c, prev: 00c2524405faa7a3472ab873a24390c734b5feb56c24f8038b7766dbd8142676, height: 12620, txs: 9950
2017-11-26 08:16:30 Pre-allocating up to position 0x1100000 in rev00004.dat
2017-11-26 08:16:32 UpdateTip:            new best=000067fe730d075473eceff46910a630e196549331ff7456df22ec38ee9c913c  height=12620  log2_work=21.623539  tx=1351776  date=2017-11-26 08:16:29 progress=1.000000  cache=0
2017-11-26 08:16:35 CommitTransaction: 697fe13acb11dfa13d9640ad55329ca050127a97da209dcf98c8b379a670bc7f, vin: 100, vout: 1
2017-11-26 08:16:35 Combine transaction for address 12RbYHmNc1XMpJeS8G4mRCsgJqAjY31D7bTc9E (100 inputs,1 outputs): 697fe13acb11dfa13d9640ad55329ca050127a97da209dcf98c8b379a670bc7f; Time:    2.529s
2017-11-26 08:16:36 ResendWalletTransactions()
2017-11-26 08:16:43 MultiChainMiner: Block Found - 000b37c3fe79d6b0b28a13999196726628fbbced7757244c3f6f7d0f9a711a4e, prev: 000067fe730d075473eceff46910a630e196549331ff7456df22ec38ee9c913c, height: 12621, txs: 4411
2017-11-26 08:16:44 UpdateTip:            new best=000b37c3fe79d6b0b28a13999196726628fbbced7757244c3f6f7d0f9a711a4e  height=12621  log2_work=21.623653  tx=1356187  date=2017-11-26 08:16:43 progress=1.000000  cache=0
2017-11-26 08:16:47 CommitTransaction: cf286c29d8f8107af68f0b858e13b37c8699a030964fb45b9898ccbd4b4d84a6, vin: 100, vout: 1
2017-11-26 08:16:47 Combine transaction for address 12RbYHmNc1XMpJeS8G4mRCsgJqAjY31D7bTc9E (100 inputs,1 outputs): cf286c29d8f8107af68f0b858e13b37c8699a030964fb45b9898ccbd4b4d84a6; Time:    2.603s
2017-11-26 08:16:55 UpdateTip:            new best=00f7cdfb53c8f78cd2ceeba0cdbf98387519433cf2393307aea526341fee02de  height=12622  log2_work=21.623767  tx=1356257  date=2017-11-26 08:16:52 progress=1.000000  cache=0
2017-11-26 08:16:55

************************
EXCEPTION: St9bad_alloc
std::bad_alloc
bitcoin in ProcessMessages()

2017-11-26 08:16:55 ProcessMessage(block, 34514 bytes) FAILED peer=1
2017-11-26 08:16:55 mchn: Synced with node 2 on block 12622 - requesting mempool
2017-11-26 08:17:07 UpdateTip:            new best=0097791563c4827774dcda0c718bbf954e52d33041c7fd8d91fb2bc92888c939  height=12623  log2_work=21.623881  tx=1356372  date=2017-11-26 08:17:05 progress=1.000000  cache=0
2017-11-26 08:17:08

************************
EXCEPTION: St9bad_alloc
std::bad_alloc
bitcoin in ProcessMessages()

2017-11-26 08:17:08 ProcessMessage(block, 32803 bytes) FAILED peer=2
2017-11-26 08:17:08 ResendWalletTransactions()
2017-11-26 08:17:16 UpdateTip:            new best=00628a9d34e5d761f2fd1329fde20adb059e716d796afbb342329f974b406a1a  height=12624  log2_work=21.623996  tx=1356453  date=2017-11-26 08:17:14 progress=1.000000  cache=0
2017-11-26 08:17:17

************************
EXCEPTION: St9bad_alloc
std::bad_alloc
bitcoin in ProcessMessages()

 

So I upgraded the memory on node 2 to 4 Gb and cores to 2 cores and after that the daemon did start but the CPU usage on node 2 kept increasing again and it crashed again. 

 

So I decided to decrease the load to 3 TPS the CPU usage did start to decrease but at one point the daemon on node 1 crashed this time. At this stage my nodes were holding about 1.5M transactions or 17000 blocks. 

 

ee579785eeaec470ff9731239a3a9f0916479eb12ac745dabca5c0bc0d783b21, vin: 1, vout: 2
2017-11-27 03:06:03 MultiChainMiner: Block Found - 00b3450ee06a23a876f08f422ceea35fd7066b776d794d6d2f1862fead42d30b, prev: 00893ac41f12a2c4a19da894f62712a282028792fbb5bd7da7e4121413a633aa, height: 16893, txs: 8
2017-11-27 03:06:03 CommitTransaction: ae36226eb2da87ea00095b6b1da09e7cc9cd920bb56aabdf9c15f14e660df576, vin: 1, vout: 2
2017-11-27 03:06:03 ERROR: ContextualCheckBlock : block height mismatch in coinbase
2017-11-27 03:06:03 ERROR: ProcessNewBlock : AcceptBlock FAILED
2017-11-27 03:06:03 ERROR: MultiChainMiner : ProcessNewBlock, block not accepted
2017-11-27 03:06:04 CommitTransaction: 72db13b3405224560a1ec2513d4eba31ecbf8b15d4c993c695d2e572aa9ddab5, vin: 1, vout: 2
 

 

With so many crashes even with a relatively small load of 3 TPS is scary for me to run multichain on production. Since it shares the bitcoin core, stablility shouldn't be a challenge. Am I missing something here? or some parameters which I should change? Please suggest. 

Answer 1

Please use getwalletinfo on the node which crashes and post the output. The problem is likely that you have a very high number of unspent transaction outputs on the node which is crashing, and this is causing out of memory errors. This is not related to the transaction rate, but the total number of transaction outputs that have been received, and have never been combined using APIs like combineunspent or the autocombine functionality.

Comments

I did try to manually invoke the combineunspent  from CLI few times.


This is from node 2
[ec2-user@ip-172-31-22-144 ~]$ multichain-cli chain1 getwalletinfo
{"method":"getwalletinfo","params":[],"id":1,"chain_name":"chain1"}

{
    "walletversion" : 60000,
    "balance" : 0.00000000,
    "walletdbversion" : 2,
    "txcount" : 1639788,
    "utxocount" : 66866,
    "keypoololdest" : 1511426390,
    "keypoolsize" : 2
}



This is from Node1
[ec2-user@ip-172-31-26-41 ~]$  multichain-cli chain1 getwalletinfo
{"method":"getwalletinfo","params":[],"id":1,"chain_name":"chain1"}

{
    "walletversion" : 60000,
    "balance" : 0.00000000,
    "walletdbversion" : 2,
    "txcount" : 1639829,
    "utxocount" : 66851,
    "keypoololdest" : 1511416662,
    "keypoolsize" : 2
}


I 'm using sendassetfrom command to create the transactions, so why there are so many requests which are not getting combined? So in a production use -case should I use autocombine as a default setting?