Chain unresponsive - IO error: lock ../chain/permissions.db/LOCK: Resource temporary unavailable

Question

So basically, i have set up a chain and created a couple of streams.I have spammed a chain with 'liststreamkeyitems' API occasionally with invalid stream name as a parameter (Stream with this name not found error). At some point, chain became unresponsive, see http://prntscr.com/dwr8ny

When i try to run the chain again - i get this message:

"ERROR: Couldn't initialize permission database for blockchain chain. Probably multichaind for this blockchain is already running. Exiting..."

 

Also at some point (i could not reproduce it now) i get an error that i've put as a topic name - IO error: lock ../chain/permissions.db/LOCK: Resource temporary unavailable

Permissions log has this message at one line:

2017-01-17 08:33:41.169 Initialize: Database corrupted 1

So what may be the cause of corruption and how to avoid it? At some point, chain will "unfreeze" and continue to work properly.

Note that i have put multiple "rpcallowip" parameters as well as high "maxshowndata" in multichain.conf.

Also i have put max available number for transaction size parameters in params.dat - since i am sending big metadata into stream.

Thank you in advance.

Answer 1

First, can you please let us know what version you are using? Presumably you hard-exited the multichaind process then launched it up again, but this corruption should not have happened.

If you are willing, it would be helpful to see your blockchain directory files. If so, before doing anything else, please zip the blockchain directory into an archive, and send to multichain.debug at gmail dot com.

Second, you should be able to fix the corrupted database by running:

multichaind chain -reindex

Comments

Im using up do date version (27). I think i didn't hard exit - i have just spammed "listkeystreamitems" and permissions got corrupted and i was not able to do anything. I will send you directory files related to this chain if that is what you meant. (.multichain/chain)

---

Thanks - we received it and will take a look.

---

Unfortunately we don't see anything special in the directory files you sent us, because they don't have history before you run with reindex=1. It looks like chain is OK now and we don't see anything special about it (it was created by older version of MultiChain, protocol=10006, but we support these chains too). Activity in the chain and stream items are perfectly normal.

I suppose the intital problem was unresponsive API, this may explain all other issues. Unresponsive API means multichaind is running, but for some reason it is stuck somewhere and don't respond to API calls (and probably doesn't perform lot of other actions...). It is very strange behaviour and we've never seen this in our tests, though we've seen this in development when we had a bug. It may be related to some issue in your system, but  if it happens again we would love to understand the reason. To do this, please send us directory files BEFORE you make reindex. Also, if you don't send thousands of transactions, running multichaind with the following switches may help to diagnose the problem (this increases the size of debug.log a bit) :

multichaind chain -daemon -debug=mchn -debug=mcapi

The rest of the problems can be related to this. You see "lock" message, but you also see  "Probably multichaind for this blockchain is already running", which is fine as multichaind is indeed running.

I suppose you've seen "Database corrupted" message after you killed stuck multichaind. This also can happen if you hard-exited multichaind. It is rare, but if this happen, reindex=1 is the solution, and it worked properly.