appendbinarycache delay

Question

Hi!

 

When using appendbinarycache in terminal, trying to append a file over 3MB, delay could be up to 30 min and sometimes (over 10meg) process doesn't complete.

Do you have any idea why?

 

My steps:

1-createbinary cache => cacheLabel

2-PDF to hex => hex

3- appendbinarycache cacheLabel hex ----> sometimes take very long time or just doesn't complete.

4- publish streamName Key '{"cache":"cacheLabel"}'

 

Regards

Answer 1

The point of the binary cache is to avoid processing large amounts of data in JSON-RPC API requests, which is slow like you are seeing, but rather using the file system directly. So instead of using the appendbinarycache command, put the data directly into the binary cache file. This file in the cache subdirectory of the blockchain directory and its name is the identifier of the cache item returned by createbinarycache.

Comments

Thanks for that answer.

I tried as suggested.

1-createbinary cache => identifier

2-Dropping the 10MG pdf into my multichain cache folder.

3-Change the name of that pdf for the identifier and erase the original identifier file.
* I also tried to drop the hex of the pdf into the identifier file but get same result.

4- Send command on terminal : publish '{"cache":"identifier"}' offchain

This is the error message I get.

error code: -26
Error: The transaction was rejected: 16: Insane fees

I searched this forum for a solution and found : https://www.multichain.com/qa/19061/insane-fees-error?show=19061#q19061.

Still, I could not understand the solution: "The solution is to either include the native currency in the output you are specifying, or else consider using the appendrawchange command to add an extra output with the change (minus the fee specified)."

I use a native currency on this chain.

Regards.

---

This solution is not relevant in your case, since you're not using the raw transaction interface. Please post your blockchain parameters here so we can understand the fee structure and try to work out what is going on.

---

Thanks a lot for your support. Really appreciated.

chain-protocol = multichain            
chain-description = MultiChain MULTITEST1
root-stream-name = root              
root-stream-open = true                 
chain-is-testnet = false
target-block-time = 15  
maximum-block-size = 8388608  
maximum-chunk-size = 1048576        
maximum-chunk-count = 1024            

# Global permissions

anyone-can-connect = true             
anyone-can-send = true               
anyone-can-receive = true           
anyone-can-receive-empty = true         
anyone-can-create = false               
anyone-can-issue = false               
anyone-can-mine = false                
anyone-can-activate = false          
anyone-can-admin = false             
support-miner-precheck = true      
allow-arbitrary-outputs = false       
allow-p2sh-outputs = true               
allow-multisig-outputs = true         

# Consensus requirements

setup-first-blocks = 60                 # Length of initial setup phase in blocks, in which mining-diversity,
                                        # admin-consensus-* and mining-requires-peers are not applied. (1 - 31536000)
mining-diversity = 0.0                  # Miners must wait <mining-diversity>*<active miners> between blocks. (0 - 1)
admin-consensus-upgrade = 1             # <admin-consensus-upgrade>*<active admins> needed to upgrade the chain. (0 - 1)
admin-consensus-txfilter = 1            # <admin-consensus-txfilter>*<active admins> needed to approve filter or library in the chain. (0 - 1)
admin-consensus-admin = 1               # <admin-consensus-admin>*<active admins> needed to change admin perms. (0 - 1)
admin-consensus-activate = 1            # <admin-consensus-activate>*<active admins> to change activate perms. (0 - 1)
admin-consensus-mine = 1                # <admin-consensus-mine>*<active admins> to change mining permissions. (0 - 1)
admin-consensus-create = 1              # <admin-consensus-create>*<active admins> to change create permissions. (0 - 1)
admin-consensus-issue = 1               # <admin-consensus-issue>*<active admins> to change issue permissions. (0 - 1)

# Defaults for node runtime parameters

lock-admin-mine-rounds = 10             # Ignore forks that reverse changes in admin or mine permissions after this many mining rounds have passed. Integer only. (0 - 10000)
mining-requires-peers = true            # Nodes only mine blocks if connected to other nodes (ignored if only one permitted miner).
mine-empty-rounds = 10                  # Mine this many rounds of empty blocks before pausing to wait for new transactions. If negative, continue indefinitely (ignored if target-adjust-freq>0). Non-integer allowed. (-1 - 1000)
mining-turnover = 0.5                   # Prefer pure round robin between a subset of active miners to minimize forks (0.0) or random equal participation for all permitted miners (1.0). (0 - 1)

# Native blockchain currency (likely not required)

initial-block-reward = 0                # Initial block mining reward in raw native currency units. (0 - 1000000000000000000)
first-block-reward = 44400000000000000  # Different mining reward for first block only, ignored if negative. (-1 - 1000000000000000000)
reward-halving-interval = 52560000      # Interval for halving of mining rewards, in blocks. (60 - 1000000000)
reward-spendable-delay = 1              # Delay before mining reward can be spent, in blocks. (1 - 100000)
minimum-per-output = 0                  # Minimum native currency per output (anti-dust), in raw units.
                                        # If set to -1, this is calculated from minimum-relay-fee. (-1 - 1000000000)
maximum-per-output = 44400000000000000  # Maximum native currency per output, in raw units. (0 - 1000000000000000000)
minimum-offchain-fee = 1000000          # Minimum fee for publishing off-chain data items, per 1000 bytes, in raw units of native currency. (0 - 1000000000)
minimum-relay-fee = 1000000             # Minimum transaction fee, per 1000 bytes, in raw units of native currency. (0 - 1000000000)
native-currency-multiple = 100000000    # Number of raw units of native currency per display unit. (0 - 1000000000)

# Advanced mining parameters

skip-pow-check = false                  # Skip checking whether block hashes demonstrate proof of work.
pow-minimum-bits = 8                    # Initial and minimum proof of work difficulty, in leading zero bits. (1 - 32)
target-adjust-freq = -1                 # Interval between proof of work difficulty adjustments, in seconds, if negative - never adjusted. (-1 - 4294967295)
allow-min-difficulty-blocks = false     # Allow lower difficulty blocks if none after 2*<target-block-time>.

# Standard transaction definitions

only-accept-std-txs = true              # Only accept and relay transactions which qualify as 'standard'.
max-std-tx-size = 4194304               # Maximum size of standard transactions, in bytes. (1024 - 100000000)
max-std-op-returns-count = 32           # Maximum number of OP_RETURN metadata outputs in standard transactions. (0 - 1024)
max-std-op-return-size = 2097152        # Maximum size of OP_RETURN metadata in standard transactions, in bytes. (0 - 67108864)
max-std-op-drops-count = 5              # Maximum number of OP_DROPs per output in standard transactions. (0 - 100)
max-std-element-size = 40000            # Maximum size of data elements in standard transactions, in bytes. (128 - 80000)

# The following parameters were generated by multichain-util.
# They SHOULD ONLY BE EDITED IF YOU KNOW WHAT YOU ARE DOING.

default-network-port = 6757             # Default TCP/IP port for peer-to-peer connection with other nodes.
default-rpc-port = 6756                 # Default TCP/IP port for incoming JSON-RPC API requests.
chain-name = MULTITEST1                 # Chain name, used as first argument for multichaind and multichain-cli.
protocol-version = 20013                # Protocol version at the moment of blockchain genesis.
network-message-start = ffdef3e8        # Magic value sent as the first 4 bytes of every peer-to-peer message.
address-pubkeyhash-version = 006b3f68   # Version bytes used for pay-to-pubkeyhash addresses.
address-scripthash-version = 056c8bd1   # Version bytes used for pay-to-scripthash addresses.
private-key-version = 80707744          # Version bytes used for exporting private keys.
address-checksum-value = 59394315       # Bytes used for XOR in address checksum calculation.

# The following parameters were generated by multichaind.
# They SHOULD NOT BE EDITED.

genesis-pubkey = 022c052f041a51c3160535c42ede960091ff413f8e5d03d4b508a462a5cd4042bf # Genesis block coinbase output public key.
genesis-version = 1                     # Genesis block version.
genesis-timestamp = 1636911099          # Genesis block timestamp.
genesis-nbits = 536936447               # Genesis block difficulty (nBits).
genesis-nonce = 123                     # Genesis block nonce.
genesis-pubkey-hash = 48bd924f6740aff3bda6c213b2d99b7b9d4ebd82 # Genesis block coinbase output public key hash.
genesis-hash = 0016c4e0dd50c447566ee97db26e0c26e5512841d0b282cae1e4638d1220ffa9 # Genesis block hash.
chain-params-hash = 30fd47a16c6f146ebb81c9842c92e0a0a2036e20c3882975d782d4b7c3add8df #

---

Thanks. Presumably this is happening because of the minimum-offchain-fee parameter, which means the fee for publishing such a large offchain item is very large. The node is protecting you from sending a transaction with such a high fee by mistake.

You might want to review this blockchain parameter to check that it makes sense, and restart your blockchain. If not, the other option is to send the transaction using a low-level API. First call this:

createrawsendfrom <from-address> '{}' '[{"for":"<stream id>","key":"<key>","data"{"cache":"<cache id>"}}]'

Then pass the result of this call to sendrawtransaction with a second parameter true (to ignore the high fees).

---

Hi, lowering the minimum-offchain fee worked. Thanks! I tested uploading files offchain up to 250meg!

Now I'm trying to retrieve a offchain file of 100meg with gettxoutdata but I get "Invalid count, must be below 64MB".

Do we need to slice the file per 64 meg into the binary cache to retrieve it? I though we could upload up to 1G file offchain.

Thanks for you time!

---

The gettxoutdata command retrieves data via the API, which is quite inefficient (double the bytes due to hexadecimal), which is why this limit is in place.

If you want to retrieve the data via the binary cache (i.e. file system), you should use the txouttobinarycache command, which does not have this limit.