JSON validation in smart filter

Question

Hi guys,

is there a good way to validate a JSON structure of an item published to a stream in a stream  filter using JSON schema notation? There are several JS libs out there which validate JSON against a JSON schema but you cannot load external JS libs right?

Do I have to code every JSON structure checking myself with basic JS myself or what do you suggest?

It would be nice to have a JSON validation without much coding!

Thanks!

Answer 1

We don't yet have JSON schema validation built in, though it is a good idea for future improvement.

In the meantime you can include one of these libraries as part of your smart filter code, and then call it from filtertransaction(). There's no problem including additional functions in a filter.

If you want to experiment with this easily, the MultiChain Web Demo might be helpful. You should of course test the performance of the filter using the testtxfilter command, hopefully it should run fast because of the V8 engine compilation and caching.

Comments

thanks! Does it also work in the filterstreamitem function which I use? So you mean I have to copy the complete schema lib code in there in a separate function? That could be much code...
these libs here: https://json-schema.org/implementations.html#validator-javascript work with nodejs. Are node.js require or ES6 import modules supported?

---

You, it would also work in filterstreamitem, and you would need to include the full validation code. There is no support for including/requiring standard or external JS libraries. It would be far too risky considering that we need to be fully deterministic with every node giving the same response for the same input.

---

thanks! Have you tested streamfilters with many >1000 lines of code?
Is it suitable for that or do you recommend just basic checks?

---

Yes, in the latest protocols there is support for long filter code. There is no hard limit but you might find you need to tweak the acceptfiltertimeout and sendfiltertimeout runtime parameters if you see a timeout error. (Note that stream filters are not consensus critical so there is no risk of breaking the network if nodes have different settings for these parameters, or if different nodes are on servers with different performance.)

---

thanks!
we found a buffer overflow. My colleague Alex will contact Gideon

---

Thanks, it's probably just a bug in getfiltercode, not in the filtering itself. Can you please post the full error response you get?

---

thanks!
we found a buffer overflow. My colleague Alex will contact Gideon

---

Thanks, we've now identified the bug as well.

---

Thanks! Another question: If I load/declare a new class/function in the smart filter which stays stable where I do initialization steps. e.g. load library etc. Is this then done again and again for each item publish or is that somehow kept and can be reused again without initializing? That way it would mean that less code needs to run again for each item and performance could be increased. Or can I somehow run initial function declarations only one time so that they are declared on this pointer and then reuse them in each stream filter call for each item? Is this done automatically?

---

The code in the smart filter is compiled by V8 when first used, and then executed in that compiled state for every new stream item. So you should see that the performance for writing/reading multiple items is much faster than for the first one. We'd be interested to hear what performance you see with and without the stream filter.