Rolling FIFO data stream to a groov data store table

Sometimes there is a single point of data coming into Node-RED that I want displayed on a groov page, but I find a single gadget isn’t quite enough – I want to keep a rolling history of values where the First In is the First Out (FIFO), and I specifically don’t want any duplication of data in my history either.

In this post I’ll show the flow and JavaScript I used to do get a rolling feed of ten strings from a twitter feed, but this could work with almost any other data source as well.

My data is a tweet string in msg.payload that comes from the raw public tweet stream filtered for the hastag #IoT, I get at most one of these per second thanks to a delay node in rate limit mode that discards intermediate messages. At this point your data could come from anywhere.
Wherever your data is injected from, use that flow to pick up the entire data store table, in my case the tag tweets:

When I use the groov read node to pick up the table I make sure to grab the full table length of 10 and put the array in a new message property (msg.tweets) so that the string sitting in msg.payload is not written over.

Once I have the table I check to see if the tweet is in English. Here I could use the or operator (||) to allow other languages, scan the location of the tweet, or any other number of filter checks before it gets compared to the list.
This shows some simple filtering that can be done before adding the item in msg.payload to the list.

if(msg.lang != ‘en’) return null;

If the flow is not ended by return null then the tweet text is in English, so I make a copy of the list of tweets currently in groov using .slice(), and make a copy of the tweet text to keep the rest of the code generic.

var list = msg.tweets.slice();
var newItem = msg.tweet.text;

After that I check to see if the new item has a valid index in the list, if so it is already being displayed and I should end the flow.
This statement handles any potential duplication

if(list.indexOf(newItem) >= 0) return null;

At this point in the code I have accepted the new data since it passed the filters and is new to the list, so I can roll down the previous values to add the new item to the top.
Copy each item down one index, starting at the bottom of the list and working backwards, then assign the new item at the start of the array.
This for loop handles the “rolling” part of the flow.

for(i = list.length-1; i > 0; i--)
    list[i] = list[i-1];
list[0] = newItem;

Now the new list should be written to the groov table, which I do using an array of dynamic write objects.

var data = [];
for(i = 0; i < list.length; i++)
    data.push({ tableStartIndex : i, value : list[i] });
return { payload : data };

All together this gives me a nice, relatively short function block to handle all of the features I need:

I have an extra delay after this function node to make sure I have time to read each new tweet that comes in, your application may or may not call for it so use your best judgement.

Also Since the #IoT public tweet stream is quite lively I also drop intermediate messages since that could turn into a very obnoxious message queue. If I were to set the node to a lower traffic feed, like tweets from people I follow, I would likely untick this box or not even bother with the delay.

Once the message has been delayed if necessary I can split the payload with a default split node and fire each of the 9 old tweets + the one new tweet at the write node set to push the data in msg.payload.value to my data store table tweets using the index from the dynamic write objects.

Now I can deploy the flow and watch the stream roll by in groov:

Happy coding!

[{“id”:“c5958a70.d8b948”,“type”:“debug”,“z”:“b0835392.0631c”,“name”:"",“active”:true,“console”:“false”,“complete”:“true”,“x”:750,“y”:420,“wires”:[]},{“id”:“382b60f4.48bf6”,“type”:“groov-read-ds”,“z”:“b0835392.0631c”,“dataStore”:“41793f8c.cbbde”,“tagName”:“tweets”,“tableStartIndex”:"",“tableLength”:“10”,“value”:“tweets”,“valueType”:“msg”,“topic”:"",“topicType”:“none”,“name”:"",“x”:450,“y”:420,“wires”:[[“1b74fefe.795001”]]},{“id”:“1b74fefe.795001”,“type”:“function”,“z”:“b0835392.0631c”,“name”:“add tweet?”,“func”:“if(msg.lang != ‘en’) return null; // end flow if not certain language(s)\nvar list = msg.tweets.slice(); // copy array into ‘list’\nvar newItem = msg.tweet.text; // new item being checked\n// if the new item has a valid list index it must already be in the list, so end the flow here\nif(list.indexOf(newItem) >= 0) return null;\n\n// otherwise shuffle everything down one\nfor(i = list.length-1; i > 0; i–)\n list[i] = list[i-1];\nlist[0] = newItem; // put the new tweet at the top of the list\n// the new tweet list will need to be rewritten\nvar data = []; // array of dynamic write objects\nfor(i = 0; i < list.length; i++) // build a write object for each table index\n data.push({ tableStartIndex : i, value : list[i] });\nreturn { payload : data }; // data array becomes the new payload. to be split.”,“outputs”:1,“noerr”:0,“x”:590,“y”:420,“wires”:[[“c5958a70.d8b948”,“83c94f38.43bb”]]},{“id”:“bfe7d036.70e”,“type”:“split”,“z”:“b0835392.0631c”,“name”:"",“splt”:"\n",“spltType”:“str”,“arraySplt”:1,“arraySpltType”:“len”,“stream”:false,“addname”:"",“x”:750,“y”:460,“wires”:[[“1d1ad5db.39789a”]]},{“id”:“1d1ad5db.39789a”,“type”:“groov-write-ds”,“z”:“b0835392.0631c”,“dataStore”:“41793f8c.cbbde”,“tagName”:“tweets”,“tableStartIndex”:"",“value”:“payload.value”,“valueType”:“msg”,“name”:"",“x”:870,“y”:460,“wires”:[[]]},{“id”:“ea717d06.45125”,“type”:“delay”,“z”:“b0835392.0631c”,“name”:"",“pauseType”:“rate”,“timeout”:“5”,“timeoutUnits”:“seconds”,“rate”:“1”,“nbRateUnits”:“1”,“rateUnits”:“second”,“randomFirst”:“1”,“randomLast”:“5”,“randomUnits”:“seconds”,“drop”:true,“x”:470,“y”:380,“wires”:[[“382b60f4.48bf6”]]},{“id”:“83c94f38.43bb”,“type”:“delay”,“z”:“b0835392.0631c”,“name”:"",“pauseType”:“rate”,“timeout”:“5”,“timeoutUnits”:“seconds”,“rate”:“1”,“nbRateUnits”:“10”,“rateUnits”:“second”,“randomFirst”:“1”,“randomLast”:“5”,“randomUnits”:“seconds”,“drop”:true,“x”:600,“y”:460,“wires”:[[“bfe7d036.70e”]]},{“id”:“53591c1a.9761a4”,“type”:“twitter in”,“z”:“b0835392.0631c”,“twitter”:"",“tags”:"#IoT",“user”:“false”,“name”:"",“topic”:“tweets”,“inputs”:1,“x”:330,“y”:380,“wires”:[[“ea717d06.45125”]]},{“id”:“41793f8c.cbbde”,“type”:“groov-data-store”,“z”:"",“project”:“5c31575e.43ac48”,“dsName”:“NodeRED”},{“id”:“5c31575e.43ac48”,“type”:“groov-project”,“z”:"",“address”:“localhost”}]

2 Likes

Hi @torchard,

Do you use any gadget in groov view to display those output? Thanks

You probably want to use a text area gadget to display each line of the file.

@Beno

I am using text area gadget but groov view didn’t show any inserted data. I inject string to my flow like this:


Debug node didn’t show any error. Array are fully occupied.
image
Function node is quite similar with @torchard
image
I am not sure where I did it wrong

The first things I would double check are the groov nodes. Can you confirm that “Table” input is writing to msg.table and not msg.payload? Also make sure that the “Table” output node does not have anything set for the index, has msg.payload.value set for the value, but make sure the table name is set for the data store tag name.

If those nodes are configured correctly we might have to have you put in some extra debug nodes to track down what’s going on here.

@torchard

Everything I have double confirm. Here is the groov node configuration. Tag from Data Store named ‘Test’ and Tag name ‘Table’. Groov Address set to ‘localhost’
image image
I am wondering whether is the tag name configuration inside groov view gone wrong.
imageimage

Ah, I see the problem… You need the # sign in the text area gadget.
Just the one will do the job, it will take the string from Node-RED and display it.
Here is what it should look like;

Note you can always click on the ‘Formatting help’ to get more information about the text area gadget.

2 Likes

@Beno

The displaying problem already solved. Thanks for your solution. But it can only show Table[0] value because of configuration in the tag name. What should I put in this index area?
image

Each line of the report has a index.
You start at index zero and then on the next line, you place a new text area gadget and it is connected to the same Table tag name, but it will be index one, then index two and so on till index nine.
At that point you will have ten text area gadgets and each will display a line of the report.

2 Likes

Thanks, @Beno and @torchard
So there is only this way to display the whole list in groov view ya. Thanks for your help all this way.