Read text file every line and do comparison on it

tansteven96249464 · June 2, 2020, 8:43am

Above is a demo flow to insert Chat Id into a text file. (Text file is inside Groov Epic)
I have met a problem when I want to read the data inside the text file. The reason why I use a function node to read the text file is that I need a WHILE LOOP to read every single line and do a comparison with each line. But I found out that the javascript in Node-Red can’t recognize the fs function(read a text file in javascript).
Example:

Scenario:
When the user enters the Chat Id that already exists inside the text file, it will not reinsert again.

loren1 · June 2, 2020, 9:32am

The problem is likely due to the Node-RED function node not supporting the require() method. I’m not sure why they made that decision - likely security related - but you’ll find discussions on the Node-RED forum about it.

This behavior can be changed by modifying the settings.js file, which can be done with SSH access.

Here’s a link to a thread in their forum: Using "require" inside function nodes · Issue #41 · node-red/node-red · GitHub

Beno · June 2, 2020, 1:59pm

@tansteven96249464 Could you by chance post your example flow so we can experiment with it a little?

If so, please post it using the </> preformated text option in the forum so that the structure remains intact.
Thanks.

tansteven96249464 · June 3, 2020, 1:05am

[{"id":"5894a44.f86d75c","type":"groov-read-ds","z":"459dfd6b.8cb174","dataStore":"64a7d751.184248","tagName":"Insert","tableStartIndex":"","tableLength":"","value":"","valueType":"msg.payload","topic":"","topicType":"none","name":"Insert Button","x":430,"y":380,"wires":[["f6219571.c104d8"]]},{"id":"39b50edf.5636c2","type":"inject","z":"459dfd6b.8cb174","name":"","topic":"","payload":"","payloadType":"date","repeat":"0.5","crontab":"","once":false,"onceDelay":0.1,"x":250,"y":380,"wires":[["5894a44.f86d75c"]]},{"id":"f6219571.c104d8","type":"switch","z":"459dfd6b.8cb174","name":"","property":"payload","propertyType":"msg","rules":[{"t":"true"}],"checkall":"true","repair":false,"outputs":1,"x":570,"y":380,"wires":[["2a7369df.e20766"]]},{"id":"caba7616.c1a8b8","type":"function","z":"459dfd6b.8cb174","name":"Check ID format","func":"var abc = msg.payload;\n\nif (abc === \"\") {\n    msg={payload:'Insert box cannot be empty'}\n    return [msg, null];\n}else{\n    //msg.payload = abc;\n    return [null, msg];\n}\n","outputs":2,"noerr":0,"x":760,"y":420,"wires":[["afcd2d44.cfb29"],["6c1a5593.e49fdc"]]},{"id":"2a7369df.e20766","type":"groov-read-ds","z":"459dfd6b.8cb174","dataStore":"64a7d751.184248","tagName":"ChatID","tableStartIndex":"","tableLength":"","value":"","valueType":"msg.payload","topic":"","topicType":"none","name":"Read ChatID","x":750,"y":380,"wires":[["caba7616.c1a8b8"]]},{"id":"2211e786.8efe58","type":"debug","z":"459dfd6b.8cb174","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","x":1130,"y":380,"wires":[]},{"id":"cbc67811.afc7b8","type":"file","z":"459dfd6b.8cb174","name":"","filename":"/home/dev/unsecured/RealData.txt","appendNewline":true,"createDir":true,"overwriteFile":"true","encoding":"none","x":1060,"y":460,"wires":[["86ae1c37.7a1cf"]]},{"id":"86ae1c37.7a1cf","type":"debug","z":"459dfd6b.8cb174","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","x":1290,"y":460,"wires":[]},{"id":"afcd2d44.cfb29","type":"groov-write-ds","z":"459dfd6b.8cb174","dataStore":"64a7d751.184248","tagName":"ShowText","tableStartIndex":"","value":"","valueType":"msg.payload","name":"","x":980,"y":380,"wires":[["2211e786.8efe58"]]},{"id":"6c1a5593.e49fdc","type":"function","z":"459dfd6b.8cb174","name":"Matching with text file","func":"var path = \"/home/dev/unsecured/RealData.txt\";\nvar fs = require(\"fs\");\n\nvar readFile = function(path) {\n    return fs.readFileSync(path).toString();\n};\n \nconsole.log(readFile('file.txt'));\n\nreturn msg;","outputs":1,"noerr":0,"x":780,"y":460,"wires":[["cbc67811.afc7b8"]]},{"id":"64a7d751.184248","type":"groov-data-store","z":"","project":"18830a55.119306","dsName":"Telegram ChatID"},{"id":"18830a55.119306","type":"groov-project","z":"","address":"Localhost"}]

Thanks for reply @loren1 and @Beno
@Beno Above is a simple example flow. This flow will associate with Telegram Bot as a Chat ID verification system. I need to create an interface in the groov view to let user key in their Chat ID so that the user no need to go into Node-Red which requires a PC. The EPIC work with no PC in a panel. This will be easier for the client that has no knowledge of Node-Red as well.

@loren1 Is that possible I can modify the Node-Red setting.js file that located inside the groov Epic device? If can, how should I change? Pardon for my ignorance.

Beno · June 3, 2020, 1:12am

First up, thanks for posting the flow, we will take a look at it…
But, that said… your post raises a LOT of questions…

Node-RED does NOT require a PC, it runs on EPIC.
The groov View user has no clue if he is using Node-RED or groov, or PAC for that matter.

Perhaps you can give us a bit of an overview of what you are trying to do here, there might be some angles that have not been considered.

tansteven96249464 · June 3, 2020, 1:54am

Oh, I think I didn’t state clearly, my bad.
What I mean is User have to login to the Node-Red to insert their User ID into the function node. But to open node-red we require a PC to do it.
Without PC, there is no way to insert the Chat ID.
Thats why I need to create an interface in groov view so they can straight away key in their Chat ID from groov view.

loren1 · June 3, 2020, 2:34am

Indeed, your function node is attempting to require(‘fs’) and that won’t work in Node-RED without altering the settings.js file.

My advise would be to stick with Node-RED nodes, and use the file in node to read your file, and process the output of the file in node with your function node.

Beno · June 3, 2020, 3:31pm

@tansteven96249464 Do you have a small example of the text file you are needing to check?
We are looking at your flow and have some ideas, but need a just a little more of you help to make sure we get it right…

torchard · June 3, 2020, 11:09pm

@tansteven96249464 depending on the format of your data file, for example if each piece of data in is on a separate line, you can use the file in node to output a single message per line, then join them together into an array with a join node. Once the data is in an array you can easily find the index of your chat in the array with either a function or a for loop.

Check out this example; it just checks a few test words on separate lines from the RealData.txt file and finds the index of msg.id:

[{"id":"986aa422.4437f8","type":"file in","z":"3c21729.4a51d8e","name":"","filename":"/home/dev/unsecured/RealData.txt","format":"lines","chunk":false,"sendError":false,"encoding":"none","x":340,"y":420,"wires":[["52a66686.db1328"]]},{"id":"52a66686.db1328","type":"join","z":"3c21729.4a51d8e","name":"","mode":"custom","build":"array","property":"payload","propertyType":"msg","key":"topic","joiner":"\\n","joinerType":"str","accumulate":false,"timeout":"","count":"","reduceRight":false,"reduceExp":"","reduceInit":"","reduceInitType":"","reduceFixup":"","x":570,"y":420,"wires":[["913b9e19.23145"]]},{"id":"7db796b7.af35e8","type":"inject","z":"3c21729.4a51d8e","name":"","topic":"","payload":"","payloadType":"date","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":100,"y":420,"wires":[["986aa422.4437f8"]]},{"id":"913b9e19.23145","type":"change","z":"3c21729.4a51d8e","name":"Read ChatID","rules":[{"t":"set","p":"id","pt":"msg","to":"that","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":730,"y":420,"wires":[["a9db77c2.adef38"]]},{"id":"a9db77c2.adef38","type":"function","z":"3c21729.4a51d8e","name":"Find match","func":"index = msg.payload.indexOf(msg.id);\nif(index != -1)\n{\n    // do something with the chat ID at this index\n    return { payload : index };\n}\nelse\n{\n    // this chat ID is not listed in the file\n}\nreturn msg;","outputs":1,"noerr":0,"x":910,"y":420,"wires":[["9dcd6b15.e16328"]]},{"id":"9dcd6b15.e16328","type":"debug","z":"3c21729.4a51d8e","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","x":1090,"y":420,"wires":[]}]

tansteven96249464 · June 4, 2020, 6:30am

@Beno

Inside the text file is only contain these Chat ID. As you can see there is duplicate value in it, so we need to program the system not to take the same value again to store it into the text file. Perhaps tips from @torchard can solve my problem.

tansteven96249464 · June 4, 2020, 7:32am

[{"id":"5894a44.f86d75c","type":"groov-read-ds","z":"459dfd6b.8cb174","dataStore":"64a7d751.184248","tagName":"Insert","tableStartIndex":"","tableLength":"","value":"","valueType":"msg.payload","topic":"","topicType":"none","name":"Insert Button","x":430,"y":380,"wires":[["f6219571.c104d8"]]},{"id":"39b50edf.5636c2","type":"inject","z":"459dfd6b.8cb174","name":"","topic":"","payload":"","payloadType":"date","repeat":"0.5","crontab":"","once":false,"onceDelay":0.1,"x":250,"y":380,"wires":[["5894a44.f86d75c"]]},{"id":"f6219571.c104d8","type":"switch","z":"459dfd6b.8cb174","name":"","property":"payload","propertyType":"msg","rules":[{"t":"true"}],"checkall":"true","repair":false,"outputs":1,"x":570,"y":380,"wires":[["2a7369df.e20766"]]},{"id":"caba7616.c1a8b8","type":"function","z":"459dfd6b.8cb174","name":"Check ID format","func":"var abc = msg.payload;\n\nif (abc === \"\") {\n    msg={payload:'Insert box cannot be empty'}\n    return [msg, null];\n}else{\n    //msg.payload = abc;\n    return [null, msg];\n}\n","outputs":2,"noerr":0,"x":760,"y":420,"wires":[["afcd2d44.cfb29"],["fd50b011.26858"]]},{"id":"2a7369df.e20766","type":"groov-read-ds","z":"459dfd6b.8cb174","dataStore":"64a7d751.184248","tagName":"ChatID","tableStartIndex":"","tableLength":"","value":"","valueType":"msg.payload","topic":"","topicType":"none","name":"Read ChatID","x":750,"y":380,"wires":[["caba7616.c1a8b8"]]},{"id":"2211e786.8efe58","type":"debug","z":"459dfd6b.8cb174","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","x":1350,"y":440,"wires":[]},{"id":"cbc67811.afc7b8","type":"file","z":"459dfd6b.8cb174","name":"","filename":"/home/dev/unsecured/RealData.txt","appendNewline":true,"createDir":true,"overwriteFile":"false","encoding":"none","x":940,"y":580,"wires":[["3aa8bafd.e8bc96"]]},{"id":"afcd2d44.cfb29","type":"groov-write-ds","z":"459dfd6b.8cb174","dataStore":"64a7d751.184248","tagName":"ShowText","tableStartIndex":"","value":"","valueType":"msg.payload","name":"","x":1200,"y":440,"wires":[["2211e786.8efe58"]]},{"id":"7d7075cc.e38c7c","type":"join","z":"459dfd6b.8cb174","name":"","mode":"custom","build":"array","property":"payload","propertyType":"msg","key":"topic","joiner":"\\n","joinerType":"str","accumulate":false,"timeout":"","count":"","reduceRight":false,"reduceExp":"","reduceInit":"","reduceInitType":"","reduceFixup":"","x":730,"y":540,"wires":[["8062ba01.041d28"]]},{"id":"fd50b011.26858","type":"change","z":"459dfd6b.8cb174","name":"Read ChatID","rules":[{"t":"set","p":"ChatID","pt":"global","to":"payload","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":750,"y":460,"wires":[["fa91d65f.6ff6f8"]]},{"id":"8062ba01.041d28","type":"function","z":"459dfd6b.8cb174","name":"Find match","func":"var arr = msg.payload;\nvar chatid = global.get(\"ChatID\");\n\nfor(count = 0; count < arr.length; count++)\n{\n    if(arr[count] === chatid )\n    {\n        msg={payload:\"Chat ID already exists.\"};\n        return [msg, null];\n    }\n\n}\nmsg={payload:chatid}\nreturn [null, msg];","outputs":2,"noerr":0,"x":870,"y":540,"wires":[["afcd2d44.cfb29"],["cbc67811.afc7b8"]]},{"id":"fa91d65f.6ff6f8","type":"file in","z":"459dfd6b.8cb174","name":"","filename":"/home/dev/unsecured/RealData.txt","format":"lines","chunk":false,"sendError":false,"encoding":"none","x":820,"y":500,"wires":[["7d7075cc.e38c7c"]]},{"id":"3aa8bafd.e8bc96","type":"function","z":"459dfd6b.8cb174","name":"","func":"var chatid = global.get(\"ChatID\");\nmsg={payload:chatid+\" has been added to the text file\"}\nreturn msg;","outputs":1,"noerr":0,"x":1150,"y":580,"wires":[["afcd2d44.cfb29"]]},{"id":"64a7d751.184248","type":"groov-data-store","z":"","project":"18830a55.119306","dsName":"Telegram ChatID"},{"id":"18830a55.119306","type":"groov-project","z":"","address":"Localhost"}]

Thanks for all the replies, I already figure it out how to do.