Groov EPIC Performance Issue

[jacksonlance02]

Hi Team, I was hoping to get some general advice on performance and speeds. We currently have 8 charts running on PAC control and very little code in node-RED, which is just logging to a SQL database. I stopped PAC Control and the speed went from 98% down to 84%, but typically without PAC control strategy running, the system sits at 60% (give or take 4%).

Can anyone advise me what the problem might be? Happy to provide any logs necessary. I did check through my logs and didn’t see anything too fishy, but I don’t know particularly what I’m looking for either.

Stopping Node-RED while running PAC Control makes the CPU usage go from 97% down to 80%.

[Beno]

I wonder if you have checked out my blog this week?
Sounds like it might be worth a look at your charts…

Add delays to speed up

[Beno]

@jacksonlance02 your question has been tugging at some lose strings in my brain the past few days…
My EPIC sits at about 12% CPU most of the time.
With about 10 to 12 test charts running (with delays) its about 18%
I also have about 20 Node-RED flows doing different things.

Are you running groov View on this EPIC?

Do you have shell access turned on in your EPIC?
If so, what does the command htop show?
Im wondering why your CPU is so high with it at rest.

[jacksonlance02]

I can’t say I know what I’m looking at, but it doesn’t look good regardless.

I will clarify a few things:

We are running Groov View and most of the gadgets running on Groov View are being derived from the I/O, as most of our I/O is just analog inputs that don’t go through PAC Control first.

We are also running Node-RED for logging to a My-SQL database located on a NAS server elsewhere in the network. We also use Node-RED to scan for I/O changes and has Telegram messaging to notify of these changes.

We run 8 charts on PAC Control, which I have since added delays to - this seemed to make a decent difference however it still performs in the late 10’s of % (as opposed to the early 20’s, before adding delays).

I am in the middle of figuring out a better logging method because I’m not sure that Node-RED is the best option for this, we have far too much data that needs to be logged, and Node-RED can’t keep up. We do however need to keep using Node-RED for communications from Denkovi I/O Boards and Emergency Audio Announcements.

After implementing the Delay changes to PAC, my total CPU usage has dropped to sit around 75% now which is incredible. It looks like the issue is Node-RED (rather, the way I have constructed it).

Update: I have cut all useless crap out of Node-RED that has kind of just been lurking in the background from previous trials and tribulations, and now Node-RED performs much better, but overall system performance CPU usage sits around 65%, and Node-RED is reporting 35%-45% CPU usage.

Because of the three different Systems we use, I am interacting with I/O a lot (Groov View, Node-RED, PAC Control). Would this be the main CPU greed here? There would be a lot of crossover. Granted I have given very generous Delays throughout PAC Control and Node-RED has generous inject intervals where I have the control over the inject rate, I’m thinking the I/O Read Nodes I am using to scan for changes within in our I/O units would be causing some grief.

Apologies that this post has turned into a novel, would just prefer to get every known vector out on to paper.

[gchase]

In Node Red, do you have multiple I/O read nodes for the same data type?(int32,float,ai….) basically one read node per value?
Or do you have a single read node for each value type that returns an array of values?

[jacksonlance02]

one read node per i/o… for example, I have circa 30 Analog Input 4-20maH temperature sensors and they all have their own read nodes. I would love to learn how to make this more modular by returning an array? I didn’t know that was possible with the I/O nodes.

This is a small example of the messaging function using Read Nodes, when the change is triggered it sends a message via telegram.

[gchase]

@jacksonlance02 Im not sure it is possible as I use snap read nodes, these can return an array, I have never used nodes for epic, we will see what @Beno says.

[Beno]

Thanks for the htop screenshot (a little wider would have been fantastic - note for next time).
Also thanks for the Node-RED example - more on that shortly.

I marked up your HTOP so we can break it down a bit…

1. My gut experience tells me that a load average of 5.0 is pretty high, which is telling you nothing you don’t know - hence this post in the first place! But its a nice sanity check for both of us.
This is a hard number to get ones head around, but if you want to take a deep dive into all things HTOP, this is one read you are going to need to take your time with: htop explained | peteris.rocks

2. Yup, Node-RED is top of the CPU list. @torchard will be along shortly with some suggestions once he and I go over your post and screenshots.

3. I think this is the ‘self scanning’ I/O nodes. That is something I am going to focus on with Terry. They are hard coded to scan at once a second and that’s a bit high since you seem to have quite a few of them and they probably don’t need to be all updating at once a second.

4. I think is groov View (your screenshot cuts off some important names on the far right).
You mention that you have one or more I/O units in groov View because you just need to get the data and don’t need it going via PAC Control. This is good. No problem there.
My question is, do you need to poll those I/O points at the default once a second?

Can you back this off? Even once every 2 or three seconds would make a big difference.
Do you have more than one I/O unit configured in groov View?
Are they all setup to scan at once a second? Can any of them be different rates?
You get the idea.

5. The MMP scanner is the Linux process that is connecting the CPU to the I/O. Its busy because of all the Node-RED I/O nodes (once a second) and the groov View I/O units (once a second) and your PAC Control charts - now with added speed delays < grin >.
As you tweak Node-RED and groov View, this will calm down.

6. I think this is the REST sever (screenshot cut off). Those Node-RED I/O nodes are polling via the REST server and so this is pretty busy doing all that authentication. MMP is quicker. I will flag that to mention to Terry.

7. I could be wrong, but I think the MMP scanner is split into digital and analog. Hence two of these processes.
You can see whats going on with the scan times via groov Manage I/O

8 and 9. Is two of your 64 charts. I know you may not have 64 running charts, but if you look at HTOP you will see 64 instances of SoftPAC, any charts that are not running are still listed, but of course not taking up much / any CPU.
This 17% and 11.6% CPU is not horrible, but also not great. I am glad that adding delays has sped things up for you. Be sure and have a delay in every path through every chart. Often times delays are only added to condition branches or like my blog, super obvious looping condition branches and main loops through the chart are forgotten. This 17% & 11.6% is telling me you still have some chart paths that might benefit from a delay review.

10. This is nice chart. Only 4.8% CPU. Good delays in this one.

Ok, bit of novel to chew through.
Hope it helps and will get Terry to add his Node-RED speed tips to this thread.

BTW, I have a long running task to rewrite the ‘best practices’ and ‘optimization’ docs into one and focus on the groov family devices in said re-write, so your topic is another reminder that I really need to get that job done!

[torchard]

@jacksonlance02

one read node per i/o… for example, I have circa 30 Analog Input 4-20maH temperature sensors and they all have their own read nodes. I would love to learn how to make this more modular by returning an array? I didn’t know that was possible with the I/O nodes.

The groov i/o read nodes do support reading an entire module at once, but the input node that your using is only per-channel.
Each channel really needs its own deadband, and since the input node relies on a deadband, you can only do one at a time.

There is a work around though! You would definitely benefit from reducing the volume and frequency of REST requests with these nodes, so just do that one module request, break out the separate channels, and deadband them separately.
The “tricky” part is splitting up the payload array into individual flow wires, but you can set a function node to have as many outputs as your module has channels.

Here’s an example I threw together for a 12 channel module:

The function node has 12 outputs set in the Setup tab, for however many channels the module has. Then, the code in the “separate” function node:

var payloadList = [];

for(var i in msg.payload)
    payloadList.push({payload: msg.payload[i].value});

return payloadList;

This takes the array of objects, goes through each and grabs the value property, and adds that to an array as a { "payload" : <value> } object. By returning that array it will send the individual objects through each output port.

Once each channel is on it’s own wire, you can deadband them, process them, and do whatever else you need individually, while doing way less REST requests for the same amount of data.

If you want to use this flow as a baseline to see how it works, here’s the import JSON:

[{"id":"6fbf2e97a94b4290","type":"function","z":"2ea736ebb17b6e9b","name":"separate","func":"var payloadList = [];\n\nfor(var i in msg.payload)\n    payloadList.push({payload: msg.payload[i].value});\n\nreturn payloadList;","outputs":12,"noerr":0,"initialize":"","finalize":"","libs":[],"x":560,"y":1020,"wires":[["b85ced26e97c31f2"],["f4a0f4c38c9b0a51"],["6e4ae0c38bfaee5d"],["61e9272afc4f356f"],["2759aa78834a8655"],["fc340f7b00194e56"],["df6c0898bd4e99a1"],["35e58ad59ffe580e"],["66125c6553f64ad1"],["1055ab6c7a8fefd6"],["782612c2fc4ca21f"],["a4e808a7808603b7"]],"outputLabels":["0","1","2","3","4","5","6","7","8","9","10","11"]},{"id":"f0515ed8da983aef","type":"groov-io-read","z":"2ea736ebb17b6e9b","device":"fb014acc80dd41d6","dataType":"module-analog","moduleIndex":"3","channelIndex":"","mmpAddress":"0xF0D81000","mmpType":"int32","mmpLength":"1","mmpEncoding":"ascii","value":"","valueType":"msg.payload","itemName":"","name":"","x":380,"y":1020,"wires":[["6fbf2e97a94b4290"]]},{"id":"b85ced26e97c31f2","type":"rbe","z":"2ea736ebb17b6e9b","name":"","func":"deadbandEq","gap":"0.01","start":"","inout":"out","septopics":true,"property":"payload","topi":"topic","x":800,"y":800,"wires":[["21483370b1e94e71"]]},{"id":"a4e808a7808603b7","type":"rbe","z":"2ea736ebb17b6e9b","name":"","func":"deadbandEq","gap":"0.01","start":"","inout":"out","septopics":true,"property":"payload","topi":"topic","x":800,"y":1240,"wires":[["f57cd2000da79763"]]},{"id":"f4a0f4c38c9b0a51","type":"rbe","z":"2ea736ebb17b6e9b","name":"","func":"deadbandEq","gap":"0.01","start":"","inout":"out","septopics":true,"property":"payload","topi":"topic","x":800,"y":840,"wires":[["cf60db75f576dc55"]]},{"id":"6e4ae0c38bfaee5d","type":"rbe","z":"2ea736ebb17b6e9b","name":"","func":"deadbandEq","gap":"0.01","start":"","inout":"out","septopics":true,"property":"payload","topi":"topic","x":800,"y":880,"wires":[["840048bdbb671dea"]]},{"id":"61e9272afc4f356f","type":"rbe","z":"2ea736ebb17b6e9b","name":"","func":"deadbandEq","gap":"0.01","start":"","inout":"out","septopics":true,"property":"payload","topi":"topic","x":800,"y":920,"wires":[["da5387227def7ebd"]]},{"id":"2759aa78834a8655","type":"rbe","z":"2ea736ebb17b6e9b","name":"","func":"deadbandEq","gap":"0.01","start":"","inout":"out","septopics":true,"property":"payload","topi":"topic","x":800,"y":960,"wires":[["2260140094b7d5c2"]]},{"id":"fc340f7b00194e56","type":"rbe","z":"2ea736ebb17b6e9b","name":"","func":"deadbandEq","gap":"0.01","start":"","inout":"out","septopics":true,"property":"payload","topi":"topic","x":800,"y":1000,"wires":[["ae4ab772e6f46824"]]},{"id":"df6c0898bd4e99a1","type":"rbe","z":"2ea736ebb17b6e9b","name":"","func":"deadbandEq","gap":"0.01","start":"","inout":"out","septopics":true,"property":"payload","topi":"topic","x":800,"y":1040,"wires":[["87e2a9ececd0b6a6"]]},{"id":"35e58ad59ffe580e","type":"rbe","z":"2ea736ebb17b6e9b","name":"","func":"deadbandEq","gap":"0.01","start":"","inout":"out","septopics":true,"property":"payload","topi":"topic","x":800,"y":1080,"wires":[["def439d78051e614"]]},{"id":"66125c6553f64ad1","type":"rbe","z":"2ea736ebb17b6e9b","name":"","func":"deadbandEq","gap":"0.01","start":"","inout":"out","septopics":true,"property":"payload","topi":"topic","x":800,"y":1120,"wires":[["a97422e29804b36f"]]},{"id":"1055ab6c7a8fefd6","type":"rbe","z":"2ea736ebb17b6e9b","name":"","func":"deadbandEq","gap":"0.01","start":"","inout":"out","septopics":true,"property":"payload","topi":"topic","x":800,"y":1160,"wires":[["59bdbd72638dcd3a"]]},{"id":"782612c2fc4ca21f","type":"rbe","z":"2ea736ebb17b6e9b","name":"","func":"deadbandEq","gap":"0.01","start":"","inout":"out","septopics":true,"property":"payload","topi":"topic","x":800,"y":1200,"wires":[["f6d26872c7632d69"]]},{"id":"4d76a6d6f2cee87d","type":"inject","z":"2ea736ebb17b6e9b","name":"","props":[],"repeat":"3","crontab":"","once":true,"onceDelay":"1","topic":"","x":210,"y":1020,"wires":[["f0515ed8da983aef"]]},{"id":"21483370b1e94e71","type":"change","z":"2ea736ebb17b6e9b","name":"","rules":[{"t":"set","p":"payload","pt":"msg","to":"","tot":"str"},{"t":"set","p":"payload","pt":"msg","to":"","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":960,"y":800,"wires":[["0f37d1be202d06e1"]]},{"id":"f57cd2000da79763","type":"change","z":"2ea736ebb17b6e9b","name":"","rules":[{"t":"set","p":"payload","pt":"msg","to":"","tot":"str"},{"t":"set","p":"payload","pt":"msg","to":"","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":960,"y":1240,"wires":[["0c3ee39fc00f7397"]]},{"id":"cf60db75f576dc55","type":"change","z":"2ea736ebb17b6e9b","name":"","rules":[{"t":"set","p":"payload","pt":"msg","to":"","tot":"str"},{"t":"set","p":"payload","pt":"msg","to":"","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":960,"y":840,"wires":[["80c8f784d9bcbd1b"]]},{"id":"840048bdbb671dea","type":"change","z":"2ea736ebb17b6e9b","name":"","rules":[{"t":"set","p":"payload","pt":"msg","to":"","tot":"str"},{"t":"set","p":"payload","pt":"msg","to":"","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":960,"y":880,"wires":[["b36f9923891be168"]]},{"id":"da5387227def7ebd","type":"change","z":"2ea736ebb17b6e9b","name":"","rules":[{"t":"set","p":"payload","pt":"msg","to":"","tot":"str"},{"t":"set","p":"payload","pt":"msg","to":"","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":960,"y":920,"wires":[["b62cade9a958ed3d"]]},{"id":"2260140094b7d5c2","type":"change","z":"2ea736ebb17b6e9b","name":"","rules":[{"t":"set","p":"payload","pt":"msg","to":"","tot":"str"},{"t":"set","p":"payload","pt":"msg","to":"","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":960,"y":960,"wires":[["4b64f8278a05a3ac"]]},{"id":"ae4ab772e6f46824","type":"change","z":"2ea736ebb17b6e9b","name":"","rules":[{"t":"set","p":"payload","pt":"msg","to":"","tot":"str"},{"t":"set","p":"payload","pt":"msg","to":"","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":960,"y":1000,"wires":[["e906f3e8b882db8a"]]},{"id":"87e2a9ececd0b6a6","type":"change","z":"2ea736ebb17b6e9b","name":"","rules":[{"t":"set","p":"payload","pt":"msg","to":"","tot":"str"},{"t":"set","p":"payload","pt":"msg","to":"","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":960,"y":1040,"wires":[["d37ba266d205de39"]]},{"id":"def439d78051e614","type":"change","z":"2ea736ebb17b6e9b","name":"","rules":[{"t":"set","p":"payload","pt":"msg","to":"","tot":"str"},{"t":"set","p":"payload","pt":"msg","to":"","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":960,"y":1080,"wires":[["8083df9537a2bcfa"]]},{"id":"a97422e29804b36f","type":"change","z":"2ea736ebb17b6e9b","name":"","rules":[{"t":"set","p":"payload","pt":"msg","to":"","tot":"str"},{"t":"set","p":"payload","pt":"msg","to":"","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":960,"y":1120,"wires":[["3d31282a8b26ebee"]]},{"id":"59bdbd72638dcd3a","type":"change","z":"2ea736ebb17b6e9b","name":"","rules":[{"t":"set","p":"payload","pt":"msg","to":"","tot":"str"},{"t":"set","p":"payload","pt":"msg","to":"","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":960,"y":1160,"wires":[["f51a172121ad77cf"]]},{"id":"f6d26872c7632d69","type":"change","z":"2ea736ebb17b6e9b","name":"","rules":[{"t":"set","p":"payload","pt":"msg","to":"","tot":"str"},{"t":"set","p":"payload","pt":"msg","to":"","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":960,"y":1200,"wires":[["05858ce63bce1676"]]},{"id":"0f37d1be202d06e1","type":"link out","z":"2ea736ebb17b6e9b","name":"link out 1","mode":"link","links":[],"x":1075,"y":800,"wires":[]},{"id":"0c3ee39fc00f7397","type":"link out","z":"2ea736ebb17b6e9b","name":"link out 12","mode":"link","links":[],"x":1075,"y":1240,"wires":[]},{"id":"80c8f784d9bcbd1b","type":"link out","z":"2ea736ebb17b6e9b","name":"link out 2","mode":"link","links":[],"x":1075,"y":840,"wires":[]},{"id":"b36f9923891be168","type":"link out","z":"2ea736ebb17b6e9b","name":"link out 3","mode":"link","links":[],"x":1075,"y":880,"wires":[]},{"id":"b62cade9a958ed3d","type":"link out","z":"2ea736ebb17b6e9b","name":"link out 4","mode":"link","links":[],"x":1075,"y":920,"wires":[]},{"id":"4b64f8278a05a3ac","type":"link out","z":"2ea736ebb17b6e9b","name":"link out 5","mode":"link","links":[],"x":1075,"y":960,"wires":[]},{"id":"e906f3e8b882db8a","type":"link out","z":"2ea736ebb17b6e9b","name":"link out 6","mode":"link","links":[],"x":1075,"y":1000,"wires":[]},{"id":"d37ba266d205de39","type":"link out","z":"2ea736ebb17b6e9b","name":"link out 7","mode":"link","links":[],"x":1075,"y":1040,"wires":[]},{"id":"8083df9537a2bcfa","type":"link out","z":"2ea736ebb17b6e9b","name":"link out 8","mode":"link","links":[],"x":1075,"y":1080,"wires":[]},{"id":"3d31282a8b26ebee","type":"link out","z":"2ea736ebb17b6e9b","name":"link out 9","mode":"link","links":[],"x":1075,"y":1120,"wires":[]},{"id":"f51a172121ad77cf","type":"link out","z":"2ea736ebb17b6e9b","name":"link out 10","mode":"link","links":[],"x":1075,"y":1160,"wires":[]},{"id":"05858ce63bce1676","type":"link out","z":"2ea736ebb17b6e9b","name":"link out 11","mode":"link","links":[],"x":1075,"y":1200,"wires":[]},{"id":"fb014acc80dd41d6","type":"groov-io-device","address":"localhost","msgQueueFullBehavior":"DROP_OLD"}]

[jacksonlance02]

Thank you both heaps for your help and suggestions - I have implemented some of these changes and the performance has increased a lot - however we are still sitting around 40% at idle. I will keep trying to make changes and optimize performance.

[Beno]

Given the database logging you are doing, Im not sure you going to get it much lower.
I think you have done really well, going from around mid 90’s CPU to 40’s.

By all means keep looking and if you find any tips, please drop back and let us know so we can consider putting them in the ‘Performance Optimization’ doc we are working on.

[jacksonlance02]

Coming back to this…

Noticing this chromium line has been sitting at the top of this screen for a few days now, from my understanding this has something to do with the groov Touch Screen, right?

We do not have any external screen connected via HDMI and we barely use the little touch screen that is on the opto system itself. What might be causing this?

[Beno]

I’m not in front of a system right now to compare.
Do you have the display timeout set? ie, does it blank after x minutes?
I’m not sure that will ‘switch off Chrome’ per say, but it might help. Or do you have a small groov View project running that you need to access on the front screen?

The other thing I will mention that I am going to dig into is that some processes seem to use some CPU when, in reality, the Linux kernel is allowing that process to share ‘nice,’ and so it’s not as scary as it seems.
I need to double check how to read HTOP as per the link I posted a few back up the page and see just how much of an impact this Chome process might be really having.