I needed to log data, then query it from the database, and generate a CSV file. This is all using sqlite.
To make it easier for my user, I generate the csv once a minute, then use http output function to serve the file.
In the beginning, there were no problems, but then after about 1 month, I started notice my CPU usage for node-red is always at 100% or more. Through a lot of troubleshooting and going through each flow, I found that once my table was over 3000 entries using the CSV block, it was using continually using 200-350mb of memory and the CPU was fluctuating between 98% & 114%. (That’s what status screen was showing)
This would very badly affect the whole system. Groov manage would go offline, ping rates would spike, pac control, everything became useless for about 15 seconds. Node-red would need to be restarted just to get back in. (Only had about 10-15 seconds to disable each flow before it was spiking again)
I found fastcsv, it is a dropin block, and it mitigates the the whole issue. I have over 15000 items in the sqlite database, and it takes less than a second to process and generate a csv file.
Just wanted to share and hopefully help someone else who might have or have this issue in the future.
A few error screens like this will show up!
Dan
