Feature Request: Recover Full Node-RED State from Backup

We would like to request the entire ~/.node-red directory to be included when Node-RED is selected for backup, so flows, node_modules, package.json, and other node state (see node-red-contrib-state) are included in backups.

This assures full Node-RED application backup/recovery, and makes O/S upgrades less time consuming for us and our customers.

Thank you,
-Loren

2 Likes

Yup, this has been on the to-do list for a little while. I am told the software guys are working on it. I should check in and see how its going… (Its not as straight forward as it seems due to the size of some peoples install directories and the backup process file size limits).

1 Like

Is there anyway to specifically note which Modules need to be reloaded when restoring Node Red Flows?

On a restore (for a duplicate system or replacement controller) it would be nice to easily know which Modules need to be loaded (especially when multiple 3rd party modules are available for similar function).

It worse when someone trying to restore in the field with no Internet (or limited) access. Especially since the EPIC / RIO needs to be linked online to update.

We can backup flows but not sure if we can backup specific modules (for later restore).
Is there a way to backup the modules directly?

It’s very difficult to backup individual modules without backing up all modules, because of the complex dependency graph. One Node-RED module usually depends on several different NodeJS modules, and when you have multiple Node-RED modules installed, NPM manages the dependency tree.

Best to back up the entire node_modules directory so the dependency graph is maintained.

You have a few points/questions here Lou, let me take your post from the top…

When you restore the flows (either from an import or from restoring a backup) Node-RED drops down a nice notice telling you exactly which nodes need to be installed.
Its simply a matter of working your down the list installing each node name that it calls out and when the last node is installed, the drop down retracts back up and you are ready to run.

The engineers are working on the issue for your exact second point. When the groov device has no Internet access installing the nodes is ‘impossible’. We have had more than a few customers have this exact issue.

As Loren points out, its very complicated determining what nodes have what dependencies. We have spent more than a few hours working on the problem to help a few urgent customer requests so have some understanding of it.

Enough to say, we are looking forward to an engineered solution that will be sustainable going forward.

Ben and Loren - thanks for the help.
Loren mentioned backing up the “entire node_modules directory” but I’m guessing that this is not something that can be done (and restored) with the EPIC / RIO. At least not without shell access.

Lou, you are correct. You need SSH access to do the backup/restore.
Its been the workaround for the EPIC customers in a jam, but that workaround is not there for RIO.

The issue with installing nodes again (vs. from a backup) is the instability of upgrading nodes vs. the stability of restoring known working dependencies. It adds a level of fragility to groov updates, coupling groov O/S update with Node-RED nodes update. You don’t have an ability to restore Node-RED node versions that were proven to work together.

2 Likes

@Lou_Bertha - You should be able to back up the node_modules directory using the Node-RED exec node. 1) Mount a thumb drive onto the USB port, 2) do a cp -R from ~/.node-red/node_modules to the thumb drive. Or do it in a zip or tar archive if you prefer.

The point is, the Node-RED exec node does have the capability and permission to perform this backup/restore. It’s a bit manual, but it will work until the functionality is a part of the groov backup, and it doesn’t require SSH access.

Really awesome solution here @loren1, but I’m really curious about your restore procedure.
Do you move / unzip the folder back into /home/dev/.node-red while Node-RED is still running? (typically I will only make changes to Node-RED configuration files and folders via shell access, while the NR runtime is disabled)

NodeJS is ok with overwriting node_modules with NodeJS running. My restore process would be:

  1. Copy files back over node_modules
  2. Reboot Node-RED

This works with a populated node_modules as well as a blank node_modules, as would be the case on a fresh O/S install.

Doing this with Node-RED quiescent isn’t an option, because it requires the exec node running within Node-RED :slight_smile:

1 Like

So… do I take it there is a step zero and step three not listed here…
0. First, restore and deploy a small ‘Inject > Exec’ flow that does the main copy, basically step one, then do 2 and…then…
3. Restore your main flow with all your nodes in it.

Or am I missing something?
(Node-RED clearly states ‘Flows not deployed due to missing nodes’ when I try it - so I cant run the exec/copy flow).

Depends on the reason for restore. If restoring from a fresh OS update and groov based Node-RED restore, you’ll have all your nodes in your flows, but some nodes won’t work due to missing dependencies. In that case if you had an Inject > Exec flow, it would be restored in the groov based restore.

Restoring after a fresh O/S install:

  1. Restore from the groov based Node-RED backup. This will restore all flows and credentials, but some nodes won’t work. It’s ok, they’ll come back to life later.
  2. Run the Inject > Exec flow that performs the manual USB based node_modules restore described in this thread.
  3. Restart Node-RED. All should be back to the way it was before the O/S update.
  4. Go back to this thread and ‘Like’ the original post so this is prioritized into the groov based backup/restore in the future :slight_smile:

Loren - Great idea BUT for a casual Node-Red user like myself, I think I’m missing something.

Trying to backup: Like Ben I’m using an Inject >Exec command but getting errors (most likely errors in the details or in my knowledge).

You wouldn’t happen to have a flow to show how this would work for the “node red for idiots” crowd?
I could spend hours trying to figure out but if its relatively straightforward, it would be appreciated.

It does sound like this would be a vector that development could use to backup/restore “all” of node red.

Hi Lou,

Here is our (@torchard mostly) flow that makes the backup.
It takes a moment to run, so click the inject and wait.
Then you will find your node directory all zipped up ready to save in groov Manage.

[{"id":"80a17e31.c03fe","type":"exec","z":"3c21729.4a51d8e","command":"cp -R ~/.node-red/node_modules ~/unsecured && zip -r ~/unsecured/node_modules.zip ~/unsecured/node_modules/ && rm -rf ~/unsecured/node_modules","addpay":false,"append":"","useSpawn":"false","timer":"","oldrc":false,"name":"backup node_modules","x":600,"y":1260,"wires":[["87d0345b.5890c8"],[],[]]},{"id":"42326c9f.9e0bc4","type":"inject","z":"3c21729.4a51d8e","name":"","topic":"","payload":"","payloadType":"date","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":380,"y":1260,"wires":[["80a17e31.c03fe"]]},{"id":"87d0345b.5890c8","type":"debug","z":"3c21729.4a51d8e","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","x":810,"y":1260,"wires":[]}]
3 Likes

If you want to make it a full Node-RED backup, I would include everything under ~/.node-red instead of just the node_modules.

The package.json file has a record of the installed modules, and different Node-RED contribution nodes save state under that ~/.node-red directory.

See: node-red-contrib-state (node) - Node-RED

2 Likes

Fantastic - works as advertised.
A picture (or flow) is better than 1000 words. I’m happy to say I had the right idea (on my attempts) but not the correct syntax.
Now that things are backup up, should be able to use syntax provided to figure out the restoration.

Had a chance to play and was able to copy node_module.zip file to USB and back to unsecured.

Now when I try to unzip (under unsecured) it creates home/dev/unsecured/node_module under the unsecured directory. Yes I could just copy the new file structure over the actual
~/.node-red/node_modules
but would rather do it correctly (if possible).
Command:
unzip -o /home/dev/unsecured/node_modules.zip -d /home/dev/unsecured

Tried other configurations that this is the only setup that worked correctly (except for expanded directory structure (did try to unzip without -d but nothing was restored).

I did copy node_modules.zip to my PC and extracted the node_module directory and zipped / compressed on my PC. Copied back to the EPIC and unzip restored node_module directory with all sub directories.

Any suggestions on the Unzip command within the EPIC?
Also is there any suggestions on the zip command that would create a zip file with only node_module directory and subdirectories ?

@Lou_Bertha to the best of my knowledge of the unzip command, the way you’re going about it, specifying paths with -o and -d, is probably the most straight-forward way to do it without having to make any other changes.


With that said, if you want to avoid the subdirectory / pathing behavior that you’re describing, you’ll need to zip it at a different relative path… Which is a little weird to do with an exec node rather than a traditional command line, but this series of commands are working well for me:

cp -R ~/.node-red/node_modules ~/unsecured && cd ~/unsecured/node_modules && zip -r ~/unsecured/node_modules.zip * && rm -rf ~/unsecured/node_modules

This should combine well with the -d unzip feature to specify a path – just make sure you specify a node_modules destination folder since you don’t want all that content flooding your unsecured file area!

unzip ~/unsecured/node_modules.zip -d /home/dev/unsecured/node_modules

Here’s all that wrapped up in a flow if you’d like to try it out:

[{"id":"caf0d5c4.f06778","type":"exec","z":"e02f10b4.235de","command":"cp -R ~/.node-red/node_modules ~/unsecured && cd ~/unsecured/node_modules && zip -r ~/unsecured/node_modules.zip * && rm -rf ~/unsecured/node_modules","addpay":false,"append":"","useSpawn":"false","timer":"","oldrc":false,"name":"backup node_modules","x":700,"y":1880,"wires":[["3df284ec.cc79cc"],[],[]]},{"id":"810742d9.2c803","type":"inject","z":"e02f10b4.235de","name":"BACKUP","topic":"","payload":"","payloadType":"date","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":500,"y":1880,"wires":[["caf0d5c4.f06778"]]},{"id":"3df284ec.cc79cc","type":"debug","z":"e02f10b4.235de","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","x":890,"y":1880,"wires":[]},{"id":"e3a05ac3.15ac48","type":"exec","z":"e02f10b4.235de","command":"unzip ~/unsecured/node_modules.zip -d /home/dev/unsecured/node_modules && rm ~/unsecured/node_modules.zip","addpay":false,"append":"","useSpawn":"false","timer":"","oldrc":false,"name":"restore node_modules","x":700,"y":1960,"wires":[["5ccfcc65.97a834"],[],[]]},{"id":"5ccfcc65.97a834","type":"debug","z":"e02f10b4.235de","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","x":890,"y":1960,"wires":[]},{"id":"aa2eb6c.8fa1e48","type":"inject","z":"e02f10b4.235de","name":"RESTORE","topic":"","payload":"","payloadType":"date","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":500,"y":1960,"wires":[["e3a05ac3.15ac48"]]}]

@tourchard - thank you very much. Works cleaner than what I had which will make it easier for customer to backup / restore in the field.

The issue we run into is that we have many system that Internet Access is prohibited on the control network and/or components tied to said network (yes even the secondary isolated NIC which I’ve argued in the past). This will make it easier to get a system / plant up and running in the event of a failure, etc.