Download Large File with HTTP Get Request

Node-RED
1

Hello,
I am trying to download a file from a server to Node Red then save as a file. The file is hosted on a server that responds to a GET request. Currently, the flow downloads part of the file using an HTTP request node and a file node, but not the whole thing. I can download the whole file with Postman, but using Node Red, i only get about 14kb of the 2.5MB file.

[{"id":"9d9d51eb.66525","type":"tab","label":"Download Pertus","disabled":false,"info":""},{"id":"741f50bf.487e7","type":"inject","z":"9d9d51eb.66525","name":"","topic":"DownloadPertu","payload":"","payloadType":"date","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":310,"y":100,"wires":[["6aa4d239.1599ac"]]},{"id":"6aa4d239.1599ac","type":"change","z":"9d9d51eb.66525","name":"","rules":[{"t":"set","p":"timestamp","pt":"msg","to":"payload","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":290,"y":140,"wires":[["26ecf671.880a6a"]]},{"id":"26ecf671.880a6a","type":"template","z":"9d9d51eb.66525","name":"Build msg.url","field":"url","fieldType":"msg","format":"handlebars","syntax":"mustache","template":"http://172.1.1.1:80/peccgi/system/driver/HDMDRV?request=listFiles&group=PERTU","output":"str","x":270,"y":180,"wires":[["d2d5213d.67719"]]},{"id":"d2d5213d.67719","type":"http request","z":"9d9d51eb.66525","name":"","method":"GET","ret":"txt","paytoqs":false,"url":"","tls":"","persist":false,"proxy":"","authType":"","x":270,"y":220,"wires":[["7c00d646.1b6018"]]},{"id":"9f5148d7.327a18","type":"debug","z":"9d9d51eb.66525","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","x":250,"y":580,"wires":[]},{"id":"7c00d646.1b6018","type":"xml","z":"9d9d51eb.66525","name":"","property":"payload","attr":"","chr":"","x":250,"y":260,"wires":[["47b40763.a27678"]]},{"id":"47b40763.a27678","type":"change","z":"9d9d51eb.66525","name":"","rules":[{"t":"set","p":"payload","pt":"msg","to":"payload.hpc.StorageGroup.File","tot":"jsonata"}],"action":"","property":"","from":"","to":"","reg":false,"x":280,"y":300,"wires":[["f291e33.4d1972"]]},{"id":"363f2a70.80ad76","type":"http request","z":"9d9d51eb.66525","name":"","method":"GET","ret":"bin","paytoqs":false,"url":"","tls":"","persist":true,"proxy":"","authType":"","x":270,"y":460,"wires":[["3b0ba9ea.269ee6"]]},{"id":"3b0ba9ea.269ee6","type":"file","z":"9d9d51eb.66525","name":"","filename":"","appendNewline":false,"createDir":true,"overwriteFile":"true","encoding":"binary","x":250,"y":520,"wires":[["9f5148d7.327a18"]]},{"id":"f291e33.4d1972","type":"change","z":"9d9d51eb.66525","name":"","rules":[{"t":"set","p":"filename","pt":"msg","to":"payload[0].$.time","tot":"msg"},{"t":"set","p":"url","pt":"msg","to":"payload[0].$.url","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":280,"y":340,"wires":[["2a337b28.3eb6d4"]]},{"id":"2a337b28.3eb6d4","type":"function","z":"9d9d51eb.66525","name":"","func":"msg.filename = '/home/dev/unsecured/' + msg.filename + '.per';\nreturn msg;","outputs":1,"noerr":0,"x":250,"y":380,"wires":[["363f2a70.80ad76"]]}]
2

I’m not sure what could be preventing Node-RED or the HTTP request node in particular from getting the entire file, but could you try using an exec node with curl -X GET '<url>' – just to see if it has the same problem?

To the best of my knowledge, Node-RED should be able to handle up to 5MB files, so I am wondering whether or not there is a limitation on the HTTP request node itself.

3

So I tried with the curl request and got about 108kb instead of 14, so a little better. When i added --location and --request to the curl command, then it started working. Now I have it working with the HTTP node. I think i just had the url wrong?

4

Thanks for checking that, definitely make sure it’s the exact same URL and headers / authentication that you’re using in Postman.
One other thing that I noticed is that you’re using “Enable connection keep-alive” in the HTTP request node, then the file node is set to over-write rather than append. If, for whatever reason, the server is keeping the connection open and sending the file in multiple parts, only the last part will overwrite the file and be saved. I’m not sure if that’s happening here, but could you put a debug node on the HTTP request node and make sure that 108kb packet is coming in all at once? If you see multiple messages then try switching the file node to append mode and see if that’s any better.