Pr1 VPN tunnel to full Ignition gateway

[bensonh]

Chiming in here as I work with Port Redirects over VPN frequently.

Assumptions:
ETH0 IP address = 192.168.1.225 (which has a valid gateway to reach the VPN Server)
TUN0 over ETH0 = 100.22.0.22 (or whatever your EPIC’s IP address is for TUN0, viewable from groov Manage > Network > Status)
ETH1 IP address = 172.168.10.10 (EPIC’s Ethernet NIC on the same network as Ignition Server)
Ignition Server IP address = 172.168.10.100
Ignition Server Port = 8088
Client PC’s VPN IP address once connected to the VPN server = 100.22.0.11 (doesn’t really matter what this address as long as it’s on the same VPN network as the EPIC)

Port Redirect Settings:
Title: Ignition Server
Protocol: tcp/udp
External port: 18088 (more on this later)
External interface: tun0
Redirect address: 172.168.10.100
Redirect port: 8088
Redirect interface: eth1

Now, the reason for the wonky “External port” is that, yes, EPIC has 8088 reserved for internal use. The same holds true for port numbers up to 1024. So, you “make up” a port number you’ll remember. In this example, I just prefaced the Ignition Server’s port number with a “1”.

When the Port Redirects are configured as above, the EPIC will accept traffic on TUN0 at port number 18088 (therefore, 100.22.0.22:18088) and redirect it to the Ignition Server on the ETH1 network (172.168.10.100:8088).

Now, once your PC’s OpenVPN client is connected to the OpenVPN Server, first, type this into the browser URL to test:

https://100.22.0.22

That should result in getting to groov Manage on the EPIC, confirming a connection.

Next, to get your Designer on your client PC to connect to the Ignition Server, in the Launcher program, you’ll configure your connection like this:

Designer Name: Ignition Server
Gateway Address: http://100.22.0.22:18088 (note http, not https here)

Let me know how you go. Happy to provide further assistance. -Benson

[Kevan]

Thank you for the responses.

I was locked out of the forums and couldn’t respond. Beno set me down the right path. I couldn’t ping the PC with the gateway. I’m still working with IT on getting that resolved, but pretty sure the was the primary issue.

[Kevan]

Still having issues with this setup. I can not seem to VPN into the gateway even with redirection set and firewalls turned off.

Not sure what I have wrong here.

[Beno]

Quick sanity check…
Can you ping the Ignition box FROM groov Manage out Eth1?

[Beno]

Also, while you are there, check the port 8080 from the port check tool in groov Manage as well and prove that is working as expected.

(I’m sort of working backwards here, start connecting / proving from the EPIC to the Ignition box and backtrack slowly till it breaks).

[Kevan]

Yes I can ping that address from the pr1 and the port is open.

[Kevan]

I checked if that port was open and I also had the gateway in the Ignition server entered which I did not need. It’s working now, thanks for the assistance!

[Beno]

Fantastic!
Thanks for sticking with it. It’s a great feature; I think you can really move forward now.
Drop back to let us know how it’s going, or ask more questions any time.

[Kevan]

Thank a ton! RDP is working nicely too! will be a great wat to work on this project

[Kevan]

Problem:

I alluded to this issue in another post, but it really belonged here. We are experiencing a weird issue where after a PR1 power cycle, and randomly we sometimes loose VPN connection to our Ignition project.

Our Temporary Solution:

We go into the status page of the network in groove manage and sometimes everything shows connected (eth0, eth1, tun0) but no connection to our Ignition project. Other times we notice in the status page that tun0 is disconnected, but after a few moments (times vary), tun0 will connect.

We then navigate to port redirect in groove manage, change nothing, and press save.

After we do this everything works and we are able to access the Ignition project through the VPN.

Fixes tried so far:

1. We updated the firmware of the PR1 and associated I/O
2. We issued shell commands given by Opto engineers. This ended up breaking the PR1, we factory reset the processor, and we started over with a new set of shell commands given by Opto engineers

Where we are today:

We have not solved this and Opto engineers are looking into the cause and solution. I’m posting here to document this for others, and hopefully find an answer quickly. Our customer is growing restless with us having to save the port redirect settings daily, and sometimes a couple times a day.

Any thought or ideas are very much welcome, and I will keep this updated if we come to a solution.

[Beno]

What VPN server are you using?
How is the VPN configured on the EPIC? (Settings file or manual?)

[Kevan]

We are using OpenVPN and we have the PR1 setup as a host with the config file uploaded to the PR1 from OpenVPN.

[Kevan]

Network diagram for context.

[Beno]

Oh… Uuh… Um… That block at the top… yeah…

Have you googled ‘using VPN over starlink’?

[Kevan]

Just looked at that. It looks like it is possible (obviously because it does work), but not ideal due to the Sat connection. I would think if connection is lost the PR1 should be able to re-establish connection though.

No different than a storm rolling in and loosing an ISP connection for a bit, or a bad cable. Maybe I missed something though. Did you read something else?

[Beno]

I’ve got access to a Starlink terminal.
Not sure when I will get time to set it up and test with EPIC, but am trying to make it a priority.

Here is my high level plan-of-attack…
Use Node-RED, something like this flow to ping / check the VPN:

Groov RIO - Health and Network Status - Example Flow

We have a customer putting a lot of RIO’s into the field and they wanted to get a good bit of information about the each of the RIOs health and well-being back to the central operations center. Most of the system data they wanted is scattered around in different posts here in the forums, so I have just pulled all those threads together here. But they also wanted some network data, like WAN IP address of the cell modem, the RIO’s local IP and more importantly, the VPN IP address of each RIO. Th…

(Just the VPN part of course - no need for most of it)
.
Then use the REST API to toggle the VPN when that flow detects its down.

I plan to use @torchard post here to speed up my development of the REST API call to restart the VPN:

Authenticating Opto API calls with HTTP request

Here are some tips to using Opto API’s like groov Manage, groov View, and PAC Control with the standard HTTP request core node. In this post I’ll go over how to set the apiKey header object and how to enable SSL/TLS in the request node. Note that this doesn’t and shouldn’t replace the groov IO nodes, groov View,or the PAC Control nodes – it just provides a way to use API endpoints that are not exposed through those nodes. For example there is no way to use a node to unmount an installed USB dri…

If restarting the VPN via the REST API does not work, then my plan is to get down and dirty and use Node-RED to restart the EPIC network services as per your post about 2 up from here were you say that ‘change nothing and click save’, this is restarting the network services, but of course, it will be done via Node-RED and only done when it detects that it needs restarting… and yes, I know its a workaround… The main goal for me in setting up the Starlink at Opto is to give access to the engineers and help them track it down and fix it at the Linux level.

[Beno]

Update for @Kevan and those following this interesting thread.

I have the Starlink terminal setup and deliberately put it in a bad location with many obstructions.

I then just ended up using two Ping nodes, one pinging a URL that’s on the Internet and another that is pinging the openVPN IP address of my Opto22 PC.
The Starlink is 100’s of miles from the Opto building, so its pretty good test I think.

The ping times are sent via Starlink using MQTT to a broker and plotted on my groov View project.

Next up, while this is running, I will get the REST API working in Node-RED running on the EPIC and make sure I can force the VPN down and up.

The last bit will be then detecting the VPN not auto-reconnecting and try the API down/up dance.

BTW, Benson has run a RIO (VPN) over Starlink for a week at a time over the years and has never seen the issue you are seeing, so I am really interested in running this test I have set up.

A houseboat vacation with groov RIO

If you have any thoughts on how I can tweak the test, please drop a reply here.
I’ve only got a limited window to use the Starlink (1 week), so I want to beat on it as much as I can.

[Beno]

Update a day and a bit later.

I decided to add a ‘reverse’ ping.
From my PC that’s on the VPN, back to EPIC. That’s the new orange trend you see in this screenshot.

At 2:14 a.m. this morning, I was awake and wondering why I was not seeing any outages, and it occurred to me that perhaps the PING was keeping the connection alive.

@Kevan, does traffic flow over the VPN at all times, or are there long periods of zero traffic?

Also, re-reading the thread in the new light of Starlink and testing things here, I noticed your comment here:

@Kevan

We are experiencing a weird issue where after a PR1 power cycle

So, does the outage only occur on an EPIC power cycle?
I was under the impression that it would drop at any time day or night?
But it seems it ONLY occurs after an EPIC power cycle?

I can start cycling power to my EPIC here today and see if I can get it to fail to reconnect with the VPN.

[philip]

@Beno

At 2:14 a.m. this morning, I was awake and wondering why I was not seeing any outages, and it occurred to me that perhaps the PING was keeping the connection alive.

OpenVPN has a built-in facility to keep the UDP connection open (see ping, ping-restart or keepalive parameters - though the defaults can be modified/disabled). I would be curious what @Kevan’s OpenVPN configuration file looks like. I have many RIOs and EPICs connected over VPN, some over cellular and I don’t see the issue he is having.

Edit: On early firmware there was an issue where OpenVPN would not reconnect. PR1 OpenVPN reconnection - #2 by Beno

An OpenVPN log could shed light on the issue as well.

[Kevan]

Our customers IT department got involved and they will be setting up their own VPN connection. We will then turn port forwarding off on tun0 and turn on port forwarding on eth1.

To answer your questions it is happening randomly and after every power cycle.