PAC R or EPIC Redundancy - PACDisplay or PACControl

PAC Project
1

In PACDisplay (PACProject as a whole) - instead of setting up the control engine IP address for the Primary and Secondary IP Address for a single Controller, has anyone ever tried to setup the Primary IP to one Controller and the Secondary IP to another Controller (both running the same strategy) ?

The scenario that I am looking at is a redundant IO setup where both controllers would be loaded with the same configuration (same IO). In the event of loss of communication of the primary controller, was thinking that PACDisplay (or PACControl looking at a peer to peer setup) could switch to the secondary controller (like we do on network failure).

Of course both the Primary and Secondary Controllers would have to be on the same subnet:
Primary Controller 192.168.1.1
Secondary Controller 192.168.1.2

I havenā€™t had the chance to test this setup (yet) but was wondering if anyone has ever tried this and/or if it would work (with PACDisplay or PACControl)?

I realize that redundant IO has its own list of issues, at this point just want to know PACDisplay (and PACControl via OPTOMMP device) would be able to work in this fashion.

2

Tried the setup and it works - now the question ā€œis this a good ideaā€?

@Beno is there a way to force PACDisplay between Primary and Secondary Addresses other than Configuration Status under View (in PACDisplay Runtime)?

Would like to have the controller (whichever is active when both are running / good) to swap Primary / Secondary based on command from logic (similar to using Application Manager to initiate a program based on a trigger tag).

3

I went down the rabbit hole. Itā€™s an interesting question, and I am still undecided if itā€™s a good idea or not.

Iā€™m sure you saw this in the PAC Display users guide, page 45:

image
image979Ɨ601 158 KB

Itā€™s the main reason I am undecidedā€¦ That and the answer to your other question:

I looked pretty hard, including reviewing the 52 forum search results for the word ā€˜redundantā€™. We added the MMP address for the ROC kit status, but I could not find a way to switch back and forth between the primary and the secondary.
It will have to be something that the operators are trained to do.

Sorry I donā€™t have stronger words of wisdom on this one Lou.

4

@Beno thanks for your insight.
I also went down the rabbit hole on this one.

From a control perspective (peer to peer interfacing) the approach works (because I can change between primary and secondary networks (or controllers) within logic) but from on operator interface perspective it could could get ugly.

I decided to use an additional controller that will aggregate data, operator interfacing, tuning (adjustments will get pushed to both controllers), etc. when everything is working correctly. In the event of a network or IO rack failure (or loss of this aggregate controller) the individual IO Rack / Controllers ensure local control / operation (selection between IO racks will be handled at the rack level).

1 Like
5

If you think thatā€™s a problem, try doing it with Groovā€¦
I gave up and set one groov to one s1 and the other to the other.

6

Iā€™ve got the same problem with getting groov to support a switchover between primary and secondary. A gadget that could open a new webpage on a tag trigger would be awesome for this. Iā€™m looking at doing a small program running in the background that will monitor a I/O point on each controller thru mmp and open the correct groov webpage if it canā€™t connect.

7

Yeah, let me know if you get this to work. Another way to approach this might be to use a Mikrotik switch, and send a CLI to turn off one port on fail and turn on another.

Another way would be to do failover in the switch, the problem might be in that case that the failover port would have a conflict with the same IP address and the main port, but there might be a way around that too.

Oh and here is another thought. Why not forget using ROK, use the relay timer switchover method, but instead of using the relay timer to power one controller up and the other down, leave both controllers running and have the second S1 constantly reading the IO state and the tags states say once every 5 seconds which could be spread out in separate reads for efficiency. Then have either two small switches which are powered through the timer relay. This way, both are updated, and as soon as the switch over and the second switch boots and the first switch is shut off, the second controller is now online with the same IP.

I guess I am now wondering if the ether ports on the S1 can be independently inhibited programmatically?

If that would be possible, then one could forget the timer relay, do a IO point to IO point connection and then it would be bullet proof and very fast.

8

Barrett, Iā€™ll let you know. I donā€™t have a choice in getting it to work. Iā€™m working on a defense project which has to be able to switch between primary and secondary and they donā€™t budge on requirements. Iā€™m too far down the rabbit hole to change any hardware or software.

9

Yah, same problem I had, engineer wanted web based otherwise I would never have used groov for this, Pac Display is a no brainer for this.

The way I ended up doing it not clean, but works rather clunky. I used two groov servers and pointed each one at a respective controller, then used a controller status connection on secondary groov to indicate that the primary was off line and to use backup server.

10

Just to let you know, I finished the Tunnel upgrade project I was on and fund some interesting info on it.

Before we changed fw we were on 10.0h. At some point I had disabled IO_Enabler because we were having issues at the time.

When we tested it this time on 10.0h, the brains would enable automatically when they went off line (more than three tries) by unplugging the ethernet at brain. Tech support initially claimed this was the correct behavior. However, there were cases for the customer where they would have to manually enable the brain.

When we changed fw for all devices to 10.5g and 9.1e (arbiter), the behavior changed and now the brains would not auto enable. When I reported this to Tech Support, I think they actually investigated it thoroughly this time and reported that the kit was not supposed to auto-enable any brains and that if you wanted this behavior, you must apply the IO_Enabler chart. I did and backed off the initial timing to 30/60/120 seconds instead of 2/60/120 in order that the IO enabler chart did not interfere with the default behavior of 3 tries. This approach worked flawlessly, at least so far, and Iā€™ll repost if that changes.

We spent a fair amount of time on the redundancy system testing it, and I have to say, once we had actually read carefully through the manual, each operation was flawless. This included doing a controller and arbiter fw change while on line.

I think it is a shame they have EOLā€™d this kit because now that they have fixed the problems, it works exactly as indicated in the manual.

1 Like
11

Lou,

There are a few ROK pieces on ebay if youā€™re interested, it makes the switchover very seamless if you handle the variables and the switch blocks correctly.

Also, it does check the backup controller (can be either primary or secondary) to see if it is qualified (controller check out and all brains are connectable (not connected)). If a brain drops while running primary, it will not switch controllers if both controllers indicate bad connection to brains, only if primary does not have path to brains and secondary does.

In addition, the controllers and arbiter can be upgraded with firmware while system is online.

Once more thing is, the controller that failed, is hard booted, after the switchover occurs, then when it comes back is checked and qualified.

The redundant manager is very handy as it give status on system and allows various maintenance tasks as indicated above.

The kit updates any variables you select in each chart separately based on each switch block you place in the strategy. This is done via an ethernet cable looped from secondary ether port to secondary ether port (controller 1 to controller 2).

I suspect that if Pac Display were pointed to two different scanners, each pointed to a different S1 in the ROK system, that it would in fact switchover because the ROK hard boots the backup.

Switching back may or may or may not work. Letā€™s assume you have the address in the arbiter that controls the switchover, if you operate that switch from Pac Display, then switching from backup to primary would create the same condition where the controller you are switching from is rebooted. However, I donā€™t see a need to switch back, since the ROK consider the new primary to be the primary.

Of course in Pac Display, you also have the option of duplicating all the screens (quite easily I might add unlike groovā€¦) and then have Pac Display switch menu bars and open a new screen when the controller status fails on one set of screens versus the otherā€¦