Best way to detect if communications is lost between Controller and Brains

skyfox · April 16, 2013, 4:00am

Trying to see if there is a tested and fool-proof method of detecting loss of comm’s in a Snap Pac System.

Current System:

R1 Controller
EB-1 Brains ( in 10 different Locations)
HMI

All on the same wired ethernet Network.

Want to Detect:

If R1 to HMI comm is Lost
R1 to each of the 10 EB-1 com is lost
If Coms from HMI to EB-1’s lost.

Ideally I want an output to ENERGIZE (TURN-ON) at each EB-1, if comms are lost at that EB-1’s location (i.e. someone disconnects ethernet cable).

So this would be each EB-1 monitoring it’s own comm status. I want the comms to be lost for > 10 seconds before the output at that EB-1 is energized and do not want to alter the state of any of the other I/O at that EB-1. Simply wants to “Turn On” a dedicated Output point.

Q1. Can I use EB-1’s Comm watchdog feature for this?

Q2. What needs to be done to reset the watchdog and restart-comms at that EB-1 (preferably a method to auto-Start Comm’s)

Q3. Is PAC Display cable of of writing a dynamic value to an R1 Varable automatically say every 5 seconds or so? (i.e. a timer script in PacDisply that can write the current value of a timer in to an R1 Variable?)

Thanks

mstjohn · April 17, 2013, 12:58am

To answer your Q1: Yes, that’s exactly what the watchdog feature is for. In PAC Control, you can turn on the watchdog feature for the I/O Unit, then on a point-by-point basis, configure what they should do when that watchdog occurs.

For Q2: the re-enabling of your I/O Units is something you’ll want to build into your strategy to work exactly the way you want it to work for your situation. Here’s a sample enabler chart to get you started.

On Q3: when you say “timer script in PAC Display,” I think you’re hoping PAC Display is smarter than it is here. PAC Display is great, very powerful, and inexpensive, but it does not do scripts (all your logic usually goes in the strategy). However, I do know we have some customers doing some pretty fancy tricks using PAC Display + AutoHotKey, see this discussion for an example.

Could you share a little more detail on what you’d like to have happen there? I’m guessing you’d like your R1 to have some logic in it that figures out if it’s been disconnected from the world, so it can do something different? (Perhaps you could have your R1 PING the HMI PC, and if no response comes back, execute that logic?) Do tell more!

Thanks,
-OptoMary

skyfox · April 17, 2013, 10:51pm

Thanks OptoMary,

I was not able to download the “[U]Sample Enabler Chart[/U]”. File appears to be missing.

[U][B]On Q3: [/B][/U]
When I said “Timer Script In PacDisplay” it was probably a bit confusing. Let me explain by way of an example what I am hoping to achieve.

In my R1 Controller: I will create a variable Name “[B]int32_CommStatus[/B]”

[U]In PacDisplay Runtime: [/U]
[U][/U]

[SIZE=2]I want a method (script, self resetting HMI based timer etc., etc.) writing a value to my R1 “[B]int32_CommStatus[/B]” variable any time PacDisplay runtime is active.[/SIZE]

[SIZE=2]I want this value to be dynamic (i.e. -32768 to +32767, or 0-60, or something similar that changes every 5 seconds or so). I want the HMI to generate this value automatically and should not depend on any operator interaction to start this process. [/SIZE]

[SIZE=2]Sooooo,[/SIZE]

[SIZE=2]Any time PacDisplay runtime is started, I want this value to be generated from the HMI and written to R1. For example, say the value I select is 0 to 60. HMI should then rollover this value back to 0 when it reaches [/SIZE][SIZE=2]a count of 60, and continue this process until runtime is stopped. [/SIZE]

[SIZE=2]In turn, I will monitor this value via logic in my R1 controller to see if it is static. If value remains static for longer than 10seconds, I will turn-on an output in the local rack.[/SIZE]

Similarly, I want to have a method to do the same on my EB-1 based I/O racks (i.e. Turn on an output on its local rack), when Ethernet communications are lost with R1/HMI etc. Since I cannot execute any localized logic in EB-1’s, I was thinking of using the watchdog feature to accomplish the same.

Regarding I/O module Communications…………

[U]Scenario# 1:[/U] Watchdog feature on EB-1 [B][U]not[/U][/B] enabled.

[SIZE=2][U]Scenario# 2:[/U] Watchdog feature on EB-1 [B][U]ENABLED[/U][/B].[/SIZE]

Ethernet communications are lost between R1 & EB-1 (Someone unplugs the cable).

When the cable gets plugged back in – Does EB-1 automatically establish communications with R-1 controller w/o requiring any further user actions under both scenario’s described above?

Does Watchdog Timeout condition “RESET” automatically when communications are re-established under scenario #2?

Thanks again.

mstjohn · April 17, 2013, 11:20pm

Hi skyfox,

Unfortunately, PAC Display Runtime does NOT have these methods you mention. You COULD easily implement the logic you describe in a strategy running on the same PC as your HMI using, SoftPAC. Or you could use that PING method I mentioned above to check connectivity between the R1 and the PC your HMI is running on.

NOTE: The watchdog feature built in to the I/O is watching for communication to the I/O only, and it doesn’t care where that communication is coming from. There is an option in PAC Display to communicate directly to the I/O (bypassing the controller). It’s in PAC Display configurator under Configure > Runtime then click the “I/O Unit Tags” so make sure you configure that appropriately.

The watchdog is automatically reset, but communication between your controller and your I/O unit is NOT, hence the need for that sample enable logic. Were you missing the inf file? Sometimes that gets stripped from the zip during download. If so, we’ll find another way to get you that file.

-OptoMary

skyfox · April 18, 2013, 8:55pm

Thanks OptoMary,

mstjohn · July 2, 2013, 10:44pm

Watchdog-related idea…

While brainstorming with a coworker yesterday (on how one might, from a PAC, detect if a PAC Display project is running), we came up with this method:

If you don't already have one, set up a simple Historical Data Log in your PAC Display project.
On the same PC, set up an FTP server. FileZilla's server is free and easy to use.
From the PAC, use an ftp Communication Handle to connect to the PC and check that the log file is updating.

Comments? Questions? Better ideas? Do share! -OptoMary

PilotMan · July 3, 2013, 12:19am

Hello,

You may be able to use Pac Display and the built in alarm notification functions such as ALARM, and ALARM CLEAR. Then have control trigger the alarm TAG variable every x seconds causing Display to send an ALARM NOTIFICATION variable back to the strategy and the strategy resets both variables to original state and starts the process over again.

As far as checking if a brains or controller is communicating, taking it a step further here we not only check that but also if the strategy is running using this method based on using the scratchpads:

Our setup consists of 10 R1’s, S1, S2 with one being a dedicated scratchpad data transfer unit receiving all data and then sending it to the other units.

There is one controller status table that is shared by all with each having a desginated table element that is constantally incrumented every time a good handshake occurs and a watchdog timer is also being reset. If there is a loss of com or the strategy is stopped alarms, bells and whistles let everyone know.

Here is the script that would be in each controller(Variables and table names would be different along with s/pad IP). It first checks the data transfer scratchpad then through it checks the others.

//CHECK IF SCRATCH PAD CONTROLLER STRATEGY IS INCRUMENTING THUS RUNNING; IF GOOD RESET TIMER WHICH IS USED TO CHK IF BELOW 30 SECS THEN OK.
//SCRATCH PAD =1 ESD=2 ALK =3 DIST=4 MCC=5 C5000=6 R1301=7 300TF=8 EMEG EVAC=9 CX801=10
STATUS_GET_DATA_XFER_SCRATCH_PAD = GetIoUnitScratchPadINT32TABLE(SCRATCH_PAD_FROM_DATA_XFER_10_10_10_33,25,0,0, CX801_TO_FROM_DATA_XFER_CONTROLLER_STAT);
STATUS_SCRATCH_PAD_IS_ALIVE_CURRENT = CX801_TO_FROM_DATA_XFER_CONTROLLER_STAT[1]; 
IF (STATUS_SCRATCH_PAD_IS_ALIVE_CURRENT &lt;&gt; STATUS_SCRATCH_PAD_IS_ALIVE_PREV) THEN
    STATUS_SCRATCH_PAD_IS_ALIVE_PREV = STATUS_SCRATCH_PAD_IS_ALIVE_CURRENT;
     StartTimer(TIMER_WDOG_SCRATCH_PAD_STRATEGY_RUNNING);
ENDIF
IF (TIMER_WDOG_SCRATCH_PAD_STRATEGY_RUNNING &lt; 30) THEN 
   SETVARIABLETRUE(STATUS_SCRATCH_PAD_STRATEGY_RUNNING);
   ELSE
   SETVARIABLEFALSE(STATUS_SCRATCH_PAD_STRATEGY_RUNNING);
ENDIF
//IF SCRATCH PAD STRAT IS RUNNING THEN CHK ALL OTHER CONTROLLERS IN TABLE SHOULD BE TRUE
//IF NOT RUNNING SET ALL FALSE
IF (ISVARIABLETRUE(STATUS_SCRATCH_PAD_STRATEGY_RUNNING)) THEN
    STATUS_ESD_STRATEGY_RUNNING = CX801_TO_FROM_DATA_XFER_CONTROLLER_STAT[2];
    STATUS_ALK_STRATEGY_RUNNING = CX801_TO_FROM_DATA_XFER_CONTROLLER_STAT[3];
    STATUS_MCC1_RACK1_STRATEGY_RUNNING = CX801_TO_FROM_DATA_XFER_CONTROLLER_STAT[5];
    //SEND OUT IS ALIVE INT
    IncrementVariable(STATUS_TOGGLE_CX801_IS_ALIVE);
    CX801_TO_FROM_DATA_XFER_CONTROLLER_STAT[10]= STATUS_TOGGLE_CX801_IS_ALIVE;
    SetIoUnitScratchPadINT32TABLE(SCRATCH_PAD_ALK_CX801_R1_10_10_10_25, 1,10,10, CX801_TO_FROM_DATA_XFER_CONTROLLER_STAT);
ELSE
    SETVARIABLEFALSE(STATUS_ESD_STRATEGY_RUNNING);
    SETVARIABLEFALSE(STATUS_ALK_STRATEGY_RUNNING);
    SETVARIABLEFALSE(STATUS_MCC1_RACK1_STRATEGY_RUNNING);
ENDIF

Then In the Scratch Pad Data Transfer Controller I check the others and send the data for them to see

//CHECK OTHER CONTROLLERS ARE RUNNING.  IF STRATEGY IS INCRUMENTING THUS RUNNING; IF GOOD RESET TIMER WHICH IS USED TO CHK IF BELOW 30 SECS THEN OK.
//DATA XFER SCRATCH PAD =1 ESD=2 ALK =3 DIST=4 MCC=5 C5000=6 R1301=7 300TF=8 EMEG EVAC=9 CX801=10
STATUS_GET_ESD_SCRATCH_PAD = GetIoUnitScratchPadINT32Table(SCRATCH_PAD_FROM_ESD_RTU1_10_10_10_13,1,2,2, DATA_XFER_TO_FROM_CONTROLLER_STAT);
STATUS_ESD_IS_ALIVE_CURRENT = DATA_XFER_TO_FROM_CONTROLLER_STAT[2]; 
IF (STATUS_ESD_IS_ALIVE_CURRENT &lt;&gt; STATUS_ESD_IS_ALIVE_PREV) THEN
    STATUS_ESD_IS_ALIVE_PREV = STATUS_ESD_IS_ALIVE_CURRENT;
     StartTimer(TIMER_WDOG_ESD_STRATEGY_RUNNING);
ENDIF
IF (TIMER_WDOG_ESD_STRATEGY_RUNNING &gt; 30) THEN 
   DATA_XFER_TO_FROM_CONTROLLER_STAT[2] =0;
ENDIF
STATUS_GET_ALK_SCRATCH_PAD = GetIoUnitScratchPadINT32Table(SCRATCH_PAD_FROM_ALK_MISTIC1_10_10_10_9,1,3,3, DATA_XFER_TO_FROM_CONTROLLER_STAT);
STATUS_ALK_IS_ALIVE_CURRENT = DATA_XFER_TO_FROM_CONTROLLER_STAT[3]; 
IF (STATUS_ALK_IS_ALIVE_CURRENT &lt;&gt; STATUS_ALK_IS_ALIVE_PREV) THEN
    STATUS_ALK_IS_ALIVE_PREV = STATUS_ALK_IS_ALIVE_CURRENT;
     StartTimer(TIMER_WDOG_ALK_STRATEGY_RUNNING);
ENDIF
IF (TIMER_WDOG_ALK_STRATEGY_RUNNING &gt; 30) THEN 
   DATA_XFER_TO_FROM_CONTROLLER_STAT[3] =0;
ENDIF

STATUS_GET_DIST_SCRATCH_PAD = GetIoUnitScratchPadINT32Table(SCRATCH_PAD_FROM_DIST_MISTIC2_10_10_10_11,1,4,4, DATA_XFER_TO_FROM_CONTROLLER_STAT);
STATUS_DIST_IS_ALIVE_CURRENT = DATA_XFER_TO_FROM_CONTROLLER_STAT[4]; 
IF (STATUS_DIST_IS_ALIVE_CURRENT &lt;&gt; STATUS_DIST_IS_ALIVE_PREV) THEN
    STATUS_DIST_IS_ALIVE_PREV = STATUS_DIST_IS_ALIVE_CURRENT;
     StartTimer(TIMER_WDOG_DIST_STRATEGY_RUNNING);
ENDIF
IF (TIMER_WDOG_DIST_STRATEGY_RUNNING &gt; 30) THEN 
   DATA_XFER_TO_FROM_CONTROLLER_STAT[4] =0;
ENDIF
STATUS_GET_MCC1_RK1_SCRATCH_PAD = GetIoUnitScratchPadINT32Table(SCRATCH_PAD_FROM_MCC1_RACK1_10_10_10_28,1,5,5, DATA_XFER_TO_FROM_CONTROLLER_STAT);
STATUS_MCC1_RK1_IS_ALIVE_CURRENT = DATA_XFER_TO_FROM_CONTROLLER_STAT[5]; 
IF (STATUS_MCC1_RK1_IS_ALIVE_CURRENT &lt;&gt; STATUS_MCC1_RK1_IS_ALIVE_PREV) THEN
    STATUS_MCC1_RK1_IS_ALIVE_PREV = STATUS_MCC1_RK1_IS_ALIVE_CURRENT;
     StartTimer(TIMER_WDOG_MCC1_RK1_STRATEGY_RUNNING);
ENDIF
IF (TIMER_WDOG_MCC1_RK1_STRATEGY_RUNNING &gt; 30) THEN 
   DATA_XFER_TO_FROM_CONTROLLER_STAT[5] =0;
ENDIF
STATUS_GET_C5000_SCRATCH_PAD = GetIoUnitScratchPadINT32Table(SCRATCH_PAD_FROM_C5000_10_10_10_21,1,6,6, DATA_XFER_TO_FROM_CONTROLLER_STAT);
STATUS_C5000_IS_ALIVE_CURRENT = DATA_XFER_TO_FROM_CONTROLLER_STAT[6]; 
IF (STATUS_C5000_IS_ALIVE_CURRENT &lt;&gt; STATUS_C5000_IS_ALIVE_PREV) THEN
    STATUS_C5000_IS_ALIVE_PREV = STATUS_C5000_IS_ALIVE_CURRENT;
     StartTimer(TIMER_WDOG_C5000_STRATEGY_RUNNING);
ENDIF
IF (TIMER_WDOG_C5000_STRATEGY_RUNNING &gt; 30) THEN 
   DATA_XFER_TO_FROM_CONTROLLER_STAT[6] =0;
ENDIF
STATUS_GET_R1301_SCRATCH_PAD = GetIoUnitScratchPadINT32Table(SCRATCH_PAD_FROM_R1301_10_10_10_19,1,7,7, DATA_XFER_TO_FROM_CONTROLLER_STAT);
STATUS_R1301_IS_ALIVE_CURRENT = DATA_XFER_TO_FROM_CONTROLLER_STAT[7]; 
IF (STATUS_R1301_IS_ALIVE_CURRENT &lt;&gt; STATUS_R1301_IS_ALIVE_PREV) THEN
    STATUS_R1301_IS_ALIVE_PREV = STATUS_R1301_IS_ALIVE_CURRENT;
     StartTimer(TIMER_WDOG_R1301_STRATEGY_RUNNING);
ENDIF
IF (TIMER_WDOG_R1301_STRATEGY_RUNNING &gt; 30) THEN 
   DATA_XFER_TO_FROM_CONTROLLER_STAT[7] =0;
ENDIF
STATUS_GET_TT_RC_SCRATCH_PAD = GetIoUnitScratchPadINT32Table(SCRATCH_PAD_FROM_TT_RC_10_10_10_7,1,8,8, DATA_XFER_TO_FROM_CONTROLLER_STAT);
STATUS_TT_RC_IS_ALIVE_CURRENT = DATA_XFER_TO_FROM_CONTROLLER_STAT[8]; 
IF (STATUS_TT_RC_IS_ALIVE_CURRENT &lt;&gt; STATUS_TT_RC_IS_ALIVE_PREV) THEN
    STATUS_TT_RC_IS_ALIVE_PREV = STATUS_TT_RC_IS_ALIVE_CURRENT;
     StartTimer(TIMER_WDOG_TT_RC_STRATEGY_RUNNING);
ENDIF
IF (TIMER_WDOG_TT_RC_STRATEGY_RUNNING &gt; 30) THEN 
   DATA_XFER_TO_FROM_CONTROLLER_STAT[8] =0;
ENDIF
STATUS_GET_EMEG_EVAC_SCRATCH_PAD = GetIoUnitScratchPadINT32Table(SCRATCH_PAD_FROM_EMEG_EVAC_R1_10_10_10_23,1,9,9, DATA_XFER_TO_FROM_CONTROLLER_STAT);
STATUS_EMEG_EVAC_IS_ALIVE_CURRENT = DATA_XFER_TO_FROM_CONTROLLER_STAT[9]; 
IF (STATUS_EMEG_EVAC_IS_ALIVE_CURRENT &lt;&gt; STATUS_EMEG_EVAC_IS_ALIVE_PREV) THEN
    STATUS_EMEG_EVAC_IS_ALIVE_PREV = STATUS_EMEG_EVAC_IS_ALIVE_CURRENT;
     StartTimer(TIMER_WDOG_EMEG_EVAC_STRATEGY_RUNNING);
ENDIF
IF (TIMER_WDOG_EMEG_EVAC_STRATEGY_RUNNING &gt; 30) THEN 
   DATA_XFER_TO_FROM_CONTROLLER_STAT[9] =0;
ENDIF
STATUS_GET_CX801_SCRATCH_PAD = GetIoUnitScratchPadINT32Table(SCRATCH_PAD_FROM_ALK_CX801_R1_10_10_10_25,1,10,10, DATA_XFER_TO_FROM_CONTROLLER_STAT);
STATUS_CX801_IS_ALIVE_CURRENT = DATA_XFER_TO_FROM_CONTROLLER_STAT[10]; 
IF (STATUS_CX801_IS_ALIVE_CURRENT &lt;&gt; STATUS_CX801_IS_ALIVE_PREV) THEN
    STATUS_CX801_IS_ALIVE_PREV = STATUS_CX801_IS_ALIVE_CURRENT;
     StartTimer(TIMER_WDOG_CX801_STRATEGY_RUNNING);
ENDIF
IF (TIMER_WDOG_CX801_STRATEGY_RUNNING &gt; 30) THEN 
   DATA_XFER_TO_FROM_CONTROLLER_STAT[10] =0;
ENDIF

/////////////////////////////////////////////////////////SEND OUT COMPILED TABLE OF CONTROLLER STAT
//SEND OUT TO ALL OTHERS ALL CONTROLLER STATUS INCLUDING DATA TRANSFER CONTROLLER IS RUNNING
    IncrementVariable(STATUS_TOGGLE_DATA_XFER_IS_ALIVE);
    DATA_XFER_TO_FROM_CONTROLLER_STAT[1]= STATUS_TOGGLE_DATA_XFER_IS_ALIVE;
    STATUS_SET_DATA_XFER_SCRATCH_PAD = SetIoUnitScratchPadINT32Table(SCRATCH_PAD_DATA_XFER_10_10_10_33, 25,0,0, DATA_XFER_TO_FROM_CONTROLLER_STAT);

A lot of this is repetative for all the controllers we have and you do not need a dedicated controller to function just as a scratch pad data transfer but we are quite large and method reduced our loop times.

mstjohn · July 3, 2013, 2:46am

I love the alarm idea! Thanks for sharing, PilotMan! Just tried it and it seems to work great, details to follow…

BTW, for those of you doing peer-to-peer (as PilotMan does above and as we describe in the manuals using scratchpad areas), be sure to check out This Example, where I provide some peer-to-peer subroutines (at the bottom of the post, not necessarily for use with groov). These subs let you write directly to a variable on another controller, without having to use the scratchpad as middleman.

mstjohn · July 4, 2013, 5:50am

Hey Watchdog Fans,

For those of you wanting to keep track of PAC Display (if it’s still running) from within your PAC strategy, I tried out PilotMan’s suggestion of using the Alarm features in PAC Display. It was nice and simple and worked great! Here’s how I implemented his idea:

I have some looping OptoScript with two int32 variables used with PAC Display, and a string to show the current status.


  // Initialize the value that PAC Display should set to 1
  nPAC_Display_to_Increment = 0; 

  // set the flag which to trigger PAC Display Alarm point 
  // to cause increment on nPAC_Display_to_Increment
  nPAC_Display_flag = 1; 

  DelayMsec(2500); // allow some time for the communication between the PC and PAC 

  if ( 0 == nPAC_Display_to_Increment ) then // nothing happened
    sPAC_Display_Status = "PAC Display not responding";
  else
    sPAC_Display_Status = "PAC Display ok";
  endif

  nPAC_Display_flag = 0; // clear the flag

  DelayMsec(2500); // we'll only check every 5 seconds, so allow the rest of that time to pass

For those who prefer action blocks over OptoScript, here’s a block-based chart.

Here’s the exported chart you can import if you want to save some typing:
PACDisplayMonitor.zip (1.78 KB)

In PAC Display, I selected:

Configure > Alarming Setup > Alert Window [tab]
UNcheck “Enable Alarm Point Alert Window” as shown here:

Configure > Alarm Points > Add.
Give it a name, select the nPAC_Display_flag used in above OptoScript
In: Setup By, choose Discrete
In: Notification Options, Check Alarm Enabled.

Click on the Alarm button, in Current Value choose Set, value out: 1.
Click the ? to choose the value to change (nPAC_Display_to_Increment as shown above). Click OK to close.

Click the Discrete tab, choose True/On.

When I had this all set up and running, watched sPAC_Display_Status changes as I started then stopped PAC Display. Neat-o! Thanks, PilotMan!

-OptoMary

p.s.
Here’s a little video snippet showing what this looked like in action: