PAC Controller in Failsafe Bootloader mode?


#1

We had an issue where a PAC-S1 was “stuck” in Failsafe bootloader mode. The site had experienced some physical damage to the electrical feed from the power company that resulted in a power outage. It is not known if there was any surges before the outage. The controller is on a dedicated 24VDC linear power supply, I think it is 50W.

After power was restored the PAC-S1 came up in Failsafe bootloader mode and would not reset with subsequent power cycling. We were able to hold down Reset button to Restore Factory Defaults.

Anyone else have instances of controllers coming up in failsafe booloader mode after power issues?

Thanks


#2

For others who may have no idea about failsafe bootloader mode, let’s start right at the beginning.

Opto22 controllers and brains usually boot up from an internal firmware program, either as supplied from factory or updated by the user using PAC Manager or similar. If a firmware upgrade fails, if some form of memory corruption occurs or if the controller suffers some hardware failure in the flash memory area, the boot process may fail. But you will probably only discover this the next time you lose controller power.

If this occurs, the user can force the controller into a mode that waits for the network to supply its boot program, usually from a PC running PAC Manager. This is known as failsafe bootloader mode. Check the manual for details of how to do this and prepare PAC Manager accordingly.

My understanding is that the controller should never switch to this mode on its own accord, (Maybe someone else can confirm this?) as in the process, all files stored in the flash including the strategy will be deleted.

Although extremely rare, main causes of flash or RAM memory corruption are electrical spikes, heavy EMF interference from nearby transformers, heavy motor starting equipment, earthing problems or continued power glitches. You would have to be very unlucky to experience these, but like being struck by lightning, they can happen. Other reasons include damaged components, constant writing to flash from application software and even old age!

You mention that the site was experiencing power problems at the time. There is probably some connection there. Also, a “brown out” (brief but not complete power dip) is the about the worst thing that can happen to any electronic equipment.

Just to eliminate them, even more obscure reasons that can lead to flash memory corruption and startup failure.

a) Are you using the FTP file area at all?
b) Are you using the instruction “write to memory map address” in your application?
c) Are you using a microSD card?
d) Are you using the inbuilt serial ports to communicate to other devices or a modem connection?
e) Are you storing the control program in battery backed RAM of saving it to Flash Memory?
f) Do you have known earthing or EMF problems in the area where the controller is installed.

I would also suggest that although it looked like the device was in failsafe bootloader mode, it was really in another state that just looked like it.

In the unlikely event of a controller or brain not starting up correctly, firstly note LED colors, blink speed and count the blink codes before looking for a paperclip or attempting reboot. Usually the controller is trying to tell you something though its LED colours, blink speed and blink codes. The blink code is a number of blinks, followed by a pause before repeating. Check the product guide for more detail on this as with latest hardware revisions up to 13 different conditions can be indicated.

So, what to do now?

  1. From within PAC Terminal File menu, Clear Control Engine RAM and Clear Control Engine Flash. Make note of persistent variables first as these will be lost.
  2. From within PAC Manager Tools Maintenance menu, Clear Flash Files
  3. Reload the controller firmware to replace any previously faulty image, switch off the controller completely and check it restarts correctly
  4. Reload all previously removed files, switch off the controller completely and check it restarts correctly
  5. If the problem repeats itself, its time to get the controller fully checked out by Opto22 or your local representative as you may have a hardware problem.