Saving Incremental Changes


#1

Hello everyone,

I’m wondering if there’s a way to write incremental changes to the flash memory on an EB2.

A little background…I’m working on a project using EB2s without a controller, and I’m trying to decide on the best way to back up the state of the controller for restoration in the event of power failure. I know I can save the full configuration to flash, and right now that’s the approach I’m taking.

What I was hoping was that there might be a way to save incremental changes (e.g. if someone changes one PID setpoint) to make the saving process faster. Because all of the basic “logic” of the project is stored as event messages, digital events, alarms, etc. in the configuration, it takes quite a while to save the whole configuration to flash each time. Does anyone have any suggestions on other ways I might go about this that would speed up the process?

Thanks for your help!


#2

Nick? You do a bunch of event/reaction stuff… How do you back it up?

(Tricia, we used to use PAC Manager back in the day at the hospital. Tools -> Import/copy I/O Unit, but not sure if Nick has another way since then. Also, I agree, PAC Manager does not solve your question here, how to restore in the event of a power failure. Also I am not sure of the speed difference between a PAC Man copy to file and a ‘Save to flash’. ).


#3

The PAC Manager feature under Tools > Import/Copy I/O unit (that Ben mentioned he and Nick used) copies the same information that’s stored to flash (either to another identical device, or a file on your PC). Not sure how speeds for doing that vs. a “store to flash” would compare. All the ideas that come to my mind for alternates involve having some kind of “master” PAC that’s keeping track of all the EB2s.

Tricia - is having a PAC in the mix for management an option? How long does the “store to flash” take? I believe that time would get longer as you add more non-zero values into your configuration. Just curious what you mean by “quite a while.”

Also, I’m guessing you’re doing a store to flash on a big list of EB2s? (Essentially writing to them in series?) If so, would writing to all of them in parallel help?


#4

hi
yeah still using file backup at the hospital, except now its automated and only does the backup once per day. any E/R and MMP stuff i had done involved a PAC to keep track of the changes (PID stuff mainly) and then store to flash when a change was detected (as mary eluded to above). not sure how you’d go about doing incremental changes. also you would want to make sure you don’t inadvertently burn out the flash by executing the store command?
tricia, sounds like you have a HMI somewhere, have you thought about SoftPAC and a simple strategy to monitor the changes? and then perform a store or file backup?


#5

Thanks for the replies :slight_smile:

I can automate backups to the computer using the I/O unit copy and I’ll probably do that, but the concern was that, in the event that the computer doesn’t come back up (for whatever reason), there would be no way to “push” the file back to the brain. The same potential issue exists with SoftPAC, though we have considered adding that for other reasons.

Ultimately, I think we’ll strongly recommend a UPS to prevent the problem in the first place, and we’ll run backups to the computer and just set it to restart and push the values to the brain after a power failure. We made it possible to save to flash from the HMI, so the user can create a ‘failsafe’ configuration in case of power loss and failed restoration of the computer. For more critical applications in the future we may add a PAC as another safety layer, but for now we’re trying to keep costs down as much as possible.

Mary, with regard to your other questions…it takes upward of 30 seconds sometimes to save to flash, but we do save in parallel (we’re saving to flash by writing directly to the brain status area on all the brains). I think with the combined strategy mentioned above we’ll make this work because it won’t need to be done frequently (with the added advantage of preserving the brain lifetime).

Thanks again!


#6

I have now taken on this project and I am working a couple of things out:

To save configuration to flash from our HMI, we send a scratchpad bit via OPC. There are two events that respond to this bit: one is a digital event that waits 500 ms and sets the bit back to 0, the other is an event message that writes x00000003 to the status write area (address F038 0000).

I’ve noticed that occasionally the write to flash fails. I usually find this out by power cycling my system and noting the state that the digital outputs come back to. Part of this seems like it may be because I have PAC Manager inspecting my EB2 on two different PCs connected to the EB2. When I close the inspect window on both PCs, the success rate of my flash writing is much better. Does this sound right? I don’t have auto-refresh or anything going on, just sitting on the status read page.

I would really like a way to monitor the result of the flash write from the HMI so the user knows when they’ve got to try again. I’ve tried watching address F030 0248 after a flash write, which the MMP protocol guide says will contain the result of certain status area write commands, including the save configuration to flash command. However, even when I have a failure, I have not seen any change in the value at the address.

Also, we have an interesting setup to save the state of certain digital outputs to flash so on a power cycle, the outputs can return to on or off state as saved to flash by the user. We do this by sending a 32-bit scratchpad integer and a scratchpad bit to trigger a swap from the 32-bit to a 64-bit integer. Then we can use bits of the 64-bit integer to trigger a digital event to either turn on or off our digital outputs after a small delay. I mention this because I’m not sure if there may be an easier way to get outputs to return to a certain state after powerup.

If anyone has any ideas, I’d appreciate it!


#7

SeanT,

There are a few things I want to comment on here… They are all somewhat linked, so I hope you can untangle my thoughts here as we walk through them all…

First up, the writing of the OX03 to mem map address F038 0000 is a bit of a backdoor way to save the unit configuration to flash.
Since you are doing the save via this method, it does not take into account what the strategy is doing when the save to flash gets triggered.
The strategy could be up to anything, and yet, this event message is just going to fire off and do its thing.
Not the cleanest solution in my humble opinion.
What I have done in the past is to do the save configuration to flash from within the strategy… We use the PAC Control command ‘Write I/O Unit Configuration to EEPROM’.
By doing it in the strategy, we have more control over whats going on when we fire the save… For example, we make sure all charts are stopped, this way we know that nothing is writing to the I/O unit when the command is issued.
Also, we can delay a good amount of time after we issue the command. Depending on your rack size (4,8,12, or 16), the number of modules in the rack the number of PID loop’s configured and the number of event reactions you have set up, the command could take a few seconds, the manual states 2 seconds, my chart waited 5 seconds.
Lastly, after that delay was up, we can re-enable the I/O unit (necessary if its an EB or an R).

You mention that you feel this command does not always work because the digital outputs are not always restored…
This command, BOTH the 0x03 or the PAC Control command do NOT save the state of the digital output points.
The commands save the [I]configuration[/I] of the I/O unit, [I]not[/I] the state… The configuration is what modules are plugged into what positions, what PID’s, what event reactions, analog filter weight, scaling and so on.
It does not save digital point or analog point state.

So to answer your question about PAC Manager inspect page being open, no, it does not make any difference. Those pages are only doing a read, if you happen to hit refresh on them while the save to flash was happening, they would time out and nothing more (They time out because the unit is offline while it does the save, hence the need to re-enable communications to it via the strategy or some other method when the command completes).

To answer your question, how to save the digital output state?
You will need to have a chart running (or slip this command into a looping chart that will do this as often as you feel is going to catch the outputs) that uses the command ‘Get I/O Unit as Binary Value’.
Look up the help for this one, you will see that it reads the status of all digital points on a single I/O unit and puts them into a variable, its a bit mask of the state of the points.
(View it in binary to make it clear).
In your case, I would be making sure that the variable you store it to is a persistent.
The last step is then to tweak your powerup chart and either do it there, or call an ‘init’ chart that then sets the digital output points from this bit mask. Once that chart has done its thing, then you can go on and do the rest of the charts in your strategy.
(Note. Depending on how much you care about the analog outputs, you could use the commands ‘Move I/O Unit to Numeric Table Ex’ to get the values, and then ‘Move Numeric Table to I/O Unit Ex’ to set the outputs. Again, make sure that the table is set up as a persistent - If you want to use the table, but don’t want to set the analog outs on power-up, then write a little logic that strips the analog outs of the table before doing the write).

Since your configuration rarely changes once you have done setting up the rack and debugging it, you simply have to the save to EEPROM once and its done.
Since the digital outputs can change as per the strategy, you need to save the rack into a bitmask or table as often as you like (after each write to a digital point if you like).
Unlike the EEPROM command, the persistent memory can not be worn out, so it can be run as often as you like.
Also, unlike the EEPROM commands, the unit is not going to go offline or take a few seconds to do the write when using the persistent method.
If you are going to use this command from the strategy, or call it via the 0x03 method, be sure and set the time out value in the I/O unit config to something more than 1 second (the default). (If you have the default 1 second still in there, this may be a reason you see odd things happening with your current set up. The unit will try and re-enable itself every second 3 times while its saving its config to FLASH).

As an aside, on the topic of saving to EEPROM, the other thing we did is to use PAC Manager to save the I/O unit to a file on the computer. We used the Tools -> Import/Copy I/O Unit dialog.
This way, if anything major happen (lighting strike), we had a back up of that I/O rack config.

Lastly, I get why you are asking for a success/failure status on the save to EEPROM, it would be good to have something, I will put in a customer feature request ticket and see what happens.

So, to sum it all up.
Save the config of the I/O unit to FLASH via PAC Manager once you are done setting it up.
Save the config to your hard drive via PAC Manager so you have a back up of the unit in a second place. (Optional).
Save the state of your digital outputs into a persistent variable or table as often as you want.
When you power up, before doing much else, set the outputs to the value of that persistent variable.

Hope that helps.

Cheers,

Ben.


#8

Hi Ben! Couple of things I probably didn’t mention that change things:

  1. The normal system configuration does not have a controller, just brains. We tried to go [I]completely[/I] controller-free with this product, because the EB2 seemed to have all the functions we needed for basic operation. However, Tricia found that running a dynamic profile that changed and ramped setpoints of PID loops using just events/reactions slowed down the scan times of the brain too much for our liking. So when a customer needs dynamic profiling on their system, we substitute an R2 that runs a profile strategy, but the rest of the event/reaction stuff is exactly the same as the EB2 version. But in 99% of our systems, it’s just going to be EB2s, and so far it’s all working great. Anything more than an EB2 for these seems be overkill, with the rare profiling exception.

  2. The state that the outputs need to be restored to after power loss isn’t really defined by us, it’s by our customers and could vary from time to time well after installation. I sort of explained my setup in backwards order there, we already have the event/reaction stuff configured to work around not being able to save output states to flash. This works well, but I was just wondering if there was something simple I was missing. I guess in my head I sort of linked PID config to digital output state because with a PID configured to come back in auto mode with a particular setpoint, that will control our outputs. Now that I write this down, maybe there’s a way to do this with the points I’m concerned about set as the outputs of PID loops.

So what’s the deal with address F030 0248? From the manual, I would assume that this could be a way to check status of a flash write, but I haven’t seen anything from inspecting this address. And how does PAC Manager get a return on the flash write?

Thanks!
Sean


#9

Hi Sean,

I think that’s a new-ish address that we added when we started to support the microSD card. I was able to get errors at that memory location when I removed my microSD card then tried to write to it. But I suspect it’s not going to help in tracking down whatever you’re seeing here.

Normally folks don’t have a problem writing to the on-board flash. From what I can tell, though, we’re not sure if the “store to flash” failed, or if the steps that happened after that failed somehow (or if what got written wasn’t what you thought)?

Digital events can be a bit mind-bending and difficult to track. I’m wondering:

  1. How consistently are you able to reproduce this incorrect behavior?

  2. Could you narrow this down a bit by limited your digital events/messages here to just what’s required to store your current settings?

  3. This bad behavior happen on more than one EB? Or is it so far specific to just one?

I’m not sure I followed all the steps you were using for getting your outputs to the desired state. I wish we had a more built-in solution for that, but I suspect your method is close to as good as you can get without a controller in the mix. (There might be a tricky way to shave a step or two, but I’m not sure it’d be worth the hassle.)

-OptoMary


#10

I’m pretty confident that the output state events are working correctly. The basic setup is pretty simple - just a digital event that turns the output on if its corresponding bit in a SPI64 is 1 or turns it off if the bit is 0. The convoluted stuff is just because we have to communicate the integer as 32-bit, but can only trigger an event using the 64-bit integers. So we send the 32-bit integer and a scratchpad bit that triggers an event message to swap x00000000 and our 32-bit integer to the 64-bit integer so we can trigger the “output” events.

I will have to test the flashing again today, but from what I remember of last week’s testing, every time the save to flash failed, it failed on both an R2 and EB2 at the same time. When it worked, it worked for both. I was getting pretty suspicious that the PAC Manager windows had something to do with it, because the save to flash failed about 10 times in a row with an inspect window open on two PCs networked to the Opto hardware, then it worked 5 times in a row after closing PAC Manager on both PCs. I’ll gather some more information on this later.