Recipes and File Formats

PAC Project
1

Since the problem on the Gen 2 R1 regarding Sdcard file write and reads cropped up preventing me from using the SDCard to store a data file on, I have been trying to come up with a method of writing a file to the PC hard drive that doesn’t involve security problems, and is secure from losing data.
After many possible scenarios considered, I finally figured that using the recipes might be simplest means and most solid means. Yes, I looked at History files in PD, but that would not work for several format reasons. Then looked at transmit string table, but that ends up involving firewall issues.
Turns out recipes could be the simplest means, except for one little quirk. The current version of recipes works really fast and is solid. The problem is it was never intended to be used to transfer data from one controller to another. The problem exists where the templet and file path begin with the controller name, which is “Pac Controller”. So I if you have more than one control engine attached to PD, how does it decide which controller it picks??? Not sure how that works, but it does…so since set up recipe manager to have the same string table uploaded from one controller, then downloaded to another (SoftPac in this case) it does not work. This because the
download manager doesn’t allow to use a separate template file. I tested this by editing the controller name from Pac Controller to SoftPac. Then triggered only the download and whala.
Looks like Ill have to have a script that will edit the rcp file controller name to SoftPac and then trigger the download.
It would be a nice feature to have to transfer large tables, or set of tables from one controller to another, or better yet, have an option for the existing upload recipe create a standard CSV file format on the PC hard drive???

2

You’ve mentioned this twice now, so I am confident that you are working with support on this, have a ticket and it won’t be long till there is an update to fix it?

Regarding peer to peer, specifically to/from SoftPac, I would trust an FTP server running on a PC more than trying to wrangle recipes.
With an FTP server (just as secure as a recipe file?) you can write your CSV file on the controller, send it to the FTP server as and when needed and then have SoftPac read it from its own drive or via a network drive. (ie, the FTP server does not need to be on the same PC as SoftPac).

We have a lot of customers using the FileZilla server and seem very happy.

3

This is what I had working, at least as far as the SDCard drive is concerned, the problem is getting past the security issues when I comes to saving the file to the PC from the R1. Yes I know File Zilla would work to pull the file onto the PC, and that’s what I had all working, then the fw problem showed up. I am hoping they will get it fixed, but I’m not holding my breath, they have a zinger on their hands. Therefore, I felt I had to keep moving forward with another solution.

If you can show me how to get past the security issues of saving and retrieving FTP files to the PC from the R1, I’m all eyes, but I made the decision a while back because of those concerns.

If they can get the Fw fixed soon, I can be done with all this as what I had built worked perfectly. You did still have to use FZ, but not that bad a workaround.

At this point, I am about to try just reading the recipe uploaded file into a string table and then simply rip out all the colons and line 0. Then I can write the table to file on PC.

Yes, Josh is handling the ticket if you’re interested in what the problem is.

4

I made a few edits, please reread the post.

I have been trying to come up with a method of writing a file to the PC hard drive that doesn’t involve security problems, and is secure from losing data.

Nothing is totally tamper-proof, but you could write to read-only media for increased file security (in addition to other storage media).

The common options for read-only media are tape and optical discs, but I’ll talk about the latter. You’ll want the read-only variants (CD-R, DVD-R) and not the read-write variants.

No additional disc writing software should be installed because those software can’t be trusted. On windows you can use imapi to roll your own disc writer and verifier. I haven’t spent much time with modern windows, so maybe there is an included utility for multisession disc writing, avoiding a bit of work in rolling your own. At some point you’ll have to swap the discs. If that is not an option, then what I have written here in this post should not be used.

You should use the multisession capabilities of CD-Rs or DVD-Rs (multisession is not an option on BD). You’ll need to be able to determine and read back the most recent session.

For each session you’ll archive the files you want to store and then take a SHA-2 or SHA-3 sum of the archive (SHA1 is not secure). Again, you should use built in methods of making SHA-2 or SHA-3 sums and not installing a third party tool. Make an archive of the SHA-2 or SHA-3 sum and the original archive, and write that to the disc session.

When restoring the archive, unarchive the wrapper archive and verify the sum of the contained archive. You can load the files directly from the disc. The files will be in memory at that point, which is read-write. Perhaps you can make cryptographic hashes of the files to be transferred and check the hashes from the controller before running them?

Optical discs can lose data, so you must back up to reliable medium in addition. Back up your files to the hard disc drive and a third medium, too. Write down the SHA-2 or SHA-3 sums to your notebook with a unique identifier to later verify integrity of the files. Make sure you back up the data offsite.

Hope that helps.

5

Thanks for the tips…wow, haven’t even thought about cds for a long time, however, that got me thinking that maybe some of these stand alone ethernet based hard drives and or USB flash drives might be usable with ftp…although I assume they have all gone the the way of SSL, but maybe this can be turned off. This is on a closed network, not connected to internet.

Edit
Here’s a thought, use a Mikrotik router like a RB951 and plug a usb in and access via non-secure ftp.

6

Beno,

I’ve decided to try to ftp to a USB drive on a Mikrotik 951(as workaround to R1 issue).
I am trying to modify the existing script to go from “file:r” command as an example used to receive the entire file as a string table in the R1.

In order to do this, I am trying to figure out the commands needed or the procedure needed, ie; can I connect FTP to the remote file then use the receive string table command to receive the file?

OR, do I have to first move the ftp file to local filespace, then read it into the table?

7

I’m wondering why you moved away from SoftPac since you seemed to have it up and running?

Another good option with SoftPac on the PC is that you can use a generic MMP I/O device for the controller and get its I/O and scratch pad. No need for any SNAP controller code to build/write/FTP the file from the SNAP, it can be all done from the strategy in SoftPAC to the local drive… (Then burnt to the read only tape/CD etc).

The router will have an FTP server, it won’t mount the USB on the network as a network drive (I don’t think - and if it does, it will only be visible to the PC that is on that network (ie SOFTPac), so you will need to FTP from either the SNAP or SoftPAC to the router FTP server.
So your strategy comm handle will need an IP and port number.

You might find Mary’s reply Sept 2015 helpful here:

8

Well the softpac only existence was the need to get the file to the PC.

The previous layout was where I had two files on the sdcard0. One is a parameter file the customer supplies, it is a csv file 1000 records long x 350 chars wide. This file provides testing parameters to flow test the meters, up to a 1000 different part numbers. This file needs to be accessible to the customer, therefore, I need it either on the sdcard or the PC. The flow test needs to be able to change test parameters on fly and then update the file on the sdcard.
The other file (same location) needs to be able to be written out to the sdcard on demand. So all the mechanics for making this happen are in place with the sdcard0.
So as a workaround, I figured the next best thing was ftp’ing to a USB drive on the network. I also considered early on ftp’ing to the pc, but that proved problematic due to fw issues ( and potential changes to network security).

Blockquote

The Mikrotik does in fact act just like any non-secure ftp server with username and pw. The UN and PW of the MT is the login for the ftp server. I used FZ to verify, works as expected.

Ok, so the following will give you some idea of what I have been trying to do.

//need to close file handle before using the “SetCommunicationHandleValue” command in case it is open
Mtr_CloseChStatus = CloseCommunication(ch_ParameterFile_Handle);

P_sFileSpec = “ftp:” + P_sIpAddress + “:” + P_sPortNumber + “,” + P_sUser + “,” + P_sPw; // set the IP address, the port number (21 for ftp), user name, and password
SetCommunicationHandleValue(P_sFileSpec, ch_ParameterFile_Handle);

//P_sFileSpec = “file:r”;
//SetCommunicationHandleValue(P_sFileSpec, ch_ParameterFile_Handle); // set handle read only

P_Result = OpenOutgoingCommunication(ch_ParameterFile_Handle); // open the handle

if(IsCommunicationOpen(ch_ParameterFile_Handle))then
P_sFileSpec = “dest:” + P_sFilePath + P_sFilename + “,” + P_sFilename; //set the file path and name
P_Result = SendCommunicationHandleCommand(ch_ParameterFile_Handle, P_sFileSpec);

//Sets the communication timeout value for receiving data, which you must specify in seconds. For example, use 0.5 to set the timeout to 1/2 second.
P_SendChStatus = SendCommunicationHandleCommand(ch_ParameterFile_Handle, "set.to:2");

SetEndOfMessageTerminator(ch_ParameterFile_Handle, 10);

endif

So to reiterate, do I need to move the file to the R1 filespace via FTP, then read it the file via Receive string table, OR can I connect to file on USB via ftp and then directly use Receive string table?

9

Btw, thanks for that bit from Mary…she’s right, network versus windows…

10

With respect to the reason for not using scratchpad…I did successfully set up and proved the concept of moving each separate line to SP via 7 elements of strings, but seemed like a problem in the making, I like to move one string table to another, and with 800 chars wide, that req 7 elements of scratch pad for one record.

Also, the other file would not work out anyway. I needed the same solution for both and need to read and write to the parameter file at will, so no the scratch at 128 char wide, wasn’t going to work.

To be honest, I wasn’t made aware of all these requirements upfront, otherwise I would have used an Epic in addition to the R1 (remember R1 was necessary because of the Profibus card of which I had no control of deciding).

11

Well I think I answered my own question that is, the ftp functions really have nothing to do with operating R1 commands for receiving string tables, etc…therefore as I have tested, the ftp commands work as expected, and I simply have to first “get” the file and put it on the filespace, then do the file:r and then use the receive string table command in a different ch.

In the docs it seemed to suggest that I could do both in one connection without moving the file first.

1 Like
12

As I had mentioned previously, I went from the sdcard0 method of uploading and downloading the parameter and data files, to using recipes to do same. I guess I should have listened to that comment you made regarding recipes…
Turns out that the upload version of two files was working, I would trigger the upload (R1 to PC) and it would in turn trigger an application launch copy.exe comm.exe and copy the received recipe file to another folder and rename it at same time the application launch when finished triggered a script in softpac. This would load the file into the softpac string table and then unformat (remove the recipe formatting) and then write it to same filename to overwrite it.
Now all this did in fact work including 1 data file and 1 parameter file.
Then I got to work on the download of the parameter file. Here I loaded the actual parameter file into the softpac string table, and then reformatted it as required by the recipe file function. This was starting to work until I got it pretty close to the finished version, then Pac Display crashed and basically wiped out the HMI. Ok, so not to be deterred even by that, I started again with a previous version of Pac Display strategy (after having reinstalled PacProjectPro and sending the whole thing into the dev) and got back to where the download was previously, and found that the download recipe was crashing Pac Display runtime. It would start to run the recipe, then crash runtime.
Ok, so I am fairly persistent…and not to be deterred…in the meantime (since Dec 23, when I threw up my hands and left the customer with a solution that works on a temp basis) the developer on the R1 Gen2 said he had a beta solution for me. I started back on that by getting a branch of scripting that had been working perfect, transplanted it into the current Pac Control strategy, and ran it… Now, yes he “might” have fixed it, but now the process of loading a 200k file into a string table has gone from 10 seconds (totally acceptable) to 24.5 minutes…

13

Sounds about right if you are transferring the recipe to the controller over a 1200 baud modem. Hope you get it worked out - sounds frustrating.

14

Yah, that’s about right…well you know the reason that I hadn’t gone back to this until now, was that I knew it wouldn’t be fixed completely…and I hate it when I’m right.

15

What’s your tolerance limit for string table loading time? How much work are you willing to put into optimizing a solution that may ultimately be too slow?

I found a 25 SLOC lz77 decompress algorithm on the internet that you could try porting to opto script. There’s also a compress algorithm. Note that I did not write or test the lz77 compress and decompress algorithms, and there may be bugs unknown to me; this is someone else’s code I found via google.

16

Thanks for all that, and I assume your eyes glazed over before you finished the read, but the previous load time was 10 seconds or less. I can handle that, the operators can too, but if it runs into minutes it’s out of the question.
No, there is no way this can take that long, it is a serious fw bug that the developer needs to fix. I’m sure he will, however, as with most things like this, it never gets fixed in one try…

17

Philip I noticed a little problem with Pac Display. I have a downtimer that I start at beginning of this download branch. I Pac Display, I have a PB that starts the branch and then displays in yellow while also displaying some text and the downtimer value. The text and downtimer values are separate and the text is dynamic “Wait…:” for bit on & “Ready” for bit off.
The strange thing is that while this string table loading (not recipe) is occuring, the timer does not display at all, even though the timer is running.
Now i realize the download string table is halting all activity in this chart, however, the timer is running and I can read this from debug, so does that mean that each chart has more than one task thread associated with it? I guess in this case the Host task is reading the timer for debug mode but the thread for this chart is busy servicing the receive string table command and is not doing anything else…

Oh, btw the command is sooo busy, it doesn’t have time to allow the host task to stop the strategy. The only thing that stopped the command is removing the sdcard.

One more edit…

Nope, that did not work, moving the timer to different chart.

18

My bad, forgot to check the PD tag var, apparently at some point it got changed during the all the frantic workarounds…

1 Like
19

Looks like as I suspected, the super slow ReceiveStringTable command when receiving a file from the sdcard, is directly linked to the sdcard firmware issue.
Latest word is, they see what the problem is, and now they are trying to figure out a solution.

I like it when they see what the problem is, good news as far as I am concerned.

Fortunately, customer is running fine and doesn’t need to download that file very often, but the flow prover is running perfect otherwise.

Customer is elated with the results. Most accurate results and fastest prover they have ever had, and they have built from scratch all the other units they currently have. Also, most flexible operating screens as well.

2 Likes