We have had a few customers ask about using rsyslog on the EPIC, so here is a quick write up to help get you going - or at least point you in the right direction.
Note that with all things SSH, this is unsupported by Opto. There are just too many variables, configurations and options for us to know what exact setup your situation requires.
Remember, with SSH comes great power and great responsibility.
rsyslog is already installed on the PR1. To enable remote logging, you just need to configure it.
So head over to your PuTTY session and log in.
Once at the command prompt, we need to set up the config file. sudo nano /etc/rsyslog.conf to open up the configuration file and set up your remote server.
Scroll down to the section on remote logging. It looks like this;
# Remote Logging (we use TCP for reliable delivery)
# An on-disk queue is created for this action. If the remote host is
# down, messages are spooled to disk and sent when it is up again.
#$WorkDirectory /var/spool/rsyslog # where to place spool files
#$ActionQueueFileName uniqName # unique name prefix for spool files
$ActionQueueMaxDiskSpace 10m # 1gb space limit (use as much as possible)
#$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
#$ActionQueueType LinkedList # run asynchronously
#$ActionResumeRetryCount -1 # infinite retries if host is down
# remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional
#*.* @@remote-host:514
You are going to uncomment just the last line and edit it to match your remote host.
Some tips.
Enter either the IP address:port or fully qualified domain name and port.
You send the data using UDP with a single @ or via TCP with double @@
So for example, to send the logs to an IP address using UDP; *.* @192.168.1.22:514
To send the logs to a hostname using TCP; *.* @@logging.com:514
Once you have made the changes and saved the file, restart rsyslog to reread the config file with the following command; sudo /etc/init.d/syslog restart
(Note that we use the command ‘syslog’ and not ‘rsyslog’).
There are a lot of logs on EPIC that are not compatible with the rsyslog format… so they don’t show up.
The solution is pretty simple (and very powerful).
EPIC uses PM2 to monitor and manage all the services, so if we can get PM2 logs into rsyslog, then we have all the following;
Now restart the rsyslog; sudo /etc/init.d/syslog restart
Next install the PM2 logger;
sudo pm2 install pm2-syslog
That’s it.
Now every time any of those services write to their respective log files, it will be sent to your rsyslog server that you set up in the first post of this thread.
Just to be clear, you need to do the instructions in the first post to enable logging and send the rsyslog to your logger. This will also enable and send the Linux kernel logs.
This second step is needed only if you want all the logs of the Opto 22 services as shown in the screenshot.
This is a slightly different take on same subject.
I have a problem with Modbus Tcp returning a grossly incorrect or IE; a corrupted VFD speed percentage from an Eaton drive. Actually, it’s happening on two Epics on two separate but identical Vfd drives. The values being corrupted or the return of an 03 Mb poll for multiple integers. At first I thought it was maybe variable types on some of the vars, but this had been working for 3+ years before the problem appeared. I also thought it was maybe polling too quick so I modified that to every 50 ms, which is plenty especially since I have a 1 second time delay for every loop. Nope, not that, so then I thought well maybe it is a strategy/fw mismatch and it did turn out there was some hanky panky going on in the Epic which has pretty old fw. So I updated everything to latest, nope, not that either…
At that point, I decided the only way to figure this out was sniffing the packets for Mb only. One problem was that the issue would resolve itself if everything was restarted including the Epic. And it would not show up for a variable number of days, once it did show up, it would continue until everything was rebooted. What I needed was a means to sniff and log to file all packets to and from Epic and VFD and filter out the Mb packets.
My first thought was using Wireshark, but then according to Claude, it would not be able to sniff the packets from the local PC at the remote well site, which I thought it was supposed to be able to do, but that is a network issue. So then I suggested we use Mikrotik (I have a MT switch that the drive and Epic are both connected to). Claude suggested we use MT to mirror the packets to the PC, but that turned out to be an issue because the mirror was causing duplicate packets occasionally, so I asked Claude if Linux had a built in sniffer, and whala…
We (Claude and I…) set up a ring buffered sniffing logger using SSH on the Epic that is set to create a new file every 24 hours with a maximum of 7 files. Works perfect, and I brough the resulting *.cpac file into Wireshark, applied the filter and bingo, I only get Modbus and the tcp ack packets to and from the drive. Here is the command he had me use:
sudo tcpdump -i eth0 host 10.1.1.13 and port 502 -w /home/barrett2/dev/diag/modbus_capture.pcap -C 120 -W 7
In addition we had to set this up with Tmux to keep it running after a logout.
It doesn’t clear the comm buffer before sending your new request, so if a previous request came in after the timeout was reached, you will have data in the buffer that the subroutine will happily digest.
The kit doesn’t validate the response protocol id, trans id or unit id/slave id. So in scenario 1, you can end up recording data from a previous request. This could be a different device in case of RTU or TCP to serial gateway, or different register range if requesting multiple groups of registers from the same device.
The first issue is easy to mostly fix and it will greatly reduce the risk of the 2nd issue. This is by changing the Open block on the kit subroutines from
if (GetNumCharsWaiting(chCommHandle) < 0) then
nStatus = OpenOutgoingCommunication(chCommHandle);
else
nStatus = 0;
endif
to
if (GetNumCharsWaiting(chCommHandle) < 0 or not IsCommunicationOpen(chCommHandle)) then
nStatus = OpenOutgoingCommunication(chCommHandle);
else
ClearCommunicationReceiveBuffer(chCommHandle);
nStatus = 0;
endif
I say mostly, because it is still possible for stale data to come in after the buffer is cleared. The validation issue will need some additional checks added to the Receive block to fix. I don’t have code for that, but did give some recommendations here on validating the Slave Id on RTU:
I guess I should get around to filing a bug report.
Well, I was trying not to open another post…so I figured this was about using SSH to do logging.
You know, when I looked through the Mb sub, that thought did briefly occur to me, why aren’t they clearing the buffer? Although, I didn’t put any thought in past that.
I still haven’t certified the problem, but will tonite. Not entirely sure the problem isn’t the drive, but I wanted to make sure that the request from Epic was accurate because the the packets are clearly responding with bad data values.
@philip I have looked all this over and of course Claude made me a real nice report to send to Eaton…but anyway the evidence in the packet files appear to indicate that the request packets are going out correctly and the returned values are corrupt. The transmit/receive and query/response sequences are correct.
I have to admit, never have had a problem with Opto’s Mb that I could actually point to.
The packet logging feature of the tcpdump command is very handy and also removes any chance of errors caused by switches that aren’t actually valid.
When the network is solid and the timeouts are appropriately set, you won’t see any issues, but it is not as robust as it should be on validating responses - it only check the CRC/LRC on serial, but does no other validation. The digesting of junk in the buffer is one I have experienced on serial networks.
That would make sense that serial is the worst offender, but I agree, it should not answer to values coming back that are not in the correct sequence…and of course, the receive buffer should always be cleared right before a transmit.
I don’t see any value in digesting values out of sync, can only confuse things.
