I have recently started to publish messages using MQTT on the Rio. I have noticed on the subscriber nodes that while the messages are being relayed fine, the retain flag is set to false. Reading through some online info, it would be ideal if this flag were set to true. Is this possible on the Rio or EPIC when they are publishing messages via MQTT?
There be dangerous waters ahead…
Retain sounds good on the surface, but you really need to take a pause here… Retain means retain. Usually for pretty much ever. So if you make any change to any topic path, you have the new one and the old one. Do that a few times and you pretty much have something very close to a memory leak of sorts.
Your broker just becomes more and more bloated with retained topics.
Depending on the broker, it can vary how it might (or might not) be possible to flush those old retained topic paths.
So that is the cost. The benefit is just that when a new client subscribes to that topic they get the last stale value vs waiting for a fresh value when it is published.
Bottom line. Node-RED MQTT node has it as an option, at this time, EPIC and RIO don’t.
Coming to MQTT and Sparkplug from other metrics systems, I’m interested in having a way to list all the nodes and devices that are present in a system. Prometheus calls this “service discovery”
If birth messages were retained, that would be a really good start.
The benefit is just that when a new client subscribes to that topic they get the last stale value vs waiting for a fresh value when it is published.
Hmm, the retained birth message will have a stale timestamp to indicate that it’s stale in that case. “Waiting for a fresh value” could be an indefinite wait, right? This is why retain would be useful, there’s nothing forcing devices to publish new values when a passive observer joins the system. The primary Sparkplug application can do that by setting itself OFFLINE and ONLINE again, but that’s not something a passive observer can do.
pretty much have something very close to a memory leak of sorts
I don’t buy this reasoning. 
If a user makes mistakes in their topic namespace, they’re going to start over. Not saying Opto22 is doing anything wrong, but lack of service discovery / inability to locate current devices points to a defect in Sparkplug potentially.
I don’t really think of MQTT as a metric system, but a messaging system.
Throwing out some thoughts:
Retain has issues due to the lack of ability to clean things up in a easy way - who wants to get a bunch of old unused data when they subscribe to #? It would be nice if they can “expire”, but that isn’t part of the standard as far as I know. On Mosquitto, I don’t have a persistence database setup, so restarting Mosquitto clears the cruft.
Ignoring Sparkplug:
If you think of MQTT as strictly a messaging protocol and ignore the storage/retention aspect, then you will come to the conclusion that the application needs a way to request updated data when it first subscribes. I set my clients to subscribe to a command that another subscriber can publish to that tells them to publish all their data. Unfortunately I can’t control 3rd party clients, so those I have to just wait for them to publish. Fortunately those clients so far have been pretty dumb and just publish on an interval.
Sparkplug:
Sparkplug has the primary app that accomplishes the above and gives the ability to know all the nodes that are connected, but, as I think you are pointing out, and as far as I know, it is not suitable for multiple (later) subscribers. I don’t know Sparkplug well enough to know if there is a suitable way to accomplish this,
I just use plain MQTT for now and deal with it at the application level (through Node-Red and Ignition).
I don’t have a persistence database setup, so restarting Mosquitto clears the cruft.
Same – I’m using a tiny MQTT library (GitHub - mochi-mqtt/server: The fully compliant, embeddable high-performance Go MQTT v5 server for IoT, smarthome, and pubsub) without persistent storage, so restarting my broker clears the “problem” of potentially disused topics. Having a preference to retain birth messages in a metric system with passive observers means treating instruments as “real” as opposed to “ephemeral”. “Forgotten topics” are just as likely to be offline devices as they are to be memory leaks.
Without retaining birth information in a Sparkplug-based SCADA system, only the primary Sparkplug application has visibility into the current state of the system since a passive observer can’t necessarily determine state about ephemeral instruments.
FWIW, there’s a split in the cloud-monitoring space about whether instruments should be considered real or ephemeral too. In a Prometheus system where counters are cumulative, instruments are considered real and the concept of “current” staleness is observable in the system. In a Statsd system where counters are “deltas”, instruments are ephemeral meaning the only way to know they exist is to see them being used and staleness can only be defined after an interval of time passes.
In my case, there’s a workaround. The MQTT server lets me add a hook to modify messages on arrival, so I can easily set the retain flag for birth messages upon arrival.
Further investigation reveals that the next version of Sparkplug will address this matter. The next Sparkplug describes an Sparkplug-aware MQTT broker as one that automatically retains birth and death messages. 