PAC Control REST API Unable to Access Strings

On one of my Groov EPIC controllers, I’m unable to access string variables or tables through the PAC Control REST API. I can retrieve numeric variables and tables without issue, but requests for strings just seem to hang indefinitely (I’m using POSTMAN to test and debug).

I can retrieve the list of string tables using /pac/device/strategy/tables/strings, but when I add a table name to the URL to retrieve values from a specific table, the request hangs. Similarly, when I try /pac/device/strategy/vars/strings, the request hangs (/pac/device/strategy/vars/floats works without issue).

Any ideas?

I just tested it with Postman and my EPIC, but I’m not seeing the same thing as you. Which firmware version are you running?

Also, have you configured the table(s) you want to read to have public read/write access?

@torchard: I should have mentioned this… this is a new phenomenon. I’ve been able to read these strings and table values for months, but it just stopped working last week in the middle of the week (no power cycle, no code download, etc.). I’m running firmware 3.4.4-b.128. I have other EPIC PR-1 controllers that I’m using the same endpoint on successfully.

I understand using Postman to test, but what are you using for the REST requests when everything is running?

I find it strange that just one endpoint stops working without any other changes… Do you have float or integer tables you can test as well? Just to narrow down if is it a table issue or just the single string table endpoint that is not working.

@torchard: I regularly access these endpoints with a PHP script running on Mac OS. I can access endpoints for integer 32 variables, float variables, integer 32 tables (listing and specific tables), and float tables (listing and specific tables). As I mentioned, I can list the string tables, but not access string variable values or string table values. I can see the values in PAC Display windows and by using debug mode on my controller, but I can’t access the values using the API.

Ok, thanks for the extra details. I’ll run a few more tests on my end to see if I can replicate the behavior, but with the inconsistency you’re seeing I would recommend reaching out to support for this one.

1 Like

@torchard: Thanks for the help. My other step before contacting support was going to be a power cycle, but I haven’t found convenient time to do that yet.

1 Like

@torchard: one update – I disabled timeouts and got the endpoints to work. What takes seconds for floats (/pac/device/strategy/vars/floats ~ 1s, 16.04KB) takes minutes for strings (/pac/device/strategy/vars/strings ~ 6m, 17.52KB). There are significantly more float variables, and the total response size is very similar between the two variable types. Still hoping I see improvement at some point, but it is technically working.

Wow, 6 minutes is… a lot. If you don’t mind sharing, what are the contents of those string variables/tables?

@Jonathan_Fischer: nothing too exciting in the contents. Lots of timestamps. Here’s a sample:

[
	{
		"value": "",
		"name": "s_3PhaseWarning_Timestamp"
	},
	{
		"value": "",
		"name": "s_75HPRuntimeWarning_Timestamp"
	},
	{
		"value": "High Level Float",
		"name": "s_AASumpLevelShutdown_Device"
	},
	{
		"value": "",
		"name": "s_AASumpLevelShutdown_Timestamp"
	},
	{
		"value": "Low Level Float",
		"name": "s_AASumpLevelWarning_Device"
	},
	{
		"value": "",
		"name": "s_AASumpLevelWarning_Timestamp"
	},
	{
		"value": "BNA_NiStrike",
		"name": "s_AccumHCL_Back_PumpName"
	},
	{
		"value": "rft_HCLFeeders_Accum_BNA_NiStrike_Minutes",
		"name": "s_AccumHCL_Back_VariableName"
	},
	{
		"value": "Dept8_A2Acid",
		"name": "s_AccumHCL_Front_PumpName"
	},
	{
		"value": "rft_HCLFeeders_Accum_Dept8_A2Acid_Minutes",
		"name": "s_AccumHCL_Front_VariableName"
	},
	{
		"value": "04/13/2023 10:00:01",
		"name": "s_AccumulateRuntimes_Timestamp"
	},
	{
		"value": "WEEKDAY",
		"name": "s_AirMakeup_CurrentState"
	},
	{
		"value": "",
		"name": "s_AirMakeup_ProbeQualityWarning_Timestamp"
	},
	{
		"value": "Air Makeup",
		"name": "s_AirMakeup_TemperatureWarning_Device"
	},
	{
		"value": "",
		"name": "s_AirMakeup_TemperatureWarning_Timestamp"
	},
	{
		"value": "",
		"name": "s_AirPressureHigh_Auxiliary_Timestamp"
	},
	{
		"value": "",
		"name": "s_AirPressureHigh_Plant_Timestamp"
	},
	{
		"value": "",
		"name": "s_AirPressureLow_Auxiliary_Timestamp"
	},
	{
		"value": "04/12/2023 13:44:59",
		"name": "s_AirPressureLow_Plant_Timestamp"
	},
	{
		"value": "NORMAL",
		"name": "s_AirSystem_State"
	},
	{
		"value": "Bead Blaster #1",
		"name": "s_BeadBlaster1_Description"
	},
	{
		"value": "Bead Blaster #2",
		"name": "s_BeadBlaster2_Description"
	},
	{
		"value": "",
		"name": "s_BeadBlasterNoMotionWarning_Device"
	},
	{
		"value": "",
		"name": "s_BeadBlasterNoMotionWarning_Timestamp"
	},
	{
		"value": "Boiler #1",
		"name": "s_Boiler1_Device"
	},
	{
		"value": "",
		"name": "s_Boiler1_FlameFailure_Timestamp"
	},
	{
		"value": "",
		"name": "s_Boiler1_LowWaterLevel_Timestamp"
	},
	{
		"value": "Boiler #2",
		"name": "s_Boiler2_Device"
	},
	{
		"value": "",
		"name": "s_Boiler2_FlameFailure_Timestamp"
	},
	{
		"value": "",
		"name": "s_Boiler2_LowWaterLevel_Timestamp"
	},
	{
		"value": "04/10/2023 05:19:03",
		"name": "s_BoilerPressure_Warning_Timestamp"
	},
	{
		"value": "",
		"name": "s_BoilersOnDuringWeekend_Timestamp"
	},
	{
		"value": "Boiler water on too long with no boilers",
		"name": "s_BoilerWater_NoBoilers_Description"
	},
	{
		"value": "",
		"name": "s_BoilerWater_NoBoilers_Timestamp"
	},
	{
		"value": "seconds",
		"name": "s_BoilerWater_NoBoilers_Units"
	},
	{
		"value": "Boiler water on too long",
		"name": "s_BoilerWater_OnTooLong_Description"
	},
	{
		"value": "",
		"name": "s_BoilerWater_OnTooLong_Timestamp"
	},
	{
		"value": "seconds",
		"name": "s_BoilerWater_OnTooLong_Units"
	},
	{
		"value": "",
		"name": "s_BoilerWaterFlow_NoBoilers_Description"
	},
	{
		"value": "",
		"name": "s_BoilerWaterFlow_TimeWarning_Description"
	},
	{
		"value": "",
		"name": "s_BoilerWaterFlow_Warning_Description"
	},
	{
		"value": "MonitorDept3Values",
		"name": "s_ChartRuntimeError_Chart"
	},
	{
		"value": "",
		"name": "s_ChartRuntimeError_Timestamp"
	},
	{
		"value": "Daily",
		"name": "s_ChilledTank1_DailyDecrease_Description"
	},
	{
		"value": "",
		"name": "s_ChilledTank1_DailyDecrease_Timestamp"
	},
	{
		"value": "Mezzanine Chilled Water Tank",
		"name": "s_ChilledTank1_Device"
	},
	{
		"value": "Hourly",
		"name": "s_ChilledTank1_HourlyDecrease_Description"
	},
	{
		"value": "",
		"name": "s_ChilledTank1_HourlyDecrease_Timestamp"
	},
	{
		"value": "04/07/2023 02:25:29",
		"name": "s_ChilledTank1_LevelLow_Timestamp"
	},
	{
		"value": "",
		"name": "s_ChilledTank1_TemperatureFault_Timestamp"
	},
	{
		"value": "Department 3 Chilled Water Tank",
		"name": "s_ChilledTank2_Device"
	},
	{
		"value": "",
		"name": "s_ChilledTank2_TemperatureFault_Timestamp"
	},
	{
		"value": "04/13/2023 04:47:48",
		"name": "s_ChillerCompressorFault_Timestamp"
	},
	{
		"value": "",
		"name": "s_ChillerNotNeeded_Timestamp"
	},
	{
		"value": "",
		"name": "s_CompressorModuleFault_Timestamp"
	},
	{
		"value": "-534|Info|(null)|N/A|0|R1_202|07:15:11|04/13/23",
		"name": "s_ControllerMessage_Info"
	},
	{
		"value": "04/13/2023 07:17:30",
		"name": "s_ControllerMessage_Timestamp"
	},
	{
		"value": "Dept5_Lane2Sump",
		"name": "s_ControlSumpPumps_PumpName"
	},
	{
		"value": "hut_Dept5_Lane2Sump_AirPumpRuntimer",
		"name": "s_ControlSumpPumps_VariableName"
	},
	{
		"value": "Lane 1 Sump",
		"name": "s_Dept5_Lane1Sump_Device"
	},
	{
		"value": "",
		"name": "s_Dept5_Lane1SumpFloatWarning_Timestamp"
	},
	{
		"value": "04/10/2023 15:13:00",
		"name": "s_Dept5_Lane1SumpLevelWarning_Timestamp"
	},
	{
		"value": "Lane 2 Sump",
		"name": "s_Dept5_Lane2Sump_Device"
	},
	{
		"value": "04/07/2023 11:17:13",
		"name": "s_Dept5_Lane2SumpFloatWarning_Timestamp"
	},
	{
		"value": "04/13/2023 05:01:06",
		"name": "s_Dept5_Lane2SumpLevelWarning_Timestamp"
	},
	{
		"value": "04/13/2023 07:00:30",
		"name": "s_EN_AccumulateWaterFlow_Timestamp"
	},
	{
		"value": "04/13/2023 10:00:01",
		"name": "s_Energy_PeriodEndingTimestamp"
	},
	{
		"value": "Chiller #1 Ambient Temperature",
		"name": "s_ExhaustFan1_Device"
	},
	{
		"value": "04/12/2023 19:27:01",
		"name": "s_ExhaustFan1_TemperatureFault_Timestamp"
	},
	{
		"value": "",
		"name": "s_Fault_LowOil_Timestamp"
	},
	{
		"value": "04/10/2023 06:42:53",
		"name": "s_Fault_Quincy30HP_Timestamp"
	},
	{
		"value": "",
		"name": "s_Fault_Quincy50HP_Timestamp"
	},
	{
		"value": "",
		"name": "s_Fault_QuincyRecip_Timestamp"
	},
	{
		"value": "BNA Acid Silo",
		"name": "s_HCLFeeders_Back_Device"
	},
	{
		"value": "04/06/2023 07:34:05",
		"name": "s_HCLFeeders_Back_DrumLevelLow_Timestamp"
	},
	{
		"value": "Air Pump (Silo to 50gal Drum) Back",
		"name": "s_HCLFeeders_Back_Pump_SiloTo50Gal_Description"
	},
	{
		"value": "BNA_A2Acid",
		"name": "s_HCLFeeders_Back_PumpName"
	},
	{
		"value": "NORMAL",
		"name": "s_HCLFeeders_Back_State"
	},
	{
		"value": "Dept 5 / Robot Acid Silo",
		"name": "s_HCLFeeders_Front_Device"
	},
	{
		"value": "04/12/2023 14:01:52",
		"name": "s_HCLFeeders_Front_DrumLevelLow_Timestamp"
	},
	{
		"value": "Air Pump (Silo to 50gal Drum) Front",
		"name": "s_HCLFeeders_Front_Pump_SiloTo50Gal_Description"
	},
	{
		"value": "Dept5_A1Acid",
		"name": "s_HCLFeeders_Front_PumpName"
	},
	{
		"value": "NORMAL",
		"name": "s_HCLFeeders_Front_State"
	},
	{
		"value": "BNA A-1 Acid HCL Feed Pump",
		"name": "s_HCLFeeders_Pump_BNA_A1Acid_Device"
	},
	{
		"value": "BNA A-2 Acid HCL Feed Pump",
		"name": "s_HCLFeeders_Pump_BNA_A2Acid_Device"
	},
	{
		"value": "BNA Ni Strike HCL Feed Pump",
		"name": "s_HCLFeeders_Pump_BNA_NiStrike_Device"
	},
	{
		"value": "Dept. 5 A-1 Acid HCL Feed Pump",
		"name": "s_HCLFeeders_Pump_Dept5_A1Acid_Device"
	},
	{
		"value": "Dept. 5 A-2 Acid HCL Feed Pump",
		"name": "s_HCLFeeders_Pump_Dept5_A2Acid_Device"
	},
	{
		"value": "Dept. 5 A-1 Acid HCL Feed Pump",
		"name": "s_HCLFeeders_Pump_Dept8_A1Acid_Device"
	},
	{
		"value": "Dept. 5 A-2 Acid HCL Feed Pump",
		"name": "s_HCLFeeders_Pump_Dept8_A2Acid_Device"
	},
	{
		"value": "b_HCLFeedRuntime_Dept5_A1Acid_ClearWarning",
		"name": "s_HCLFeeders_VariableName"
	},
	{
		"value": "04/06/2023 07:38:09",
		"name": "s_HCLFeedersStateError_Back_Timestamp"
	},
	{
		"value": "04/12/2023 14:03:23",
		"name": "s_HCLFeedersStateError_Front_Timestamp"
	},
	{
		"value": "04/13/2023 07:33:50",
		"name": "s_HCLFeedNeeded_BNA_A1Acid_Timestamp"
	},
	{
		"value": "04/13/2023 08:08:40",
		"name": "s_HCLFeedNeeded_BNA_A2Acid_Timestamp"
	},
	{
		"value": "04/13/2023 08:34:39",
		"name": "s_HCLFeedNeeded_BNA_NiStrike_Timestamp"
	},
	{
		"value": "",
		"name": "s_HCLFeedNeeded_Dept5_A1Acid_Timestamp"
	},
	{
		"value": "",
		"name": "s_HCLFeedNeeded_Dept5_A2Acid_Timestamp"
	},
	{
		"value": "",
		"name": "s_HCLFeedNeeded_Dept8_A1Acid_Timestamp"
	},
	{
		"value": "",
		"name": "s_HCLFeedNeeded_Dept8_A2Acid_Timestamp"
	},
	{
		"value": "",
		"name": "s_HCLFeedRuntime_BNA_A1Acid_Timestamp"
	},
	{
		"value": "",
		"name": "s_HCLFeedRuntime_BNA_A2Acid_Timestamp"
	},
	{
		"value": "",
		"name": "s_HCLFeedRuntime_BNA_NiStrike_Timestamp"
	},
	{
		"value": "",
		"name": "s_HCLFeedRuntime_Dept5_A1Acid_Timestamp"
	},
	{
		"value": "",
		"name": "s_HCLFeedRuntime_Dept5_A2Acid_Timestamp"
	},
	{
		"value": "",
		"name": "s_HCLFeedRuntime_Dept8_A1Acid_Timestamp"
	},
	{
		"value": "",
		"name": "s_HCLFeedRuntime_Dept8_A2Acid_Timestamp"
	},
	{
		"value": "04/13/2023 07:28:19",
		"name": "s_HistorizationWarning_Timestamp"
	},
	{
		"value": "04/13/2023 00:00:00",
		"name": "s_IndustrialWater_LastReset_Timestamp"
	},
	{
		"value": "Industrial water gallons exceeded",
		"name": "s_IndustrialWater_Warning_Description"
	},
	{
		"value": "04/10/2023 02:45:20",
		"name": "s_IndustrialWater_Warning_Timestamp"
	},
	{
		"value": "gallons",
		"name": "s_IndustrialWater_Warning_Units"
	},
	{
		"value": "04/13/2023 07:27:46",
		"name": "s_IOError_Timestamp"
	},
	{
		"value": "",
		"name": "s_kWhWarning_Timestamp"
	},
	{
		"value": "",
		"name": "s_LabRectifier_ElapsedTime"
	},
	{
		"value": "",
		"name": "s_LogData_LogTrigger"
	},
	{
		"value": "",
		"name": "s_LogData_ParseError"
	},
	{
		"value": "12",
		"name": "s_LogData_Value"
	},
	{
		"value": "",
		"name": "s_MonitorAirCompressorPM_Trigger"
	},
	{
		"value": "b_Chiller3Compressor2_TemperatureFault",
		"name": "s_MonitorChillerFaults_VariableName"
	},
	{
		"value": "3",
		"name": "s_MonitorChillerRunTimes_Chiller"
	},
	{
		"value": "Chiller3Compressor2",
		"name": "s_MonitorChillerRunTimes_ChillerCompressor"
	},
	{
		"value": "2",
		"name": "s_MonitorChillerRunTimes_Compressor"
	},
	{
		"value": "04/09/2023 07:00:03",
		"name": "s_MonitorChillerRunTimes_Timestamp"
	},
	{
		"value": "rst_Chiller3Compressor2_WeeklyTimestamps",
		"name": "s_MonitorChillerRunTimes_VariableName"
	},
	{
		"value": "dt_MonitorIO_R1_202",
		"name": "s_MonitorIO_Variable"
	},
	{
		"value": "Dept5_Lane2Sump",
		"name": "s_MonitorSumpPumps_SumpName"
	},
	{
		"value": "b_Dept5_Lane2SumpFloatWarning_ClearWarning",
		"name": "s_MonitorSumpPumps_VariableName"
	},
	{
		"value": "b_Dept5_Lane2SumpLevelWarning_Log",
		"name": "s_MonitorTriggers_LogVariableName"
	},
	{
		"value": "",
		"name": "s_MonitorTriggers_Trigger"
	},
	{
		"value": "s_ChartRuntimeError_Timestamp",
		"name": "s_MonitorTriggers_VariableName"
	},
	{
		"value": "Dept 3 Chiller",
		"name": "s_NeedDept3Chiller_Device"
	},
	{
		"value": "04/07/2023 18:44:38",
		"name": "s_NeedDept3Chiller_Timestamp"
	},
	{
		"value": "Dept 5 Chiller",
		"name": "s_NeedDept5Chiller_Device"
	},
	{
		"value": "04/10/2023 05:04:09",
		"name": "s_NeedDept5Chiller_Timestamp"
	},
	{
		"value": "",
		"name": "s_PMNeeded_Quincy30HP_Timestamp"
	},
	{
		"value": "",
		"name": "s_PMNeeded_Quincy50HP_Timestamp"
	},
	{
		"value": "",
		"name": "s_PMNeeded_Quincy75HP_Timestamp"
	},
	{
		"value": "",
		"name": "s_PMNeeded_QuincyRecip_Timestamp"
	},
	{
		"value": "04/13/2023 00:00:00",
		"name": "s_PotableWater_LastReset_Timestamp"
	},
	{
		"value": "Potable water gallons exceeded",
		"name": "s_PotableWater_Warning_Description"
	},
	{
		"value": "04/10/2023 04:50:21",
		"name": "s_PotableWater_Warning_Timestamp"
	},
	{
		"value": "gallons",
		"name": "s_PotableWater_Warning_Units"
	},
	{
		"value": "04/13/2023 07:00:00",
		"name": "s_PreviousDayEnd"
	},
	{
		"value": "04/13/2023 07:00:01",
		"name": "s_PreviousShiftEnd"
	},
	{
		"value": "04/09/2023 00:00:01",
		"name": "s_PreviousWeekEnd"
	},
	{
		"value": "Quincy 30HP Supplemental Compressor",
		"name": "s_Quincy30HP_Device"
	},
	{
		"value": "Quincy 50HP Primary Compressor",
		"name": "s_Quincy50HP_Device"
	},
	{
		"value": "Quincy 75HP Manual Compressor",
		"name": "s_Quincy75HP_Device"
	},
	{
		"value": "Quincy Reciprocating Auxiliary Compressor",
		"name": "s_QuincyRecip_Device"
	},
	{
		"value": "",
		"name": "s_Recipes_Status"
	},
	{
		"value": "",
		"name": "s_RecipUsageWarning_Timestamp"
	},
	{
		"value": "",
		"name": "s_ROFault_Timestamp"
	},
	{
		"value": "RO Water Tank",
		"name": "s_ROLevelWarning_Device"
	},
	{
		"value": "",
		"name": "s_ROLevelWarning_Timestamp"
	},
	{
		"value": "04/12/2023 12:12:24",
		"name": "s_SaltSprayTemperatureWarning_Timestamp"
	},
	{
		"value": "",
		"name": "s_SodiumSulfite_HighLevel_Timestamp"
	},
	{
		"value": "04/10/2023 06:08:14",
		"name": "s_SupplementalAirNeeded_Timestamp"
	},
	{
		"value": "04/13/2023 06:23:26",
		"name": "s_SynchronizeControllerClock_Timestamp"
	},
	{
		"value": "",
		"name": "s_TestWarnings_Trigger"
	},
	{
		"value": "",
		"name": "s_TestWarnings_VariableName"
	},
	{
		"value": "3 PHASE POWER MAY BE OFF",
		"name": "s_TriggerDescription_3PhaseWarning"
	},
	{
		"value": "75HP SCREW RUNTIME LIMIT EXCEEDED",
		"name": "s_TriggerDescription_75HPRuntimeWarning"
	},
	{
		"value": "A/A SUMP IS TOO HIGH - WATER NEEDS TO BE SHUT OFF",
		"name": "s_TriggerDescription_AASumpLevelShutdown"
	},
	{
		"value": "A/A SUMP IS TOO HIGH - CHECK PUMP FOR OPERATION",
		"name": "s_TriggerDescription_AASumpLevelWarning"
	},
	{
		"value": "A PLANT AIR TEMP PROBE IS NOT WORKING",
		"name": "s_TriggerDescription_AirMakeup_ProbeQualityWarning"
	},
	{
		"value": "PLANT AIR TEMPERATURE IS TOO LOW (WEEKEND)",
		"name": "s_TriggerDescription_AirMakeup_TemperatureWarning"
	},
	{
		"value": "AIR PRESSURE HIGH: AUXILIARY",
		"name": "s_TriggerDescription_AirPressureHigh_Auxiliary"
	},
	{
		"value": "AIR PRESSURE HIGH: PLANT",
		"name": "s_TriggerDescription_AirPressureHigh_Plant"
	},
	{
		"value": "AIR PRESSURE LOW: AUXILIARY",
		"name": "s_TriggerDescription_AirPressureLow_Auxiliary"
	},
	{
		"value": "AIR PRESSURE LOW: PLANT",
		"name": "s_TriggerDescription_AirPressureLow_Plant"
	},
	{
		"value": "BEAD BLASTER OFF - NO MOTION",
		"name": "s_TriggerDescription_BeadBlasterNoMotionWarning"
	},
	{
		"value": "A FLAME FAILURE HAS OCCURED ON BOILER #1",
		"name": "s_TriggerDescription_Boiler1_FlameFailure"
	},
	{
		"value": "BOILER #1 WATER LEVEL IS LOW",
		"name": "s_TriggerDescription_Boiler1_LowWaterLevel"
	},
	{
		"value": "A FLAME FAILURE HAS OCCURED ON BOILER #2",
		"name": "s_TriggerDescription_Boiler2_FlameFailure"
	},
	{
		"value": "BOILER #2 WATER LEVEL IS LOW",
		"name": "s_TriggerDescription_Boiler2_LowWaterLevel"
	},
	{
		"value": "BOILER PRESSURE IS OUTSIDE OPERATIONAL LIMITS",
		"name": "s_TriggerDescription_BoilerPressure_Warning"
	},
	{
		"value": "BOILERS ARE ON DURING THE WEEKEND",
		"name": "s_TriggerDescription_BoilersOnDuringWeekend"
	},
	{
		"value": "BOILER WATER FEED ON WITH NO BOILERS (WEEKEND)",
		"name": "s_TriggerDescription_BoilerWater_NoBoilers"
	},
	{
		"value": "BOILER WATER FEED ON TOO LONG (WEEKEND)",
		"name": "s_TriggerDescription_BoilerWater_OnTooLong"
	},
	{
		"value": "CHART RUNTIME ERROR",
		"name": "s_TriggerDescription_ChartRuntimeError"
	},
	{
		"value": "DAILY ALLOWED LEVEL DECREASE EXCEEDED",
		"name": "s_TriggerDescription_ChilledTank1_DailyDecrease"
	},
	{
		"value": "HOURLY ALLOWED LEVEL DECREASE EXCEEDED",
		"name": "s_TriggerDescription_ChilledTank1_HourlyDecrease"
	},
	{
		"value": "THE CHILLED WATER TANK LEVEL IS LOW",
		"name": "s_TriggerDescription_ChilledTank1_LevelLow"
	},
	{
		"value": "MEZZ CHILLED TANK TEMPERATURE IS HIGH",
		"name": "s_TriggerDescription_ChilledTank1_TemperatureFault"
	},
	{
		"value": "DEPT 3 CHILLED TANK TEMPERATURE IS HIGH",
		"name": "s_TriggerDescription_ChilledTank2_TemperatureFault"
	},
	{
		"value": "A CHILLER COMPRESSOR HAS FAULTED",
		"name": "s_TriggerDescription_ChillerCompressorFault"
	},
	{
		"value": "A CHILLER IS ON, BUT IS NOT NEEDED",
		"name": "s_TriggerDescription_ChillerNotNeeded"
	},
	{
		"value": "A COMPRESSOR MODULE HAS FAULTED",
		"name": "s_TriggerDescription_CompressorModuleFault"
	},
	{
		"value": "DEPT 5 LANE 1 SUMP IS TOO HIGH - SHUT OFF WATER",
		"name": "s_TriggerDescription_Dept5_Lane1SumpFloatWarning"
	},
	{
		"value": "DEPT 5 LANE 1 SUMP IS TOO HIGH - CHECK OPERATION",
		"name": "s_TriggerDescription_Dept5_Lane1SumpLevelWarning"
	},
	{
		"value": "DEPT 5 LANE 2 SUMP IS TOO HIGH - SHUT OFF WATER",
		"name": "s_TriggerDescription_Dept5_Lane2SumpFloatWarning"
	},
	{
		"value": "DEPT 5 LANE 2 SUMP IS TOO HIGH - CHECK OPERATION",
		"name": "s_TriggerDescription_Dept5_Lane2SumpLevelWarning"
	},
	{
		"value": "CHILLER EXHAUST FAN TEMPERATURE IS HIGH",
		"name": "s_TriggerDescription_ExhaustFan1_TemperatureFault"
	},
	{
		"value": "LOW OIL FAULT: QUINCY RECIP",
		"name": "s_TriggerDescription_Fault_LowOil"
	},
	{
		"value": "COMPRESSOR FAULT: QUINCY 30HP",
		"name": "s_TriggerDescription_Fault_Quincy30HP"
	},
	{
		"value": "COMPRESSOR FAULT: QUINCY 50HP",
		"name": "s_TriggerDescription_Fault_Quincy50HP"
	},
	{
		"value": "COMPRESSOR FAULT: QUINCY RECIP",
		"name": "s_TriggerDescription_Fault_QuincyRecip"
	},
	{
		"value": "50 GALLON FEED DRUM LEVEL LOW - BACK",
		"name": "s_TriggerDescription_HCLFeeders_Back_DrumLevelLow"
	},
	{
		"value": "50 GALLON FEED DRUM LEVEL LOW - FRONT",
		"name": "s_TriggerDescription_HCLFeeders_Front_DrumLevelLow"
	},
	{
		"value": "HCL FEEDER STATE & HOIST STATE IS INCOMPATIBLE",
		"name": "s_TriggerDescription_HCLFeedersStateError_Back"
	},
	{
		"value": "HCL FEEDER STATE & HOIST STATE IS INCOMPATIBLE",
		"name": "s_TriggerDescription_HCLFeedersStateError_Front"
	},
	{
		"value": "BNA A-1 ACID HCL FEED NEEDED",
		"name": "s_TriggerDescription_HCLFeedNeeded_BNA_A1Acid"
	},
	{
		"value": "BNA A-2 ACID HCL FEED NEEDED",
		"name": "s_TriggerDescription_HCLFeedNeeded_BNA_A2Acid"
	},
	{
		"value": "BNA NI STRIKE HCL FEED NEEDED",
		"name": "s_TriggerDescription_HCLFeedNeeded_BNA_NiStrike"
	},
	{
		"value": "DEPT5 A-1 ACID HCL FEED NEEDED",
		"name": "s_TriggerDescription_HCLFeedNeeded_Dept5_A1Acid"
	},
	{
		"value": "DEPT5 A-2 ACID HCL FEED NEEDED",
		"name": "s_TriggerDescription_HCLFeedNeeded_Dept5_A2Acid"
	},
	{
		"value": "DEPT8 A-1 ACID HCL FEED NEEDED",
		"name": "s_TriggerDescription_HCLFeedNeeded_Dept8_A1Acid"
	},
	{
		"value": "DEPT8 A-2 ACID HCL FEED NEEDED",
		"name": "s_TriggerDescription_HCLFeedNeeded_Dept8_A2Acid"
	},
	{
		"value": "BNA A-1 ACID HCL FEED RUNTIME",
		"name": "s_TriggerDescription_HCLFeedRuntime_BNA_A1Acid"
	},
	{
		"value": "BNA A-2 ACID HCL FEED RUNTIME",
		"name": "s_TriggerDescription_HCLFeedRuntime_BNA_A2Acid"
	},
	{
		"value": "BNA NI STRIKE HCL FEED RUNTIME",
		"name": "s_TriggerDescription_HCLFeedRuntime_BNA_NiStrike"
	},
	{
		"value": "DEPT5 A-1 ACID HCL FEED RUNTIME",
		"name": "s_TriggerDescription_HCLFeedRuntime_Dept5_A1Acid"
	},
	{
		"value": "DEPT5 A-2 ACID HCL FEED RUNTIME",
		"name": "s_TriggerDescription_HCLFeedRuntime_Dept5_A2Acid"
	},
	{
		"value": "DEPT8 A-1 ACID HCL FEED RUNTIME",
		"name": "s_TriggerDescription_HCLFeedRuntime_Dept8_A1Acid"
	},
	{
		"value": "DEPT8 A-2 ACID HCL FEED RUNTIME",
		"name": "s_TriggerDescription_HCLFeedRuntime_Dept8_A2Acid"
	},
	{
		"value": "HISTORIZATION ERROR",
		"name": "s_TriggerDescription_HistorizationWarning"
	},
	{
		"value": "INDUSTRIAL WATER FEED EXCEEDED GALLONS LIMIT",
		"name": "s_TriggerDescription_IndustrialWater_Warning"
	},
	{
		"value": "I/O ERROR",
		"name": "s_TriggerDescription_IOError"
	},
	{
		"value": "15 MINUTE KWH LIMIT EXCEEDED",
		"name": "s_TriggerDescription_kWhWarning"
	},
	{
		"value": "THE DEPT 3 CHILLER IS OFF BUT SHOULD BE ON",
		"name": "s_TriggerDescription_NeedDept3Chiller"
	},
	{
		"value": "THE DEPT 5 CHILLER IS OFF BUT SHOULD BE ON",
		"name": "s_TriggerDescription_NeedDept5Chiller"
	},
	{
		"value": "POTABLE WATER FEED EXCEEDED GALLONS LIMIT",
		"name": "s_TriggerDescription_PotableWater_Warning"
	},
	{
		"value": "RECIP COMPRESSOR USAGE LIMIT EXCEEDED",
		"name": "s_TriggerDescription_RecipUsageWarning"
	},
	{
		"value": "RO SYSTEM FAULT",
		"name": "s_TriggerDescription_ROFault"
	},
	{
		"value": "RO STORAGE TANK LEVEL OUT OF LIMITS",
		"name": "s_TriggerDescription_ROLevelWarning"
	},
	{
		"value": "SALT SPRAY TEMPERATURE OUT OF LIMITS",
		"name": "s_TriggerDescription_SaltSprayTemperatureWarning"
	},
	{
		"value": "SODIUM SULFITE FEED TANK LEVEL IS HIGH",
		"name": "s_TriggerDescription_SodiumSulfite_HighLevel"
	},
	{
		"value": "SUPPLEMENTAL AIR COMPRESSOR NEEDED",
		"name": "s_TriggerDescription_SupplementalAirNeeded"
	},
	{
		"value": "DEPT 3 NZ ZINC PERCENT IS INCORRECT",
		"name": "s_TriggerDescription_ZincNickelPercentWarning"
	},
	{
		"value": "",
		"name": "s_ZincNickelPercentWarning_Timestamp"
	}
]

Ok, 230 or so variables. I’m looking at the code for it right now and each of those string variables requires a couple of round-trips between the API service and the control engine, so that’s not great.

Few more questions:

  1. Do you have a similar number of string tables?
  2. If you request a single string variable or table does it take a similar amount of time, or does that return quickly?
  3. Do your other EPICs not showing the issue have a similar number of string variables?

My gut feeling right now is that it’s a memory pressure or fragmentation thing. You said it’s been running for awhile and the issue started with no obvious cause, but general memory churn over time could get to a point where the service is spending a lot of time on garbage collection before it can return results.

@Jonathan_Fischer:

I know that PAC control has options to designate variables as public or private, but does that actually factor into this API? I don’t usually touch that box because I haven’t seen it make a difference.

  1. Way fewer string tables than string variables. Probably 40 total string tables.
  2. I haven’t tested retrieving an individual string variable, but requesting single relatively small string table is still quite slow. I tested this morning on a 16 element/20 character wide string table that contains timestamps. That request took approximately 2 minutes.
  3. I have not seen this issue on other EPIC controllers running similar and larger strategies.

This being a memory issue probably makes sense. If necessary, I can implement a policy of regularly restarting the controller, though I would obviously prefer to not have to do that.

I’m not sure on that either.

Do you have shell access to that EPIC? If so, you can restart just the service that handles the PAC REST API with:

sudo /etc/init.d/pac-bridge restart
1 Like

@Jonathan_Fischer: I don’t know that I have a shell license on this controller, but I’ll set that up and give it a shot. That would be a more convenient thing to restart on demand if necessary.

Ran this command. No difference in request performance.

What does PAC display show when viewing s_ControllerMessage_Info? Does it include the substring, “(null)?”

	{
		"value": "-534|Info|(null)|N/A|0|R1_202|07:15:11|04/13/23",
		"name": "s_ControllerMessage_Info"
	},

Do the other string tables you’ve tested with the REST API include s_ControllerMessage_Info?

Does the strategy use any binary strings?

Welcome to the Opto forums Indigo.

This post seems unrelated to the thread, do you want me to move your post to a new thread?

PAC Display is not part of this RESTful call. But to answer your question, Yes, it will show the string exactly as it is since its just a string of characters.

The strings in question are process strings. The one you mention are is the result of a specific ‘get information’ type call, so are not part of the issue.

If you mean do they have any binary payloads, it would not matter, they would still be characters, strings are just characters.
As per page 64 of the PAC Control users guide, strings can only be set to be viewed as ASCII or Hex. (Binary is only an option for variables).

image

1 Like

@Indigo: that variable isn’t used in PAC Display, but I don’t think it would matter. “(null)” is a hardcoded string of characters, not some sort of null character.

2 Likes