Home/Blog/Chasing ghosts
Field notes · Troubleshooting

Chasing Ghosts: Diagnosing Intermittent PLC Faults That Come and Go

By Jonathan Gilmour··8 min read

Come-and-go faults on wet, corrosive plants are rarely in the code. Here's how to corner one instead of resetting it every shift.

You know the one. The line trips at 4 a.m. on a Sunday, drops a batch, and by the time anyone with a laptop reaches the panel the fault has cleared and the HMI is green across the board. No alarm history worth reading, no smoking gun, nothing to grab. Intermittent PLC faults are among the most expensive you'll ever chase, because they bleed you twice: lost throughput when they hit, and engineering hours when you go looking and come back empty-handed.

If you run an organics recycling, composting, or anaerobic digestion site, you get a special edition of the problem. Your control gear lives in one of the harsher environments industrial electronics survive: warm, wet, laden with hydrogen sulfide (H2S) and ammonia, dusty, cycling through big temperature swings daily. That environment doesn't throw random faults — it throws the same handful of faults on a schedule you haven't decoded yet. And on a digester, the knock-on effects are what hurt: a signal dropping out for a moment can stall a batch, upset the biology, or trigger an odor complaint that brings a call from the county.

"Intermittent" usually describes the environment, not the electronics

When a fault comes and goes, it's almost always tracking something you haven't identified yet. Ask what's different at the moment it trips. Time of day is usually ambient temperature; wash-down means moisture; digester load moves vibration and current draw together. A trip that only shows up on cold mornings after a humid night isn't a mystery — it's condensation forming across a terminal strip, bridging something it shouldn't and burning off by mid-morning. You don't have a ghost. You have a dew point.

On these sites the usual suspects cluster in the physical layer. Moisture tops the list: enclosures breathe as they heat and cool, pull in humid air, and drop it as water on the coldest surface inside. H2S and ammonia do the slower work, growing a high-resistance film on terminals and connectors that reads fine on a continuity check and goes marginal under load or vibration. I've opened field boxes where the copper had gone black as a spent match, and the "sensor drift" someone chased in software for a month was a sulfided ferrule adding a few ohms every time the humidity climbed.

Then there's everything mechanical. Vibration from blowers, mixers, and screens finds every termination that went in a quarter-turn shy on install day. Thermal cycling does the rest: brass, steel, and aluminum expand at different rates, so every hot-cold cycle works a joint a little, and over a season the clamp takes a set and loosens — usually reading worst when everything's cold and contracted. Long analog runs are antennas for every VFD on site, and a shield grounded at both ends, or nowhere, or bonded to a floating "ground" gives you 4–20 mA signals that wander and spike just like a failing sensor and aren't.

Physical-first, before you open the code

When a fault won't reproduce, there's a strong temptation to go straight into the logic and start reading ladder. Resist it a while. Most come-and-go faults on wet, corrosive, shaking plants are field problems — wiring, power quality, sensor degradation, connectors — and if you modify code to chase a corroded terminal, you'll "fix" it by coincidence and watch it come back the first cold snap of winter.

So set the baseline physically first. Is the 24 VDC feeding your remote I/O holding up under full load, or sagging on every blower start — an inrush dip deep enough to reset a marginal module or brown out a rack for a scan? Are shields and grounds bonded the way the drawings claim, or the way it actually got wired in a hurry on a Friday? A structured walk of the panel, the power, the grounding, and the field terminations — done methodically and written down — gives you a documented starting point, and often turns up the problem before any logging is needed.

Catching a fault that refuses to reproduce

When the physical survey doesn't hand you the answer, stop trying to catch the fault live and set a trap so it catches itself. Read and timestamp the controller's fault and diagnostic buffers first. Every platform keeps some form of fault log and module diagnostic word, and people ignore them because the display just says "faulted." Line the timestamps up against your process historian and see what else moved in that second. Then instrument the suspects: trend the signals you distrust — the analog that drifts, the discrete that flickers, the remote I/O rack's comms status — sampled fast enough to see the event instead of averaging it away. Where you can, arm an event trigger that snapshots the tags the instant a condition goes true, so you catch the state at the trip, not the tidied-up aftermath. Log the environment alongside it — that correlation is usually where the answer's been hiding.

I'll be straight about the cost. I've watched it go both ways. Sometimes you instrument a signal, wait a week, and the thing simply doesn't fire while you're watching. Meter's running the whole time. That's the honest reality of intermittent I/O fault troubleshooting, and it's exactly why you don't lean on logging alone.

It isn't always the field

After three sections on the wiring, I won't pretend software is witchcraft. Occasionally it really is the controls or comms layer, and those causes hide just as well. A flaky network segment or a marginal fiber run drops packets only when it's hot, or when someone leans on the cable tray. EtherNet/IP connection timeouts, Profibus token drops, a duplicate IP that only conflicts when a maintenance laptop joins the network — all of it presents as random I/O dropouts. On the software side there's scan-time creep, a watchdog that only bites under a rare state combination, or a timer that misbehaves when three conditions line up: maybe one event in eight.

You tell these apart with the same discipline. If the buffers show clean I/O and healthy comms at the moment of the trip and the logic still faulted, then you open the code — this time with evidence instead of a hunch.

One category earns its own sentence. An intermittent trip on a light curtain, a tired E-stop contact block, or a safety relay is sometimes the system working correctly, catching a real, marginal fault you'd much rather it caught than missed. Do not "solve" a nuisance safety trip by desensitizing the safety function. Find out what it's actually seeing.

What this looks like on a real plant

A compost site had a screening line that faulted a handful of times a week, always clearing before anyone arrived, always blamed on "the PLC glitching." Once someone timestamped the fault log against the historian, every event fell within twenty minutes of the morning wash-down. Trending the I/O supply and enclosure humidity made it plain: wash-down drove the humidity up, condensation formed on a corroded terminal strip in a field box downstream of the screening area, and a digital input the line depended on went marginal for a few scans.

There was no single clean root cause to bill as a hero fix. That terminal strip was one of a whole population that had breathed the same sulfide for years, so the corroded suspects got re-terminated, a filtered breather and drain went in so the enclosure stopped trapping wet air, and one analog run got re-routed while the covers were off. Targeted replacement plus a bit of environmental hardening — no dramatic single-part fix to point at, but it held. On a biogas plant, remember that some enclosures sit in classified areas — sealing, venting, or pressurizing one is a hazardous-location question under the NEC as much as a moisture one, not something to fix with a drill and good intentions.

When it's worth bringing in help

The honest version, for whoever signs off on the time: keep chasing it yourself while the cost of looking stays below the cost of the fault. That's often the right call — nobody knows your plant like the people who run it every day. But once you've made two or three trips and come back empty, or the fault is safety-related or sitting on a line you can't afford to lose, a fresh set of eyes usually earns its keep. Someone who chases these across a lot of different plants tends to have met the same ghost before, and the pattern lands faster for it.

Whoever does the chasing, the method is the same: start in the field, not the ladder, log the environment right next to the signal, and let the fault catch itself instead of waiting to catch it live. If you're stuck on one that won't quit, tell me what's tripping — happy to talk it through.