You know the one. A remote I/O rack faults, a conveyor trips, an alarm scrolls past, and by the time an operator walks out to look, everything is running like nothing happened. The fault log says "module not responding" or "connection lost," and thirty seconds later, "connection restored." Maintenance replaces a sensor. It happens again the next week. Somebody swaps the I/O card. It happens again. Eventually the trip gets bypassed or the alarm gets acknowledged reflexively, and now you have a plant running on a known-bad system that everyone has quietly agreed to stop trusting.
The reason these are so miserable to chase is that the evidence deletes itself. A loose field wire you can wiggle. A failed proximity switch you can meter out. But a network fault that lasts 400 milliseconds and then clears leaves nothing behind except a timestamp and a vague controller message, and those messages are written from the controller's point of view, which is the worst possible vantage point for understanding what actually happened on the wire.
Why the controller's error message lies to you
When a programmable logic controller (PLC) reports a lost connection to a remote rack, it is telling you the symptom it experienced, not the cause. On EtherNet/IP, the controller opens a connection with a requested packet interval (RPI) and watches for those packets to keep arriving. Miss roughly four in a row (the default timeout multiplier is four times the RPI) and the connection times out. The controller faithfully logs "connection timeout." What it cannot tell you is why the packets stopped: a switch dropping frames under load, a duplicate address on the subnet, a cable with a marginal connection that only opens up when the tray warms up in the afternoon, or a segment swamped by multicast traffic because IGMP snooping was never set up and the switch is flooding cyclic I/O out every port as if it were broadcast.
So you get a precise, authoritative-sounding message that points at the wrong layer. The usual maintenance response is to go straight to the field device the message named, because that is the only thing the message named, and the field device is fine. The fault lives in the fabric between the controller and that device, and that fabric is invisible unless you go looking for it with the right tools.
The usual suspects, and how they hide
Duplicate Internet Protocol (IP) addresses are the classic. Someone commissions a new drive or a spare panel and reuses an address off a spreadsheet that is three revisions out of date. Most modern EtherNet/IP devices run Address Conflict Detection (ACD) and will refuse to come online, or will flag the conflict outright, so on current hardware you often get caught at commissioning. But disable ACD, or run older gear that never supported it, and the two devices quietly coexist: whichever one the switch's MAC table happens to favor answers a given request, so communication works most of the time and fails in a way that looks random and moves around the plant. People burn days on this because "it works most of the time" feels like it rules out an addressing conflict, when that is one of its signatures.
On Profibus, the equivalent is the token dropout. The network passes a token around the masters and polls the slaves, and if noise, a marginal termination, or a failing spur corrupts a frame at the wrong moment, you get retries. A handful and the bus shrugs it off. Let them stack up and a slave finally drops its station and re-registers before anyone's even walked to the panel. The bus diagnostics count those retries even when the segment looks healthy, and a rising retry count is one of the most useful early warnings you will ever get, if anyone is watching it, which usually nobody is. When you do go looking, that means checking the active terminators at the segment ends and putting a scope on the differential A/B pair to see the reflections and glitches the counter is only summarizing.
Then there is fiber, which is nasty because it degrades instead of breaking outright. One I chased was a connector nobody cleaned at install; you could see the smudge on the ferrule under a scope. Add a bend radius someone violated pulling the run through a packed tray, or a transmitter slowly losing output as it ages, and the link stays up at marginal signal, passing every casual test, until temperature or vibration nudges it over the edge for a moment. Measure the received power in dBm against the receiver's sensitivity and you find you have almost no margin left, which means every small disturbance becomes a dropout. That is not something clever controller code can fix. Sometimes the honest answer is cleaning a connector, re-terminating a run, or dropping in a new transceiver, and a power meter or an OTDR will tell you which.
What it actually looks like on a plant
A material handling operation I worked with had a screening and transfer line that faulted a remote drive rack two or three times a shift, always under load, and (this took a while to see) mostly on the afternoon shift. The maintenance log ran to pages: two drives replaced, a comms card swapped, the fieldbus cable re-pulled, all of it reasonable, none of it the cause. The pattern that finally mattered was that the dropouts clustered when a particular downstream section started and stopped frequently, which nobody had connected because the two areas were a hundred feet apart and on different one-line drawings.
Reading the managed switch's own port statistics, then leaving a laptop on a mirror port running a capture filter into a ring buffer across a couple of shifts, told the story. The switch feeding that rack logged bursts of CRC-errored frames every time the downstream section cycled. Those counts came from the switch, not the capture: a switch drops errored frames before they ever reach the mirror port, so the laptop only showed the gaps. The root cause was a slightly damaged Ethernet cable running alongside a motor lead in a shared tray, picking up interference on every motor start. Cyclic EtherNet/IP I/O runs over UDP with no retransmission, so most of the time the timeout multiplier rode out the occasional lost packet, but under the packet timing that heavy cycling produced, enough frames went missing in a row to time the connection out. The afternoon clustering was simply the plant running its heavy campaigns after lunch. Reroute one cable out of the tray, and the dropouts stopped. Software never entered into it; the controls value was purely in reading the network's own diagnostics to point at the offending cable, so the fix went to the right place the first time instead of the fifth. That interplay between what the process is doing and what the network is doing is most of the job in heavy process plants, where the electrical environment is hostile and a trip that looks electronic often traces back to something mechanical or physical.
What to check before you swap another part
Start by refusing to trust the controller's error message as a diagnosis. Treat "connection timeout" or "module not responding" as "something in the path failed," and go read the diagnostics that sit closer to the wire: managed switch port statistics for error and discard counts, fieldbus master retry and diagnostic counters, and optical power against budget on fiber links. Log them over time instead of glancing once, because the fault never happens while you're standing there with a meter. Then correlate the fault timestamps against what the process was doing, because "always under load" or "always when that other line cycles" or "always after lunch" is a clue a static test on a quiet plant will never reproduce.
And keep an honest accounting of what software can and cannot reach. A duplicate address, a saturated segment, a badly configured connection interval: those you can genuinely fix in the controls layer. A cable sharing a tray with a motor lead, or a corroded connector, or an optical budget that has run dry, you cannot touch from the engineering workstation. Those are physical, and the right move is to fix the physical thing rather than paper over it with longer timeouts and more retries, which only trains the system to tolerate a fault that is going to get worse.
If you have a rack that keeps dropping and a maintenance log full of parts that didn't fix it, it is usually worth someone reading the retry counters and the fault timing before you swap another card. I am happy to go through the diagnostics with you and sort the physical from the configurable; there is a short note describing how that works here.