This is back around 1988, when we are in the purpose-built datacenter
that I designed, running some flavor of MVS on a pair of IBM 3033, with
3330-11 DASD and a dual STC 4305 per 3033. It is before we get the
3851 MSS, about which multiple stories are to be told.
One of the 3033 starts dying every afternoon, right about 5 PM, half an hour after the regular day folks leave for the day. It is very nearly as regular as clockwork, and always with an 028 wait state: something w0rng in the Link Pack Area page dataset, which is on one of the two 4305 boxen on that 3033. Everything passes all the diagnostics perfectly, and the STC techs are tearing their hair out.
Finally I stay late one day, to see what's happening. Janitors come in, one pushing a mop bucket in front of him. It hits the corner of the problem 4305, and the system goes into a 028 wait state Right Then. I walk over, open the door to the power supply frame, and look. All's OK. I kick the frame hard, and a power supply OffAndOn. I hit the frame gently with the heel of my hand, and that power supply OffAndOn again. I start wiggling switches on that supply, and find that one of them has no detent action whatever.
I call the tech and tell him to bring a new switch. He comes in, replaces it, and it works solid from then until we sell the box to someone else years later. We dissect the switch and find that the spring, that should provide detent action and force the contacts closed, has b0rk3n right in the middle, so that there is no force at all being exerted on the contacts.
That was the only mechanical failure I ever saw on any of the STC 4305 gear, but it took a _long_ time to isolate.