9
$\begingroup$

I wonder how can I make hard disk drive more durable? Specifically, I works in a factory environment. And almost every few months, hard disk drive in some of the factory machine are corrupted, and even unrepairable. We already implement several SSD on some of them, but that's not much of help. They're being corrupted too on factory machine with heavy task.

So every time it happened, we always restoring using our backup image. And we already attach all of the hard disk drive with shock absorber to reduce the shake.

Is there any other option or prevention we could use? Perhaps adding any anti magnet material to prevent magnet friction, or something else? And what material we need?

Edit: Factory machine with heavy task I mention above basically machine to create car metal body and car frame mould.

And when I meant the disk is corrupted, it means unreadable. The whole disk. Not the program files or software related. So it won't boot at all.

$\endgroup$
  • 10
    $\begingroup$ SSDs should not be affected by magnets. The most likely problems are temperature, humidity, chemical contamination (leading to corrosion of electrical connections, etc), dust and dirt (preventing proper air cooling) or the electrical power supply going out of specification - for example power spikes caused by large electric motors stopping and starting. Without knowing more about the "factory" it's impossible to guess which is most important. $\endgroup$ – alephzero Jun 13 '17 at 6:21
  • $\begingroup$ @alephzero Sorry, heavy task machine I mention above are factory machine to make car body and frame mould. Can you share possible solution based on your experience regarding my problems? Because it only happens to machine with heavy task. $\endgroup$ – adadion Jun 13 '17 at 7:00
  • 6
    $\begingroup$ Hard disks contains very sensible mechanical component, therefore I would also definitely recommend the use of solid state drives for all moving/vibrating location. However from reading your issue, I think your main problem has also a chance to a be a high voltage issue. If your power input is not stable, it will affect read and write, and eventually damage some components. $\endgroup$ – Jonathan Jun 13 '17 at 7:22
  • 4
    $\begingroup$ Are you sure this is actually a hardware issue and not related to software? I understand issues with hard disk drives, but SSDs should be immune to most of those issues. Can you provide more details about the "corruption" you see? $\endgroup$ – jcaron Jun 13 '17 at 9:39
  • 1
    $\begingroup$ In addition to a UPS, consider using two (or more) SSDs in a RAID1 configuration. If you set up monitoring of RAID errors, that should let you replace a failed drive without having to restore from backup. $\endgroup$ – Tor Klingberg Jun 14 '17 at 12:27
14
$\begingroup$

Quite likely, the SSD killer is electrical. We can't entirely rule out mechanical vibration, but SSDs are pretty robust mechanically. A simple rubber mounting would increase the resiliency even further. Also make certain that both power and data cables have enough slack. Vibration might cause them to come loose, when under tension.

So, to address the electrical reliability, we have to consider two factors. Firstly, the heavy machinery may draw large currents from the power supply. This could cause voltage drops, which in turn can negatively affect the SSD. This is easily solved by an online UPS. Essentially, this type of UPS powers the computer from a battery, while the mains power is used to charge the battery.

A more unusual problem could be electro-magnetic radiation. High-power machinery will have large currents running, often at 50 or 60 Hz. Unintentionally, this will cause cables to act as antennas. The big cables in the machine act as senders, and the cables to the SSD can act as receivers. The solution here is to have a proper Faraday cage, ideally grounded. That's why normal PC cases are made of metal; they work as Faraday cages, keeping EM radiation out in frequency bands between 50 Hz and several Ghz.

$\endgroup$
  • 5
    $\begingroup$ +1 for mentioning electromagnetic effects. (I don't know how serious that would be for PC cases that we are considering). $\endgroup$ – Gürkan Çetin Jun 13 '17 at 10:11
  • $\begingroup$ I think also it is the answer of the problem, but not of the title question :) you get my vote up anyway. $\endgroup$ – Jonathan Jun 13 '17 at 10:29
  • $\begingroup$ It's entirely likely that the heavy machinery is causing problems with the ground that might be alleviated by isolated grounds. $\endgroup$ – chrylis -on strike- Jun 14 '17 at 0:48
  • 1
    $\begingroup$ It seems voltage drops are the main problems. We implements some online UPS to some of them, and the meter showing great stable result. And in addition, we put grounded Faraday cage to 2 of them (the old ones). Superb!! Will projected SSD installed on all of them, and make the old one as backup. $\endgroup$ – adadion Jun 14 '17 at 7:45
11
$\begingroup$

First off, electrical and magnetic problems are not as bad as problems of vibrations and air contamination. Moisture in air plus dust or chemicals can corrode or short paths quite easily, and in our installations they are the primary reasons of failures if the devices aren't protected properly.

The best option is just keeping all that's not necessary on the production floor off-site. Keep minimalist embedded controllers by the machines, keep the PCs in a neat office communicating with the controllers over LAN.

If that's not possible, you need sealed cases. Possibly with heat transfer elements, if needed; airtight boxes that keep most of moisture out, some silica inside to absorb the rest of moisture - neither dry dust nor clean air moisture alone are a big problem, but combined they quickly lead to oxidation of contacts, other corrosion-related problems.

In my experience, EM disturbances are rarely powerful enough to cause any lasting damage. They may knock a device out, forcing a reboot, but a well-built device will recover from that. Power surges are a different matter; without a good surge protection you may see random damage of parts.

Finally, vibration. The vibration conducted by the floor is easily reduced to negligible levels through a sponge mat or similar. The vibration of a machine, in case the device is directly attached to the machine... there's little that can be done about it. There are dampening systems, but they are only efficient against certain vibration scales... really, just move that control box 2 meters away.

Also, temperature range must be "within acceptable levels". You WILL see corruption on overheating devices, and moisture will condense on too cold ones. This is rarely a concern on a production floor, where too many machines depend on it, but as you seal the disk (intentionally, or unintentionally, e.g. through dust) you'll see overheating.

$\endgroup$
  • 2
    $\begingroup$ You can use KVM extenders and USB extenders to get the machines further from the sources of heat/vibration/contamination, including the other side of a wall. $\endgroup$ – Chris H Jun 13 '17 at 11:08
  • $\begingroup$ @ChrisH: Never did that, but it should work. In our cases, the minimal embedded controller in a tightly sealed box sits right next to heavy machinery deep in the mine, providing a small console for immediate control using a watertight membrane keyboard installed in the face of the box, while a PC for all regular operations and monitoring is in a control center several kilometers away. There's an ethernet switch with spare sockets in that box, so if you need on-site control for whatever reason, you can bring a laptop. But that's only for servicing, not regular use. $\endgroup$ – SF. Jun 13 '17 at 13:12
  • $\begingroup$ I've run USB over CAT5 (proprietary extender) for keyboard, mouse and a webcam and VGA over another CAT5 (another proprietary box) for 100m to allow a system (a fibre-coupled spectrometer) to be controlled from two locations. It worked very well. Over longer distances, real ethernet would work, or better still do everything over fibre and opto-isolate at the same time. Comms to the actual machinery can be an issue though $\endgroup$ – Chris H Jun 13 '17 at 13:35
  • 2
    $\begingroup$ Tibbo manufactures "RS232 over TCP/IP" bridge devices. They are a bit of a bother, but there are city-wide networks of traffic lights controllers that depend on these; "command center" running a software bridge (dozens of virtual RS232 ports on a PC) and the controllers all over the city, connected to fiber, controlled through the small bridge boxes. Awfully antiquated approach, if you ask me, but works, and these are not airtight cases. So if the machine is controlled over RS232/485 you can use this sort of device. $\endgroup$ – SF. Jun 13 '17 at 13:51
3
$\begingroup$

Estimated lifespan is given by the manufacturer usually as "MTBF = 2000hrs" but in "normal conditions" - what you describe is not normal.

Why are the SSD drives failing - physical damage or poor connections.

One anti shock mounting used in the past was a mercury bath but you probably won't be allowed that !! But you could make an oil bath version...

$\endgroup$
  • $\begingroup$ My thought it is because shock occured by the task is pretty hard, so ordinary antic shock mounting won't help. How can I make the oil bath version? Believe me, when I googling oil bath anti shock, google provides me silly picture. Now I'm being dumb. $\endgroup$ – adadion Jun 13 '17 at 6:45
  • $\begingroup$ A rectangular tray holding the oil (of sufficient depth) and a smaller tray floating on the surface holding the disk drive - have a loop in the cable to absorb movement. Small soft springs to stop the floating tray touching the sides - pen or biro springs may be good. Choose the oil so it is not too viscous. $\endgroup$ – Solar Mike Jun 13 '17 at 6:56
  • 1
    $\begingroup$ For the second version you can consider a lid... good luck $\endgroup$ – Solar Mike Jun 13 '17 at 7:04
  • 1
    $\begingroup$ @adadion If you don't have a vibration sensor / meter there are some smartphone apps that will give an idea - accuracy may be an issue but may be sufficient for comparison. $\endgroup$ – Solar Mike Jun 13 '17 at 7:24
  • 1
    $\begingroup$ @adadion: If you're willing to experiment, you could give something like this DIY hard disk anti-vibration mounting setup a try. It's designed to keep vibration from the drive itself from getting to the case, but it should work just as well both ways. Of course, it may not help much against high-amplitude low-frequency shocks (like what you might experience driving fast on a bumpy road) that could make the drive bump against the mounting bracket, but if you're having a lot of those, you've got bigger issues anyway. $\endgroup$ – Ilmari Karonen Jun 13 '17 at 11:52
3
$\begingroup$

Really a comment but too long:

I've dealt with PCs on the factory floor (woodworking), they proved quite resilient.

Our initial setup which was basically trouble free: We mounted the PCs inside a cabinet, the front was clear plastic slats (think what you sometimes see on a walk-in refrigerator freezer). The original intent was to maintain a slight positive pressure with clean air but this was never done and proved to not be needed.

Unfortunately, after that plenty of machines got installed with less care. The usual "failure" mode was thermal shutdown, take the cover off and blow it out, it would work fine, although these did prove more problematic as the dust did some damage.

The main problems, however, came from their wires. We specced shielded cables but management went cheap on us, the building was wired with ordinary network wire and later modifications were often made by electricians rather than computer guys. This caused a lot of interference and was probably responsible for the high failure rate of the network cards. (Really, now, a Cat-5 tossed over a 480V, 400?A main power bus??? Or even more extreme, a Y connection in a cat 5--which actually worked, albeit with network error problems!) Don't put a computer on the same circuit as a heavy motor. Don't run any computer wires parallel to heavy power wires even if they're on separate circuits.

The initial machines were all diskless (not an option these days) and even after that everything of importance was stored on the network so if a machine did act up it could be swapped out very quickly--it took longer to carry the new machine to the station than to get it up and running in place of the problematic one.

Top lesson--don't let the electricians be anything but carefully-watched assistants when wiring things.

$\endgroup$
2
$\begingroup$

We are assuming that the problem is caused by shock or vibration. There can be several other causes, such as temperature, humidity, corrosion, chemicals, as pointed out by others.

One more approach would be to get the drives to a non-hostile place and extend the connection with cables. This may need your computers to run from external drives.

If you cannot take the external disks far away, you can still place them in a cushion (sponge like material) to avoid shock.

If you still cannot escape the vibration or shock, investigate changing your computers with tough ones. This will probably cost your company some dollars, but it's probably better than production stopping.

$\endgroup$
  • $\begingroup$ As the OP said "reduce the shake" most seemed to have assumed the problem is due to vibration... $\endgroup$ – Solar Mike Jun 13 '17 at 8:45
  • 1
    $\begingroup$ Agree that vibration was the main/big cause however probably the second cause is now in place. Which can be electrical or any other, as Jonathan suggests. Isolating the disks from the pc would also circumvent this option as USB power is calmer than the main board? $\endgroup$ – Gürkan Çetin Jun 13 '17 at 8:49
  • $\begingroup$ Wouldn't it be too risky to place the drive way out from the station? Human error couldn't be avoid. :( $\endgroup$ – adadion Jun 13 '17 at 8:53
  • 1
    $\begingroup$ That's a matter of ergonomics, and if the source is the station itself, and not the surrounding environment, this way can be the way to go. For human errors, do you mean cables on the walkways? You could also consider just raise them not touching the machine.. I can also think of wireless disks, which would probably cause several more other concerns. :) $\endgroup$ – Gürkan Çetin Jun 13 '17 at 10:09
2
$\begingroup$

As others said, SSDs are resistant against vibrations - there are no moving parts, unlike magnetic hard disk drives.

Both of these technologies are, however, vulnerable to electromagnetic fields, as others stated as well. Providing protection against that may help.

You should, however, also not discount other factors.

(A quick note: this list definitely isn't complete. Just look at scope of the other answers - from voltage levels, to heat, to software - there is a LOT that can cause these issues. Unless you're confident with computing, you might want to look at hiring someone to figure this out for you, because they might be looking at different factors on-site that you didn't think of. That said, here are a few factors that you should also consider.)

  • There might be a problem with the cable, and such errors can be very subtle in that they just sporadically show up. Test the "defective" hard disks in a normal environment on a different PC, with different cables - to make sure they actually have physical damage.

  • It can be your memory as well. Unless you're using ECC memory, this can be difficult to identify. If your bits flip in memory, and that just happens to be where your program, the operating system or its drivers reside, then all bets are off. It might do nothing, it might crash, or it might just write garbage all over your disk.

  • It might not be a hardware issue at all. A software bug can also corrupt data. Having an exotic driver stack can make your system more prone to corrupting data.

Depending what exactly is the cause (you need to determine that first!) we can recommend possible solutions. There are plenty of solutions - from isolation, to RAID, to checksumming file systems such as ZFS - but you need to determine the cause first.

$\endgroup$
2
$\begingroup$

In addition to the other answers: in the environment you mention, it's possible there is metallic dust in the air. When that gets into the computer, you can get electrical shorts. A sealed case (or ventilation with high-quality air filtering) can help if that's the case.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.