I can say using accelerometers to detect displacement won't work. These devices have so much of inherent noise that discerning very slow acceleration from remaining immobile is about impossible. You won't be able to detect acceleration of order of 0.1 mm/s2 and it takes just a several seconds at such acceleration to exceed your 5mm.
You'd have a hard time with a camera as well, especially if the device is hand-held. True the camera looking along X axis would work with Y,Z plane displacement in confined environments (relatively nearby objects in focus) but it would get easily confused by rotation. You'd need at least two cameras looking in different (and not just opposite) directions and environment that is conductive to image detection (you won't get much displacement measurement from the image of the sky). Also, rotation will be a problem.
What will work with ~5mm distance ranges is frequency-based distance metering - preferably ultrasound, though infrared/visible light would be possible too (though obviously more difficult due to frequencies being much, much higher). Unfortunately plain "reflective" ultrasound distance sensors won't help much about rotation/tilt of the hand, distance from walls changing rapidly despite the device not moving. What you need are three ultrasound emitters (speakers), each of a different frequency, placed around the location where the measurement is to take place, and a sampler (microphone) in the "handheld". By filtering given frequency you get signal from one emitter, and then by analyzing the change of phase of the wave in time intervals equal to the wave period, you have displacement of the device relative to the emitter. By triangulating the three distances you have total displacement.
(also, a gyroscope will help you account for rotation.)
One more word of note as to why accelerometers are so ungrateful to work with:
For your 3-axis accelerometer, the "immobile" [X,Y,Z] readouts are [0,0,-9.807] m/s2.
That last factor changes with latitude, somewhat with longitudal location and altitude, not to mention internal properties of the accelerometer, like temperature or air pressure.
A cursory peek at some generic 3-axis accelerometer datasheet will give you a ballpark figure of maybe 1% precision. (0.3% non-linearity, 1% cross-talk, 300μg/√Hz noise RMS, thermal and other minor ones). Let's be overly generous and change it to 0.1%.
0.1% of 9.807m/s2 is 9.807mm/s2. That's the error you can expect from your accelerometers - changing from immobile to nearly a centimeter per second over course of one second, sunk in the noise.