The bolt pattern should be a standard; if there are three bolt holes then there should be 120 degrees between each hole. If there are nine bolt holes then there should be (360/9) = 40 degrees between each hole.
Let's talk for a second about vibration. Vibration will exist if the centerline of your flange does not coincide with the centerline of the existing flange. While your bolt pattern does matter, it doesn't matter that much. If you are relying on your bolt pattern to perfectly align two pieces of rotation machinery you are doing it wrong. The machinist that attaches the two flanges together should have a means to shim one piece of equipment, or the other, or both.
You need to rely on the shims to bring the two centerlines together, both coincident and parallel. Again, if you're relying on the bolt pattern to do this for you, then that implies that the flange needs to be perfectly attached to the shaft, that the bearings have to be perfectly sized with no room for dimensional tolerance, the body has to be perfect, the mounting feet have to be perfect, and the sub-structure to which both pieces of equipment attach has to be perfect. You need to have room for adjustment where the machinery bolts down. You will never attain the degree of perfection required to count on a bolt pattern to mate two pieces of rotation equipment.
So, that said, where vibration does matter with regards to your bolt pattern is that your bolt pattern should be EVEN. If you have three bolts that are supposed to be 120 degrees apart, but two are 110 degrees apart and the others are 125 each, then you'll get vibration because the bolt pattern isn't rotationally symmetric.
So, my advice to you would be to get the bolt pattern dimensions from the existing flange, then ignore minor variations and make your new pattern as symmetric as you can achieve. If the existing flanges are threaded, then your holes are through holes. There needs to be some tolerance there for the bolt to be able to pass through, and the tolerance should also allow for the minor variations in hole placement in both parts.
The clamping force of the bolts holds the flanges together. The shims under the mounting feet of the machinery brings the centerlines together. The location of the holes matters for vibration only if they're not rotationally symmetric. The fit of the the flange doesn't have anything to do with vibration provided you have adequately clamped the two flanges together and you aligned the shaft centerlines to be coincident, and again, that's done by moving the equipment, not by adjusting the flange bolt holes.
I'm harping on this to try to impress on you that a flange with through holes is not, and should not be, a piece of high precision equipment. You might need the face to be very flat if you're using a gasket, and it should definitely be perpendicular to the shaft centerline, but the through holes are just through holes. If one flange is able to rotate relative to the other flange then you haven't torqued it well enough and/or you haven't used enough fasteners. If the flanges were able to have relative motion then you're "riding the clutch" so to speak and can expect (very) premature joint failure.
So, all that said, here's how to calculate what the parameters should be for your circular bolt pattern.
- Find the angular distance between bolts, $\theta$, by dividing 360 degrees by the number of bolts. A 3-bolt pattern is 120 degrees between fasteners, 4-bolt is 90, 9-bolt is 40, etc.
- For a threaded hole, thread a bolt into each of two adjacent holes. Not necessary if you can get the measurement tips of a pair of calipers into the holes, but it does make it easier.
- Measure the largest outside-outside distance between the two bolts. Measure the smallest inside-inside distance between the two bolts. Average those numbers (add together and divide by two) and you get the center-center distance.
- The bolting pattern's radius is given by:
$$
r = \frac{\mbox{average distance}/2}{\sin{(\theta /2)}} \\
$$
Now you can put your blank flange on a lathe, find the center, measure out a radius $r$, mark that circle, and locate your through hole centers at the appropriate angular positions on that circle. If you're doing it by hand you could open a compass to the $\mbox{average distance}$ between bolt holes that you found, put the pointy end anywhere on the circle and sweep it, then move the pointy end to any sweep-circle intersection and sweep again.
Here's a graphic if that'll make it any clearer: