In general, each tooth of an involute gear is made up of two curves: the involute curve and the root fillet. The involute curve is essential for gears to operate properly, and allows power to be smoothly transmitted from one gear to the other. The root fillet on the other hand is not involved directly with power transmission or the kinematics of the gears (i.e. how the motion of one gear influences the motions of a meshing gear.). This means that the root fillet curve's shape is reasonably arbitrary, provided, of course, that the root fillet on one gear does not at any point make contact with the teeth of the other gear. In addition, the root fillet also plays a roll in determining how much bending stress develops at the root of the teeth during operation: if the root fillet has a low radius of curvature, this will result in a large stress concentration at the root.
Despite the root fillet curve having the potential to be almost any shape, there are a few shapes that occur commonly, the two which I believe to be most common are as below:
1. Connecting involute curve and root circle with radial lines
The simplest root fillet to determine mathematically is simply a straight radial line that connects the start of the involute curve to the root circle. This is shown below:
The advantages of the root "fillet" (a sharp corner is not much of a fillet I guess) is the geometry is very simple. However, there are two big disadvantages: 1. The sharp corner produces a very large stress concentration at the root of the gear, meaning larger root bending stresses. This means that these gears are just not suitable for applications that subject them to strong torques. 2. In cases where undercut interference becomes an issue, these radial lines are going to get in the way of the teeth, causing gears to jam. For gears with pressure angles at the standard value of 20°, undercut interference usually occurs for gears with teeth less than 18 (if gear is meshing with a rack gear) or 13 (if gear is meshing with another gear of equal number of teeth). For these reasons, you will rarely see gears with this kind of root fillet in industrial application, but are typically fine in areas where teeth numbers are quite high and the gears are not subjected to high torques.
2. Trochoidal root fillet
One of the most common processes for generating spur gears and helical gears is hobbing, and the root fillet that is naturally produced as a result of hobbing is the trochoidal curve. An example of a gear with trochoidal root fillets is shown below:
The individual trochoidal curve and involute curve are displayed along with the root circle and outer circle as below:
The first thing to notice is that the trochoidal fillet is considerably more rounded than the previous case, so less bending stress will develop at the root of the tooth and so the gear is more suitable in high-torque scenarios. Also, notice how the trochoidal curve intersects with the involute curve at a point slightly to the right of the start of the involute: some of the involute curve has been inadvertently lost by the hobbing process! The smaller the number of teeth, the greater this involute loss will be, and it becomes apparent as undercut interference starts to occur: in such a case, the gears are said to be undercut. While for radial-line fillet curves the undercut interference would have caused the gears to jam, gears with trochoidal fillets will not jam in such circumstances.
Whenever gear teeth are generated by hobbing, it is as though a rack gear is rolling along a gear blank (circle with diameter equal to outer diameter of gear tooth), material being subtracted wherever the rack gear has been. The below animation should illustrate how the rack gear can generate the shape of a spur gear with trochoidal fillets:
Notice how the teeth on the rack gear are trapezia shapes (and also that the rack gear here, being a cutter, has an addendum of $1.25m$ and a dedendum of $m$, unlike usual rack gears), and the cutting of these trapezium teeth into a gear blank manages to magically create two curves, the involute and the trochoidal curves. More precisely, the involute curve is generated by either of the diagonal lines of the trapezium, and the trochoidal curve is generated by either of the two points on the end of the trapezium. With this information, it is possible to express the trochoidal fillet as a parametric equation.
The next section of this question will look at the mathematics behind deriving such a parametric equation.
Deriving the trochoidal curve
First of all, let's consider a single trapezium of the rack gear, and see how a point on the end of a trapezium moves relative to the gear being cut. The curve that this point follows will be the trochoidal curve. The pitch point of the trapezium, $P$, is the point on the trapezium that will initially lie on the gear's pitch circle, as shown below.
Note that the pitch circle radius is $R$, and that there is a general point $Q$ that lies on the trapezium, offset from $P$ by $x_0$ and $y_0$ in the $x$ and $y$ directions (later, we will set this point to be one of the points on the end of the trapezium). Also, note the solid black triangle: this is to help identify the orientation of the gear. The current position of $Q$ can be expressed as follows:
$$
\left[ \begin{array}{c} x \\ y \end{array} \right] = \left[ \begin{array}{c} R+x_0 \\ y_0 \end{array} \right]
$$
Now, let's rotate the gear anticlockwise by an angle $t$, which then pushes the trapezium up by distance $Rt$ (no slip between pitch circle and pitch line). Illustration:
Note how the solid black triangle has moved in accordance with the gear rotation. The position of $Q$ is now:
$$
\left[ \begin{array}{c} x \\ y \end{array} \right] = \left[ \begin{array}{c} R+x_0 \\ Rt+y_0 \end{array} \right]
$$
The trapezium has moved, but so has the gear: we are interested in the motion of $Q$ relative to the gear. Therefore, we need to rotate the entire system about the origin by angle $t$ clockwise. This simply rotates the perspective so that it returns the gear to its default position while retaining the new movement of the trapezium, i.e. we get the motion of a trapezium as the rack rolls around the stationary gear, similar to the animation above.
To get the position of point $Q$, we need to multiply the previous position by a clockwise rotation matrix, so the position is expressed as follows:
$$
\left[ \begin{array}{c} x \\ y \end{array} \right] = \begin{bmatrix} \cos{t} & \sin{t} \\ -\sin{t} & \cos{t} \end{bmatrix}\left[ \begin{array}{c} R+x_0 \\ Rt+y_0 \end{array} \right]= \left[ \begin{array}{c} (R+x_0)\cos{t}+(Rt+y_0)\sin{t} \\ -(R+x_0)\sin{t}+(Rt+y_0)\cos{t} \end{array} \right]
$$
Finally, by setting $Q$ to be the "upper left" point on the trapezium, i.e. $x_0 = -1.25m$ and $y_0 = \frac{1}{4}\pi m - 1.25m\tan{\phi}$, where $\phi$ is pressure angle, and also noting that $R=\frac{1}{2}Nm$, where $N$ is number of gear teeth, you get the parametric equation for the trochoidal fillet curve:
$$
\left[ \begin{array}{c} x(t) \\ y(t) \end{array} \right] = \left[ \begin{array}{c} (\frac{1}{2}Nm-1.25m)\cos{t}+(\frac{1}{2}Nmt+\frac{1}{4}\pi m - 1.25m\tan{\phi})\sin{t} \\ -(\frac{1}{2}Nm-1.25m)\sin{t}+(\frac{1}{2}Nmt+\frac{1}{4}\pi m - 1.25m\tan{\phi})\cos{t} \end{array} \right]
$$
Note that the parameter $t$ will start at $t=-\frac{y_0}{R}=-\frac{\frac{1}{4}\pi - 1.25\tan{\phi}}{\frac{1}{2}N}$ and increases up until a particular value that represents the intersection of the trochoid with the involute.
Hope that answers your question :)
