Ductile iron is actually likely to expand during the casting process, as shown in the table found on the Casting Wikipeda page, which is in part cited from this casting text by Stefanescu. However, as the tables show, the range of volume change is fairly large and not one-sided, which is to say the ductile iron may either expand or contract depending on other conditions of the casting process.
However, another concerning aspect is the range of precision that you're looking for in this casting. A tolerance of $^{+0.05"}_{-0}$ is pretty tight for a ductile iron casting. The ISO standard that lists general tolerances for castings (ISO 8062-3) says that for a machine-molded sand casting of ductile iron, the tolerance range on a linear dimension of $100-160mm$ is at least $\pm 0.035"$, and could be up to $\pm0.14"$. For a hand-molded casting, that goes from $\pm0.098"$ to $\pm0.24"$. These are general tolerances, so you could look for tighter tolerances on your critical feature, but I would suggest (if it's possible in your process) to plan on machining out that 4" diameter and set your core diameter up so that you are cleaning up the entire circumference of the bore. Otherwise, I think hitting the tolerance range you are looking for as-cast will be tricky.