Looking through texts such as Shigley's Mechanical Engineering Design, you come across tables of the stress concentration factor (defined as $K=\sigma_{max}/\sigma_{avg}$) such as the following:
This plot is sourced from Pilkey's Peterson's Stress Concentration Factors, which, unfortunately, I do not have access to (and even less so now, since interlibrary loans are on hold), so I do not know what underlies this plot. However, the line seems to be related to Kirsch's elastic stress distribution since the value of the stress concentration factor approaches 3 when the ratio r/d approaches zero.
That exact item, though, is what gets me. In the limit of radius going to zero, the bar no longer has a hole in it, and according to Saint-Venant's principle, the stress a a point in the middle of a long bar should be equal to the average stress on that cross section. Hence:
Why does the stress concentration factor go to 3 instead of 1 as r approaches 0?