Half of your statement is correct: If we are considering the following system:
$$G : \begin{aligned} \dot{x} &= Ax+Bu \\ y &= Cx \end{aligned}$$
then $A$, $B$, and $C$ all influence the zeros of $G$, but $C$ certainly does not affect the poles. The poles $G$ are the eigenvalues of $A$.
This makes sense from a purely common-sense consideration: the matrix $C$ is the output matrix, it is only capable of scaling and adding the states to result in an output. It has no effect on the states themselves, therefore it won't affect things like stability of $G$ (which is related to the poles).
You can easily see this if you look at the transfer function equation you provided, with the inverse expressed as the determinant and the adjunct (or adjugate):
$G(s) = C(sI-A)^{-1}B = \frac{C \text{adj}(sI-A) B}{\text{det}(sI-A)}$
$C$, $B$, and the adjunct of $A$ all appear in the numerator, but only the determinant of $A$ appears in the denominator.
However...
Here's where your problem is likely coming from: If we consider $G$ to be a system plant being controlled through some feedback element with a transfer function $H(s)$ (like the following picture from Modern Control Engineering, 5th ed. by Ogata):
Then the overall transfer function for the entire system, from reference input $r$ to output $c$ is
$$\frac{C(s)}{R(s)} = \frac{G(s)}{1+G(s)H(s)}$$
Now, since $G(s)$ is in both the numerator and the denominator, you can see that $A$, $B$, and $C$ affect both the poles and zeros of the closed-loop transfer function.
In conclusion:
- Only $A$ affects the poles of the system plant $G$
- $A$, $B$, and $C$ affect the zeroes of the system plant $G$
- $A$, $B$, and $C$ affect both the poles and zeroes of the closed loop system
Reference: Modern Control Engineering, 5th ed. (2009) by Katsuhiko Ogata