There's a reason why they're called "factors of safety"
The answer to your question is in its title.
When designing a structure, we need to make sure of two things:
- that the structure will remain a structure and not end up as a pile of rubble (i.e. that the structure is safe)
- that the structure can actually be used as intended.
The first is easy enough to understand: you either have a structure or a pile of rubble; do the math to ensure it's the former even under extraordinary circumstances (as described by factors of safety)
The second is more subjective... what does it even mean, really? Well, basically, that human beings will be able to perform whatever tasks they wish while using the structure.
Would you be able to sleep at night if you look up and the concrete beams are all cracked? If your bed bounces with every step your neighbor takes during their late-night walk to the kitchen for a cup of milk? Would you feel safe if you get to your office and the beam at the reception is drooping like Atlas' shoulders?
Of course not. So that's the purpose of serviceability limit state checks; they aren't there to make sure the structure is safe, that's the job of ultimate limit state checks.
Serviceability limit state is merely about whether the beam will suffer large deflections, and/or vibrations (and other checks, depending on the situation, such as cracking in concrete). And you only care about these under "normal" circumstances, which are described by the characteristic load values (if the structure undergoes an extreme, once-a-generation loading, do you really care if it deflects a bit too much for comfort?).