It surely will affect one’s score if he doesn’t speak very clearly. However, I think stuttering will NOT be a problem if it is natural. A lot of time, it is taken into consideration. However if you stammer due to lack of coherence and flow it could cause some issues. This is because the marking scheme would not consider it natural but a deviation.
So what matters most is not HOW you talk, but WHAT you talk.