I had the same issue, then realised the voice needs to be speaking to turn up the volume, otherwise I was just turning up the ringer volume in my iPhone.
I went into settings-audio and clicked between medium and loud volume setting to then enable the voice which allowed me to turn him up!!