That "isolation" is effectively just the "protect the public" element, which is part of the purpose of prison, but far from the whole point. The point, effectively, can be summed up as :-
- punishment
- rehabilitation
- protection.
Clearly, punishment is part of it. You break society's rules, society will punish you. Break them badly enough and that will include prison. Break them very badly and it'll include a long period of prison.
So no, the removal of rights is not just a side-effect - it's a primary purpose. See the above Justice Ministry quote.
But yes, that same society does still regard prisoners as having rights, just not all the rights free members of society have. The most obvious right you don't have is to come and go as you please, but it's far from the only right that's removed from prisoners. On the other hand, you are quite correct that just because you're a prisoner doesn't mean you have no rights at all .... or at least, not in this country is doesn't, and we're talking about this country.
So really, this whole argument comes down to whether the right to vote is one of the rights that prisoners should lose, or one of the rights that they don't. Prisoners do, as a matter of fact and a matter or policy objective, lose rights. Is this, and should it be, one of them?