I've had this problem, too. There are some books that I was uncertain if I really disliked it, or if I just couldn't get past the reader's performance. Likewise, there are some books that I might have only liked because the quality of the narration overcame the story's flaws. I always use the preview sample to listen to the narrator first on Audible, but that's not always enough to know for sure, especially if the problem is a mismatch with the narrator's tone or interpretation, rather than just a wooden performance.

Yes, I also listen to the Audible sample, which is especially important to me when there are several versions to choose from. But, like you mention, that isn't always adequate to know for sure. On this particular book, there were no other choices.

This has happened to me too--I've disliked a character who I think I would have liked in print just because of the narrator's voice.

Glad to know I'm not alone on this issue!