I haven't read this series, but I'm so with you on the audiobook experience. New-to-me narrator angst seriously should be a thing; and I *hate* it if they switch narrators mid-series (just as I used to hate when they would switch actors for TV series characters back in the day). Yech.
I think the first time I paid attention to this sort of thing was when they replaced Barbara Bel Geddes by Donna Reed as Miss Ellie in "Dallas". They may have had a reason for doing it (Bel Geddes was recovering from cancer surgery IIRC), but it still felt wrong -- and the changes have gotten more and more arbitrary ever since. WHY??? You're right, it destroys the whole thing.