I’m not really an R user, and I’m not experienced with plotting in R, so all I can say is that based on looking at the sources, passing rescale = FALSE to plot prevents this rescaling (but doesn’t adapt the plot range).
Perhaps someone else can help more.