Did you read Shadows of Empire? The novelization of ROTJ? It's all there.
In short, Vader changed a lot during that time, due his paternal feelings resurfacing, also being close to Frank basically robbed him of his free will. Palpatine was just that powerful to the point of eclipsing Vader's will. One thing is Vader ploting while on the Outer Rim and Palpatine in Coruscant, another is with the guy in the next room. He couldn't do much with the Emperor so close of him and Luke.