To remove the handle only, just grab it and pull straight out from the door.
To remove the inner door panel, take off the armrest, take out the speaker, and then you need to work your way around the edges, pulling the panel out bit by bit. There are a lot of plastic "christmas tree" clips holding the panel into the door.
-Karl