I am so sick of trying ro get my doorbell out of its holder

Every time I need to get the doorbell out of its holder, to charge it, I fight with it for 5 or ten minutes. I tried putting the “key” in at every possible angle, but it just won’t budge. Finally, at some seemingly random point, it will pop out, but I see no pattern. The manual implies that it is ultra-simple, but I am sick of it.