Help with multiple layers moving position

Hi I am new to After effects so apologies if this is a daft question.

I have a map as my main layer and I have created a text layer.I want to animate the position of the map which can easily be done.However all other layers move with the map layer.So if I move across the map all other layers moves aswell.When I want them to stay in a fixed position

I would be grateful for any help

Thanks