Am I supposed to stay near an actual major station?

Can someone explain something to me.

So lets say I want to stay in Shinjuku or Shibuya. Do I pick a hotel thats close to Shibuya Station or Shinjuku Station? Or are there other stations throughout the city that are still a few minutes away from a station?

I am a little confused as to how this works. Do trains just go from Point A to Point B? Or do points in the city just take me straight to the major station?