Is a half-day trip to Sengen Park (overlooking Mt. Fuji) enough?

And after that will go back to Tokyo for more sight seeing within the day feasible (Sensoji, Shibuya Scramble and Sky tree)? We will only be in Tokyo for 3 days and want to slide in a side trip to famous mt Fuji pagoda.

Thanks in advance.

by PrinceNebula018