Mt Fuji Climbing – Does the 4PM restriction apply when descending the mountain as well?

I wanna start climbing early in the morning but what happens if on my way down I reach the 5th station by let's say, 5PM? Does the 4PM restriction apply only to those on their way up?