Will I be taxed in remittance for property purchase?

I've read that remittance are taxed as income, but I'm not 100% sure how that works, because it doesn't make sense (to me) to get taxed on the money that's already taxed somewhere else.

I have to transfer around 50 million yen from US for a down payment, should I expect to pay 45% tax on that amount? (based on my income level). I'm terrified now….

Thanks for any advice.