Do I lose my spouse visa when I leave Japan?

My Japanese fiancée and I are getting married next spring (yay)

I have a working visa valid until 2027, but I'm thinking about changing it to a spouse visa.

We are thinking about moving to America in 2025 or 2026 and then coming back to Japan in 2027 or 2028.

If I leave now, I understand that I will lose my working visa as soon as I move out. But, does the same thing happen for spouse visas? If I can keep my spouse visa while we're in America, it will make job hunting a bit easier when we come back.

Also, we're not planning on maintaining a place of residence in Japan while we're gone.

And before anyone starts recommending getting PR:

1. We need to be married for 3 years before applying.

2. I've only lived in Japan for 5 years, which isn't enough to apply on my own.