Keeping an apartment in Tokyo without being classified as a tax resident

I'm giving up my residence card at the end of the year, and moving abroad to a country with no tax treaties with Japan.

I'm planning to keep the apartment I'm currently renting because I plan to spend in Japan a few months a year in order to have my daughter (half japanese) experience Japanese culture during the summer.

As I often read and as it works in most other countries, if I give up my Juusho and dont spend more than 180 days in Japan I should be fine from a Japanese taxation point of view.

The issue is that after consulting a few tax specialists in Tokyo, I haven't been told a single truth, one guy going as far as telling me that even having an apartment with my furniture inside is enough to infer that I plan on returning to Japan so I can be considered a resident of Japan and subject to universal taxation (ie financial gains abroad), even if I don't spend more than 6 months in Japan.

Did anyone hear something along these lines? As for context this guy's opinion is that given the amount of assets (ie exit tax paid) and the fact I'm married to a Japanese national, the tax agency would find any excuse to come after us and it's just better to cut all ties.