Struggling to speak

So bit of background first. I’m probably about 10 years of studying in, though when I say 10 years I say it loosely. I was on and off with it. I’m currently above N5 level but not quite at N4. Previously my speaking was infinitely better than my reading/writing. These days, my readding/writing is much better. I’m definitely at my strongest point in terms of how much I know. But I feel since before my speaking is just terrible now. I get it, if you don’t use it you lose it. Thing is, I can read stuff and speak it out no problem. But in actual conversation I’m useless.
Today I done my first language exchange with a Japanese person. She would ask me the most basic stuff and I just blanked. I couldn’t answer almost at all. Even stuff such as what time is it. I understand sentences much more complicated than that. I know how to say it but I find myself constantly saying “did you say X” but in English. I get nothing from doing language exchange it seems. Do any of you have advice on how I can progress at this point?