How can I re-finance my other condo unit?

I am the owner of 2 unit low-rise condo in Virginia Beach VA. We live in the top condo unit and rent out the lower condo unit. I had no problem getting a fixed mortgage for my top live-in unit. I have an ARM for the rental unit. Over the years, I have tried to refi the lower rental unit into a 15 or 30 year fixed mortgage. I am always told I cannot do that because I own both units. Any ideas on how I can solve this problem?