Can condo association loan be refinanced for less money?

We have been using North Shore Bank for our condo association loan. We took out $500K four years ago and paid it down to $340K.  With lower rates we asked them to refinance their answer was: "Not interested" asked them why? What is the criteria to refinance? The VP just kept saying "We are not interested, you can get a refinance from another bank at a lower rate and we will not waive the early prepayment penalty." Is this the norm at association banks? or was this VP just being....