Owner of house on 2 lots calls foul on attempt to collect past fees

We purchased a house that originally was on 2 lots. The previous owner filed with the town and had the lots joined. When we purchased the home, the previous owner was only paying HOA fees for 1 lot.

For the last 4 years, we too have only been paying for 1 lot. We just got a notice that we have 2 lots as defined in the covenants and we now owe for the last 4 years on the lot they NEVER before billed us for.

It doesn't seem legal that they can require us to pay for anything prior to this notice, since they have neglected to bill us each year.