He is probably just saying he owns more property then he does cause he doesn't want you to build it so he doesn't have to look at it. I would build it anyway and tell the guy to go get bent.