We have one REALLY annoying neighbor. We just want him to stop coming over every single day.?

When he comes over, tell him this is a bad time, you're busy and do not have time for a visit. Do not let him in, after telling him this, close the door and make sure it's locked. Repeat as necessary, he'll get the hint. You don't owe it to him to include him in any plans with the other neighbors. I'm sure they all feel the same way about him.