Can my estranged brother obtain a restraining order against me & my parents?

OK so over the past 7 years my family has been dealing with my nephew being abused as an infant. He had a broken collarbone, fractured ribs, a broken arm, dislocated elbow and a broken femur. He was not allowed to return to the parents home and subsequently lived with my parents for a few years....there are now 3 other babies. After all was said and done the court gave my ex-brother and his wife probation for the offenses. My parents as well as myself have been supportive of the children this entire time, having them all weekends, holidays, my Dad would take them and pick them up from school daily, etc. They moved to my parents street, yes, the same street and after they were sentenced they cut off all contact with me and my parents. My mother tried calling her son and he cursed her out up and down and said the if her and her husband and daughter keep driving by his house hes gonna get a restraining order? Is this even possible? None of us have contacted him in over a year but they live on the same street and we do need to drive by their house to get to the stop light to cross traffic. This situation is absolutely ridiculous. Oh and we live in New Mexico so any insight into this situation would be a great help. We miss the kids but we dont stop in front of their house, we simply wave to them if they happen to see us. So my thinking is how will they support trying to get a restraining order against us if they were the ones that moved right down the street....literally there are 5 houses separating us. SMH