Not worried. They did it in the dark because they knew it was "illegal", otherwise they would have done it in the middle of the day with no issues if some of the neighbors had said "whatcha doin?"
Easements are a big problem here. They are part of your "real land" but they provide ingress/egress for your neighbors on either side and allow for utilities, emergency vehicles and services, like trash, to go from point A to point B.
When someone throws up a barrier, it has to come down immediately, because if you call the Sheriff in, it's all about "Land Court". It's really ridiculous. Our little corner of paradise is basically "land locked". There's several places where I have to drive on private property and State Land Trust to get out of here because idiots have fenced in the easements.
Land Court can literally take YEARS, and you have to file down in Phoenix because it's handed over to the State to take care of. Most people don't have the Title Insurance that pays for an attorney and fees if there is a dispute, they just guarantee that you do have a legal ingress/egress, whether it's fenced or not. 7 years ago my BF had to go to court to get a barrier took down on his easement, it was done here in the county because the evil neighbor threw in an injunction against harassment, BF's title insurance was top of the line, with a VA guaranteed loan, and the sheriff already had issues with the neighbor.
So.......long story short, the sheriff told us 20 years ago that if a barrier goes up and we're within our legal rights, TEAR it DOWN before calling them.