This is the only problem I have ever had at the Freedom trail head. I have camped there more than 200 times, is with the forest rangers. If you camp on the north side of the east and west road there you are on B.L.M. land on the south side of the road is where the S.B. National forest land begins. I was camping on B.L.M land and they gave me a ticket for not having a forest adventure pass. I had to go to court in the Black Rock desert (near Palm Springs) to win my case.
The only other time you could call it a problem was with some drunk Korean women. They were overly aggressive, I am used to that problem.
The decision is yours , I hope you have a great weekend. Please post a trip report and let us know how things go for you and your friends.
Glad 2 help.