Another good place is at the start of Barrett-Stoddard Road. Just hike down the old baldy road (it's blocked by big rocks and is washed out in places). There are some great swimming holes down there and very few people.
http://maps.google.com/maps?hl=en&ie=UTF8&ll=34.218474,-117.662716&spn=0.031015,0.068407&t=p&z=15
You can see cars parked at the trailhead:
http://maps.google.com/maps?hl=en&ie=UTF8&ll=34.222255,-117.667573&spn=0.001938,0.004275&t=h&z=19