Guys, I didn't realize my "incident" would have led to such a spirited discussion, but all in all I think a lot of good points have been raised by you all. And the bottom line is a lot depends on the individual situation and the attitudes exhibited by the people involved. Without repeating myself too much, I truly felt the guy who ran up to me to insist I don't use the phone was way out of line. We all know that people too often make assumptions about a situation without gathering full information. This guy clearly wrongly assumed I was taking pictures of nudes with my cell phone. Personally, I would NEVER do that, or take anyones picture without their permission. So yes, I very much resented the fact that he assumed I was taking pictures and therefore implying that I'm a pervert or a predator. Even after I told him I was on a emergency phone call with my wife, he still insisted I hang up the phone.
Again, I really felt like giving this guy a massive roundhouse, but I made myself calm and realized that this was not a fight that was worth it. Pick your battles!
On the other hand, I understand Sam's perspective and his case shows what can happen when someone takes people's pictures without permission. None of us want to see our naked bodies plastered all over some seedy Internet porn site.