The hike in is the easiest part. Tough on knees for the downhill but even if it was hot, it goes by quickly and you get to instantly cool off when you get there.
I found that hike out at 3pm to be the tough part. Uphill and hot with almost no shade stops. Then no swim at the end. I bet hiking out near sundown is much nicer.
It is supposed to get hot Wed/Thur this week though. But I bet it would be a fine. I saw several groups hiking in at 3-4pm on Friday. Thinking that 2-3pm hour is the worst. I'd hike in at around 1pm no problem if I knew I was going to have time to cool off and the hike out was at a cooler time.