There needs to be an easier way to unblock them, and unblocking should be possible if it isn't already.

I know how to unblock someone once the hangout is over.  But I don't see a way to remove the "temporary block" that persists for the remainder of the hangout.