As a punishment for something naughty that we did, my little brother and I have to whitewash both sides of a fence. We start at the same time, and we each work at a constant rate. If we each whitewash one side, I'll finish in $2$ hours and my brother will finish in $3$ hours. But I'm a nice kid, so after I finish my side, I go around to the other side and help my brother finish his side. From the time I start helping him, how many minutes does it take us to finish the job?