How to compute Intersection over Union for multiple overlapping boxes?

I have a set of predicted bounding boxes and a set of target bounding boxes. In each set, some of the boxes overlap. I understand that IoU is computed by counting the total area of overlap between pairs of corresponding boxes in predicted and target sets, and then dividing it by the total area of the pairs. I'm not entirely sure how that extends to boxes that overlap within a set.

For example, lets say red boxes are in predicted set, and blue boxes are in target set (i made them opaque, but there are just 4 boxes here - 3 squares and a rectangle):

Assume that the small red square corresponds to the small blue rectangle. And big red square corresponds to the big blue square.

Now, the IoU for the first pair is 1/5. For the second it's 4/9. So is the total IoU (1+4)/(5+9) = 5/14? Or is it 4/9 - all area covered with at least one blue divided by all the area covered total? And if it's the prior, how would you go about calculating it if you don't explicitly know which boxes in the prediction set correspond to which boxes in the target set (like if I didn't tell you that small red goes with small blue and big red goes with big blue)?