Recently, I have noticed that there appears to be no rhyme or reason to the sizing of any given LEGO set box. For example, in the picture below, you can see three sets that have a similar amount of pieces, yet three completely different box sizes. Why does this occur? Is there any rhyme or reason to the sizing of LEGO boxes?
This is marketing department wizardry.
From a pure engineering point of view, the closer to a perfect cube (or rather sphere) you are, the greater of internal volume you can enclose with the same amount of material.
What does this mean? By not making cube shaped lego boxes, Lego isn't trying to be as efficient as they can be when it comes to their box design. This costs them more in terms of packing materials, shipping container size, and store shelf capacity.
And yet, they are happy with this choice because the marketing department has determined that big pictures of sets on the front of the box attracts consumer's attention and results in greater sales. In a vacuum (say at a LEGO store), they could be any size and the packaging would not be competing with other products (cough, cough mega blocs), but in the real world both companies are in a arms race to catch the eyes of uninformed grandparents who don't exclusive buy LEGO for their grandkids. So big, wide box fronts are ideal. If LEGO could get around the other problems of cost, shipping, and shelf limitations, they would make every box set look like it contains a big screen tv.
The ratios of height and width are interesting, as they seem to dance around a ratio between 4:3 to 16:9. (I cannot think of too many sets that are taller than they are wide). Some will be due to the dictates of the set inside (baseplates or special pieces), but LEGO clearly favors a ratio that mimics TVs and monitors. This is probably because they can keep the box height constant but play with box width to accommodate bigger sets. This makes it so that stores don't have to do shelf resets all the time, or mix sets of differing heights.
Magnus' answer doesn't explain why different themes have same sized boxes or same front area (but different depth). Also same set re-release can have different sized box (take Mini from Creator for example). So there doesn't seem to be a rule for that.
What I heard from TLG employee is that front area of the box is basically a marketing space. So TLG tend to adapt the size of this area to desired shelf space. There is certain levels where you can put each shelf, so each box must utilize all the given space without sacrifying advertisement value.
Layout of the sets you see in the shop is science, sort of. And TLG spend considerable effort to reach desired value. They even test new layouts in their employee shop at Billund.
The size of the box seems to be theme specific.
It appears that they choose a specific box design for any given price range and then stick to that size for the rest of the theme. For example, in the pictures below, you can see same sized boxes used in the same themes. There is definitely some crossover, but for the most part this holds true.