While there are multiple sizes of instructions and lack of database with dimensions my answer is based on personal observation.
Looking at the stash of my instructions from multiple sets (including several largest Technic models) I can state that the largest dimension I have is of A4 paper size, which is 210 x 297 mm. I'd like to note that this is for Europe based sets. There might be different sizes for other markets as LEGO provides two options for their instructions online - optimized for A4 as well as Letter sized paper.
Edit 1. As masterX244 pointed out in the comments Star Wars UCS sets may have bigger instructions. This review on Brickset of SW UCS set with highest number of parts (75192 Millennium Falcon) says instructions are 42x2942 x 29 cm in size. This is very close to A3 paper size, which is twice the size of A4.
Picture below, from the very same review by Brickset, shows the instruction relative to side of the box (45 x 39 cm).
Edit 2. JPhi1618 found that UCS Star Destroyer 10030 set has a booklet that is slightly larger - 43 x 30 cm (landscape orientation, including the spiral)