I would suggest not using hinges of any kind, but just making a plain panel with a tile on top, resting on a tile in the wall. For an example from LEGO itself, have a look at booklet 2 of the instructions for set 7094. Notice they use panels instead of flat tiles which is even less visible from the outside (although they did it not for that reason but to make sure it was easier to remove).
In your case, you'll want to block it from behind (with slopes at each corner for example) as you don't want it to go into the structure.
To remove it, you'll either need to draw it by some protruding fixture if you can fit one in, or push it from the inside by another mechanism (for example, pushing on something on the top would activate a lever which would push the wall)