How do the costs compare between a railroad layout constructed with LEGO trains and a nearly identical layout using traditional scale models, such as HO?
I'd say that an initial HO scale set was probably cheaper than a LEGO version, although a direct comparison isn't easy, and it can rapidly get out of hand once you start adding more elements in.
The closest I can find is probably comparing 7939 LEGO Cargo Train, which is currently selling in the LEGO shop for £130:
With something like the GWR Passenger Freight from Hornby, which is selling for £80:
I chose that one as it has the basic oval plus "Track Pack A", giving some additional track including points.
Add a TrackMat accessory pack for £30 for some buildings or platforms and you're still cheaper than LEGO set.
An entry-level set from Hornby with just the basic oval of track retails at around £70, compared with £102 for the Passenger Train set:
However, once you venture out of the initial sets, go for the digital options, or move out of the "RailRoad" brand, the cost goes up to the point where you're paying £200-300 for an engine, up to the giddy heights of "Live Steam".
Extending your track
The comparison becomes even harder now, however this is where LEGO wins on price, if not on range.
The LEGO shop lists a Switching Tracks set with a pair of points (left and right) and 4 curved rails for £16, compared with Hornby's price of £12 for each point - or £29 for a pack with a pair of points, 2 curves and 2 points.
HO Scale will offer you more options in general however: Hornby for example have 4 different radii of curves, a number of different lengths of straight tracks, 4 different styles of points (Standard, curved, express and Y), crossings, and two flexibilities of "flexible" track (about £4/m compared with LEGO's £12/m), as well as a much larger range of locomotives and rolling stock.
I say this as having been a model railroader (http://zoorail.wordpress.com) for more than 10 years. You will never get the level of detail, scale or even perhaps value with Lego railroad sets as compared to normal model railroading but I just bought two Lego Train sets 7939 and 3677 and was blown away by the care that has not only gone into how they look but also how they operate. Out of the box, you are not only given a full train set and track; but the controller also allows you to independently operate up to 8 functions i.e. trains / lights etc. This IS possible with model railroading but it certainly isn't as straightforward or even if it is, if you have young children (like me) the marriage of ease of operations with child-proof playability makes this something you may want to consider.