The most valued sets, when not using price as a criterion, will vary from collector to collector, based on their interests.
In my experience doing LEGO Customer Service, collectors of all ages tended to focus on particular themes. While the Ultimate Collector's Edition Millennium Falcon may be the most prized jewel to a Star Wars collector, it may have little to no appeal to an avid Train collector. It was very hard to talk about any themes outside the fan's specific interest. Even then, a Trains person may have interest in the MAERSK ship(not a train, but related), but none in the Hogwart's Express. Another sought-after set is the Grand Carousel, which isn't likely of interest to someone that only collects licensed sets.
For myself, the most prized LEGO toy I could attain is an authentic wooden set from the pre-ABS era. The closest thing I have to it is the 2011 employee-only duck that resembles the pre-building system era wooden duck draw toy the company made (post-yo-yo fad, using the leftover yo-yo halves as wheels).
Other categories of collectors are completionists who may value SDCC Bizarro minifig or a limited UK Sun bag as their most prized possession (similar to the Mr. Gold mentioned before.)
The currently-selected answer is worded to be exclusively about how expensive the item is, which the question specifically requested not be used for this judgment. However, lacking any other criteria, it's not feasible to give a specific answer.