I know that Lego has a history of both popular (Lego city) and non-popular (Jack Stone) themes. However, is there any hard data that back up popularity? What were/are the most popular Lego themes/sets of all time and were what the least popular?
By definition, the popular themes are the ones that sell well, and the unpopular ones sell less well.
TLG are in the business of selling LEGO. It makes business sense for them to offer for sale themes that have already proven popular, and withdraw from sale those themes which are least popular.
Based on these two observations, it seems obvious that themes that have a long run are typically the popular ones, whereas those that don't last long are less popular.
One major caveat to this line of reasoning is that TLG focuses on total profit, rather than total sales, so even of a theme is popular, TLG may still withdraw it if it doesn't make money. For example, lots of people claim that they would buy monorail sets if LEGO made them again, but TLG's research suggests that the cost of the moulds would make them unprofitable.
There will be other exceptions, too. For example, LEGO might only have a short run of licenced theme, not because it is unpopular, but rather because f the ternms of the licence. And, on some occasions, LEGO might simply make a mistake, and stop a theme too early or too late.
Nevertheless, the long running themes will generally be the ones with the best sales, ie. they will be the popular themes.
So, based on all this, I suggest that LEGO's own catalogues are a good source of information on which themes have proven popular, and which have been less so.
Assuming I'm right, I would say that City is the most popular non-licenced theme, with Police being the most popular sub-theme.
Star Wars is clearly the winner on the licenced front.
Least popular is harder to judge, because there are lots of themes that have only lasted a single season. Juding by the slating it gets, Galidor wasn't popular, but I don't have figures to back that up.
I'd be interested in actual numbers, if anyone has them; but do bear in mind that TLG may have commercial reasons for not wanting to share the whole story.