I don't think think would be an accurate.
For ex. My Son in Law is 6 ft 2 in and size 9 1/2
He is also 60 lbs heavier than me.
I am 5 ft 10 in and size 10 shoe
I also agree that it wouldn't be accurate to judge a persons height on shoe size.
Anecdotally, I would agree with you both.
Scientifically, though, I think that there are formulas that are used, and general correlations that can be made between foot size (length and width), and height.
Here is one example, from Forensic Science International, where they did a study on the issue.
Stature and sex estimate using foot and shoe dimensions:
The aim of the study was to develop a formula to estimate the stature and sex of an individual using foot and shoe dimensions. To this aim the stature, right and left shoe sizes, and maximum and minimum feet length and width measurements of a target group of 569 individuals were taken.
The group was composed of 294 males and 275 females. The highest correlation coefficient was found in length measurements.
A notable difference between males and females existed with regard to both right and left foot and shoe length and width averages and shoe sizes (
p
<
0.001).
Among the group, a significant correlation was found in regard to stature and right shoe length (
r=
0.591,
p<
0.001), with the correlation between stature and right foot length and stature and right shoe length being 0.579 (
p<
0.001); as for the female group, there was a significant correlation between stature and right foot length and stature and right shoe length (
r=
0.460,
p<
0.001).
In application of the formula, if sex is lower than 0.50, the shoe belongs to a male, if higher, then to female. The formula which was obtained in regression analysis in order to estimate the stature when the measurements of shoe and foot were known.
It was understood that foot and shoe sizes are a criteria to estimate the stature of a person that there was a strong relationship between foot and shoe length and width and that these can be used to aid estimation.
It was found that in sex estimate, foot and shoe lengths are better in helping the estimate than width measurements, and that the use of shoe measurements rather than bare foot measurements are better to obtain meaningful results.
http://www.fsijournal.org/article/S0379-0738(04)00576-6/abstract