perhaps it never was a "fragrant harbour"

Apparently Hong Kong (香港) might not mean "Fragrant" (香) "Harbour" (港) as often said,

but rather the "Incense" (香) exporting "Harbour" (港), though "exporting" in this case might only be up the Pearl River to China.

apparently either the timber used for incense was shipped out of here, or there were incense-manufacturing plants in the town, and incense was shipped out.