Australian English overwhelmingly prefers –our spellings to –or (most style guides specify –our and Pages for Mac’s Australian English autocorrect changed Victor Harbor to Harbour), but usage has swung back and forth over the years. Around the time Victor Harbor was officially named, –or spellings were in favour, so that official spelling was given and has been retained; likewise with the Australian Labor Party. (For more about these spellings, see Wikipedia in general, about the Labor Party. Basically, the –or spellings reflect the Latin original and the –our spellings reflect French –eur.)
Note also that Shellharbour is now one word, while Victor Harbor remains two. I guess running four syllables together is just too clunky.