The rich goldfields of Victoria attracted thousands of migrants and prospectors from all corners of the globe, but they were not rich for everyone.