
Several reasons. One is just like genuine racism (see #2). Another is this genuine fear of not being in power. This is pretty common in a lot of societies where there is an unequal power dynamic. There’s this fear that if the oppressed group takes over they will treat the privileged group badly (the driving ideologies of Rhodesia, Apartheid South Africa, and the Confederacy come to mind).
And I think part of it is also just like white people generally aren’t used to being a minority, so the idea of not being the default is just kind of inherently scary. There’s also an ethnonationalism element to it where they feel if the country is no longer mostly their demographic it will no longer be their country.
If white people largely changed the makeup of a population, "peacefully" settled, started creating ethnic enclaves inside of other countries, and created their own quasi governments you would almost certainly call it colonialism and regard it as evil and imperialistic, especially if it took place in Africa.
The reason the afrikaners and rhodesians were disproportionately influential was because they held all the government and economic power and used it to take resources from native Africans. That is what distinguishes immigration from settler colonialism. Immigrants do not hold disproportionate power and exist alongside the already present population.
Yeah institutions they built by taking land from the people already present. I’m not here to argue about the history of South Africa, I’m pointing out that Indian immigrants don’t control the government of Canada to the exclusion of other groups and never will that’s why percentages don’t matter.