There’s a few things you miss out on. You lose out on the spice pepper, cinnamon, and coffee. Perhaps the biggest loss would be that you would lose out on wheat and rice, but with corn, amaranth, wild rice, various nuts, and quinoa you can conceivable make good alternatives, it’ll just be more expensive.
You’d also lose out on a good portion of commercial fruits (apples, bananas, citrus, melons, mangos, coconuts) but you can balance that out with the local world plants like persimmon, pawpaw, all the berries, and maybe buy more expensive less common imported fruit like papaya, açaí, and soursop.