Why were people who did not invest in the stock market impoverished by its fall?

Because most people did. They might not have been affected, but their employers, grocers, neighbors, and mostly everyone around them were. So because their employers were crippled, so were their pay-checks. The grocer has no money to buy products, so they couldn't get food. Couldn't even ask for help from friends, or neighbors.