Why didn't northern bankers, merchant ,mill owners and unskilled workers want to end slavery?

They didn't want to end slavery, because slavery made their lives that much easier.

exg bankers had shark tons of money, so they bought slaves and had the slaves clean, tend to their needs, etc.

merchants could have slaves carry their stock. 

mill owners needed alot of man power back in the day to get things going to produce a fair amount of output into the market, and paying good looking muscular men was much more expensive than one time buying a few slaves and having them work hard.

unskilled workers might have wanted slaves because then they didn't need to learn the given skill, they could have a slave do it FOR them.