Your motor will be big enough to reach hull speed. Due to the shape of the upturned stern on a driftboat it will never go fast. You have to drastically increase the horsepower to get above hull speed. That is why power boats have flat bottoms from the midpoint to the stern, for planing.
6hp that's the least I'd go...my ideal would be the 8hp yamaha. I had a 4hp yamaha and it was only great going down hill or in flat water...really tough going upstream with any kind of current and impossible in heavy flows.
I ran a 5hp honda long shaft on my 16LP for kicking around lakes and it was great. I would not want a bigger motor, or trying to run up a river. A bigger motor will just dig you a deeper hole as you will never get a drift boat on plane.