Ok, I can definaitly correct it, it needs about 4/5ths of the trim on the tail.
Once I get above the ground a few feet and hover should it stop drifting to the left? Because it needs full right trim for the "ailerons" when I'm only a few inches above the ground.
That is way too much trim IMHO. You will need to move your tail servo, rather than over-trimming it.
Did you carefully check your CCPM setup, including levelling the swash? I have about 10% of trim on my aileron – my BR lifts straight up with just a fraction of right stick, but once it is airborne at about eye level it sits level without hardly any input. A good setup will do this for you. Spend time on setting it up right and the BR will be a piece of cake to hover.
All the info to do the above is well presented at
MyCoolHeli.