Serious Aggression towards owner

HELP!

My neutered male JRT (Cosmo) is now 3 years old and has developed a serious aggression towards me. I adopted him when he was an year old from my brother. My brother never had him neutered or had him properly trained and let him pretty much have the run of the house. Since adopting him, I've invested quite a bit of money with a personal trainer. He has always shown aggression when he doesn't get his way. It started with not wanting to go in his crate and has gotten progressively worse.

Each time I try to leave the house, he growls and attacks me. He does this when he has a chew bone near him, when he's near or in his bed, on my couch. The only time I feel safe, is when I allow him to sleep with me. Then he's a complete angel.

I have tried everything I can...I talked to my Vet and does not recommend medication. I spend a lot of time with him & I exercise him at least 2 hours each day.

I feel he may be better suited with a male owner and think my last option is to try and have him adopted. But I don't know how to even begin this process or if I should. I'm very attached to him and love him dearly but have developed a fear towards him. The last time he bit me, he puncture my skin pretty deep that I bled.

He does not behave like this with anyone but me. He's very gently with my friends and family. It's gotten to a point where noone really believes that he behaves like this.