is growling ever acceptable?

Last week, my family adopted a 1 year old JRT. He had been in two different homes in his first year being given up because of allergies of the owners. He took to my wife and 3 children the moment he went through our door. He is always to be found in their company and was very comforatble around them from day one. However, his behavior toward me is a different story. The first day home, he tolerated me, but from then on, he growls whenever I come near. Over the last couple of days, he has become more emboldened in his growling and sometimes seeks me out in the house just to growl at me. I asked a person who knew the previous owners if she thought there might have been mistreatment from a male sometime in his past and she thought it unlikely. If I try to approach him, he growls and runs away. He has never attempted to bite me or anyone else. He also won't accept food from me and if I try to walk him, he'll lead me in a circle back to the house as soon as I allow him to.