Closed displaced 1st metatarsal

I broke my 1st metatarsal on my left foot almost three weeks ago. My doctor (general practice) took x-rays and felt that I might require surgery and suggested I go to Florida (I live in the Bahamas) and have a Orthopaedic Surgeon look at it. The Orthopaedic Surgeon looked at the x-rays and said all I needed was a cast and he would like to see x-rays of my foot in two weeks. I was concerned about how far the bone was pushed down but he said it would be OK. I went back to the islands. I had x-rays taken again yesterday. From the top it looks ligned up. From the side view the the first half of the bone looks in its normal position but the second half is way down and slightly underneath(the fracture is angled back slightly) with only about half to a third of the bones appear to be touching each other.
My doctor(in the islands) was suprised that it was left like that and not realigned and suggested I get a second opinion from another Ortopedic(in the US) which I am doing tomorow. Today I went back to the Orthopedic Surgeon to share my concerns. He didn’t seem to interested in discussing anything with me but said I could get a second opinion. The only information I can find online is from the American Academy of Orthopedic Surgeons. When they discuss toe and forefoot fractures ‘the bone ends of a displaced fracture must be realigned and the bone kept immobile until healing takes place’. Am I(and my doctor) being overely concerned? How much can the bones be displaced before I should be concerned? I want this to heal quickley and I don’t want to have any problems in the future.
Thank youClay