bunionectomy recovery

I had abunionectomy a little more than 4 weeks ago.
Even though my foot was broken as part of the procedures, my doctor didn’t place me in a cast.
I have been using a removable velcro boot.
At first I kept the boot on all the time, but it’s uncomfortable.
For the last 2 weeks I have removed it in order to sleep and when I am going to be sitting for long periods of time.
Do you think removing the boot as I have described will delay healing? I’m afraid to ask my doctor about this since he always threatens to put me in a cast.
At my 4 week check-up, he said I was healing well and the x-rays looked good.