Rejected petition Mother & Child - the right to challenge/refuse contact with father

It seems these days men can do whatever they like and expect to be able to have contact with their child.
My husband left when I was 14 weeks pregnant saying he didn't want a baby; it wasn't part of his long term plan; he left after I saw texts off another woman, he left me in a mountain if debt which he refuses to contribute to. No contact from him until after giving birth he applied for 50% residency and contact. He refuses to pay child support so CSA have had to obtain AOE. My son becomes very emotionally stressed during his visits. My son should not have to go through this because of the choices made by my husband. My son cannot speak for himself. I should be able to say on his behalf he doesn't want this; he doesn't want his life disrupting; he doesn't want to be influenced by such a man or have contact forced upon him. As his mother I should have the right to insist that this does not happen until my child can verbalise this is what he wants.