Contact to children am i being fair?

Hi do you think i am being fair? My ex has contact with our children three evenings a week and every other weekend one night. He also sees them on tuesdays. He wants to take me to court for not allowing him 2 nights every other weekends i have denied this cus he works sat morns so therefore will not be looking after the children he plans to disrupt them and leave them wiv other people. I would agree to this if he was caring for the children. He does not even look after them when he has them every other sat leaves them wiv his mum which annoys me cus he wants more contact but he is not actually caring for them. Also he has wen eva he wants in the holidays which we can agree on mostly only cus i contact him about having them in the holidays he seems to think its ok to call on the day and ask to have them i think its best for the children to no in advance. I have got recorded phonecalls of him being abusive and threatening hence the reason i do not communicate by phone to him. I no he would never hurt the children though. However he says things to them which the children do not need to no. He says alot of bad things about me and my partner and makes up reasons why we split up. I have to pick up the pieces and take the crap when my children get home it really does get frustrating however i would never cut contact.