Requesting refund from contractor, he refuses to communicate in writing

Husband and I hired a contractor to tear down and replace a two story deck, as well as add some hardscaping. Several mistakes and shortcuts were made in the construction of the deck, and we are not comfortable continuing the project with them. We paid 33k in advance with another 20k to be paid at completion. I have repeatedly asked for a breakdown of labor and materials for what has already been done, and for a refund of the excess. I’m not trying to screw the guy over, I just want the extra that we paid refunded. Every time I contact him, he immediately calls me even though I have said that I prefer everything be in writing from now on. What are my next steps here? I really don’t want to be stuck with the framework of a deck and having paid out the nose for it.