Is it wrong not to give my goldfish flakes, just vegetables and bloodworms?

I have two goldfish( 1 fantail 1 black moor) and I only feed them peas, courgette, lettuce and bloodworms. The reason I don't feed them flakes is that my black moor had swimbladder and ever since I stopped with the flakes he has been so much better. I tried soaking the flakes but it made no difference. But I am wondering if I am giving them enough nutrients through what I am feeding them now.... I try to be good to my fish but don't always know the best way forward....