About non-vegetarians Lord Krishna in Bhagavad Gita clearly states that consuming non-vegetarian food by a seeker, traveler of spiritual path is not advisable. Non-vegetarian diet produces tamasic pravattis whereas a traveler of spiritual path should all the time remain sattvic both in eating habits and also thoughts. Invoking impure thoughts is prohibited on path of pure spirituality.
Consumption of vegetarian food is good not only for the body but also promotes mental peace. In fact, spiritual journey is meant for vegetarians and not non-vegetarians.
It is agreeable that non-vegetarians did not themselves indulge in killing of animals. But their indirect participation in consumption of non-vegetarian food contributes to passive karma which is detrimental to spiritual progress.
What if the meat came from an animal that was slaughtered in front of the infant animal! In such circumstances one unknowingly becomes a participant in the sin committed as we are cursed by the infant whose mother was killed in front of his eyes.
I distinctly remember an incidence of my childhood. I purchased a pair of leather sandals and whenever I wore them, I could distinctly hear voice of a crying infant buffalo. It was so much uncomfortable wearing those sandals that I just could not wear them even for five minutes. I then realized to make those sandals the hide of a mother was used who was killed in front of the infant. And the curse of the infant carried along with sandals.
I ultimately gave away those sandals that disturbed peace of my mind every single time.