Augustine, de Con. Evan., 4, 5: We must take care that this saying of the Lord appear not to be contrary to that where He says, "He who is not with Me is against Me." [Luk 11:23] Or will any one say that the difference lies in that here He says to His disciples, "For he that is not against you is on your part," but in the other He speaks of Himself, "He who is not with Me is against Me?" As if indeed it were possible b that he who is joined to Christ’s disciples, who are as His members, should not be with Him.
How if it were so, could it be true that "he that receiveth you receiveth Me?" [Mat 10:40] Or how is he not against Him who is against His disciples? Where then will be that saying, "He who despiseth you, despiseth Me? [Luk 10:16] But surely what is implied is that a man is not with Him in as far as he is against Him, and is not against Him in as far as he is with Him.
For instance, he who worked miracles in the name of Christ, and yet did not join himself to the body of His disciples, in as far as he worked the miracles in His name, was with them, and was not against them; again, in that he did not join their society, he was not with them, and was against them.
Be because they forbade his doing that in which he was with them, the Lord said unto them, "Forbid him not:" for they ought to have forbidden his being without their society, and thus to have persuaded him of the unity of the Church, but they should not have forbidden that in which he was with them, that is, his commendation of the name of their Lord and Master by the expulsion of devils.
Thus the Church Catholic does not disapprove in heretics the sacraments, which are common, but she blames their division, or some opinion of theirs adverse to peace and to truth; for in this they are against us.
BTFO by a dude that lived 1500 years ago OP.