I've also had this happen to me tonnes of times.
It's sort of like, a sorrow and an anger that arises when you happen to fall away on account of sin and doubt.
For me, these events acted like disciplinary measures, reminding me that i shouldn't be prideful in anything even my faith since i backpedal and sin, making me no better if not worse than people who actively take part in these sins.
It also might come down to the way you can't consciously decide what your soul or heart wants to feel, so it continues looking for god even though you say you have enough of it and want to relapse.
All in all you're fighting the good fight since this is clearly a spiritual battle you're engaged in, but you're gonna start seeing some real hardships and trials if you keep clinging to your faith in God, which is why his rewards are so sweet.