Jesus said this to crowds who had been worn out by religious leaders piling heavy rules and burdens on them. So this invitation lands directly on anyone hurt in a religious place.
Come to me, all of you who are weary and carry heavy burdens, and I will give you rest. Notice He does not defend the people who wore you out. He offers Himself as the alternative to them.
Then He tells you what He is actually like: humble and gentle at heart. That is the opposite of the harshness that hurt you. The leaders may have been proud and heavy-handed. Jesus is not, and He is careful to say so.
My yoke is easy, my burden is light. The people who hurt you may have made faith feel crushing. Jesus says walking with Him is not supposed to feel like that. Do not let what they did define what He is like.