All of us have some things in our life for which we needed to ask forgiveness. We have hurt our friends, family, co-workers, etc... However, in the same vein we have all been harmed at one time or another and to one degree or another. While the extent of our harm varies we all share in that we are not perfect. We have both harmed, and been harmed. Hence the need for forgiveness.
It occurs to me that forgiveness is not for the person who has done the trespass. Forgiveness is for the person who has been harmed. Follow me for a moment here.
Have you ever been angry with someone for something they did to you? Did they know you were mad at them? If so, did they care? If they cared about your relationship perhaps they did, but what if they didn't. You could be angry and they would never know. You would sit up at night and they would sleep. You could let the harm shape the rest of your life, ruining relationships. You could even hate them. It won't matter. They'll sleep like a baby.
The only exception I see to this is when a person feeling guilt requests forgiveness. In that case however, the person asking for forgiveness often feels as thought they have harmed themselves while harming you. In the end whether you forgive them they must forgive themselves.
Do yourself a favor. Forgive the person who has hurt you the most because they aren't even thinking about you. Forgive yourself too. You deserve it.