This is probably a result of your frustration with yourself.
As you perceive others getting annoyed with you, you get frustrated and your anxiety increases. When this occurs, it becomes harder to concentrate and focus on the task at hand, and that distraction makes it more difficult to perform your rituals correctly.
This is especially true for individuals who are perfectionists. If you fear that people will judge you if you do not do things perfectly, you may interpret people’s annoyance at your rituals as actually being annoyance at your not doing things perfectly.
This can unfortunately reinforce the idea that people would accept you more if you did everything perfectly.
As you put more pressure on yourself to conform to these expectations, then you will have a harder time actually satisfying them, and the cycle will just get worse and worse.