Cats really don’t like having air blown at them, particularly in their faces. In fact, if your cat is getting too rough while playing with you, blowing in his face is a good way to get him to back off. Why, since a puff of air is harmless? Apparently cats associate blowing with hissing—their own sign to the world that a serious threat is near.
If you are close to a hissing cat, you will experience not only the distinctive sound, but also a jet of air being expelled from the cat. So, when you blow air at your cat, you are (so the cat believes) hissing at him, and he will respond accordingly.