A loud noise or sudden movement can wake him up, and you may feel a kick or punch as a result.

http://www.babycenter.com/404_will-my-lack-of-sleep-harm-my-baby_7601.bc