wipe the smile off (one's) face
To stop smiling or appearing pleased. Often used in an aggressive imperative addressed to one who appears to be enjoying the frustration or annoyance they are causing the speaker. Wipe that smile off your face, you little punk. You're going to pay for that! He can gloat all he wants, but this subpoena will wipe the smile off his face.
wipe the/that grin off (one's) face
To stop smiling or appearing pleased; to do something that makes someone stop feeling happy, smug, or superior (about something). He can gloat all he wants, but this subpoena will wipe the grin off his face. Wipe that grin off your face, you little punk. You're going to pay for that!