Removing the switch from Maul Hasbro saber

[Gilgamesh]

Anyone else have trouble removing the knob switch from the Maul fixed blade Hasbro saber? It has this little ridiculously tiny pin that you have to push in with a punch in order to get the thing off. If you can't take the switch off, you can't open up the hilt. Initially I just tried to use a nail, and when that failed I bought the appropriate sized punch at Home Depot, and still can't get the darn thing off. All I've managed to do is scratch the thing. Am I doing something wrong here?

[Lord Dottore Matto]

When I used to do them, I just pushed them in with the nailset.