Two axes of abuse
WebSocket abuse comes in two shapes: too many connections from one source, or too many messages on one connection. The mitigations are different. Connection limits cap server resource exhaustion (file descriptors, memory). Message rate limits cap noisy/malicious clients without dropping their connection.
Per-IP connection cap
Track active connections per source IP. Reject new connections beyond the cap with code 4029 (an application convention; 429 is the closest HTTP analog). Be careful behind a proxy: websocket.client.host may show the proxy IP, not the real client. Use X-Forwarded-For if your reverse proxy sets it.
Per-connection message rate limit
Track timestamps of recent messages per WebSocket. If the count in the last second exceeds the limit, drop the message and send back a {type: 'error', code: 'rate_limited'}. Do not close the connection — that escalates and looks like a network failure. Just drop the abusive message and keep going.