If you want specific players to never join your server, you can use Valheim's banlist feature to be able to ban them forever. No more griefers or unwanted guests who got the password for your server.
Make sure to get each player's Steam ID so you can add them to the banlist.
1. Open the File Manager
2. Navigate to /.config/unity3d/IronGate/Valheim/
3. Right click banlist.txt
and click Edit
4. Add each Steam ID, one per line.
5. Save the file and Restart the server.