The area around ports where cargo is handled is securely fenced. Security guards make sure that only those people who have permission to visit a port can go in. There are usually security cameras and precautions for keeping unwanted people out.
Security is very important in ports.
Wharfs are very dangerous places for people to be wandering around. There could be many large transport machines and cranes at work. Large conveyors can also be dangerous.