First thing that crossed my mind when I saw that beautiful castle.
(warning: nerding out on history facts)
It's not a castle, it's a wall that surrounds the entire inner city. Fun fact: the primary purpose of the wall wasn't to defend against hostile invasion, but to keep the local peasants out. Visby was one of the richest cities in the Baltic during the iron age/ middle ages, so the potato munching plebs wanted a slice of the cake.
When the Danish king Valdemar Atterdag landed on the Gotland shore, the peasants went to Visby to cooperate against the invaders. But the city closed the gates and the Danish forces slaughtered the peasants against the city wall. The city then opened the gate and welcomed Atterdag into Visby. For hundreds of years this was covered up and people thought that it was a hostile takeover of the city rather than a peaceful agreement.
The wall has also been used to defend against foreign invasion, though. Visby's been attacked by Germans a couple of time IIRC, and of course local peasants. One of the guard towers still have a cannonball stuck in a porthole from one of the assaults.
Eventually Sweden invaded Gotland and took it from the Danish. Before I moved here I thought it was Swedish before the Danish takeover, but it was actually independent at that point.
In addition to the city wall, there's fifteen known church ruins (and probably more undiscovered ones) in the city.
I took some photos of the local medieval defense wall (Visby)