The castle is situated all the way down to the sea where the main building, the beautiful wings, and the pavilion are reflected creating a unique manor landscape. This large Renaissance construction was built by Christian IV in the 1630s as a replacement for an old hillfort (Kærstrup). The terrain by the coast is relatively flat and rises steadily toward the highest point of the island, Bregninge Kirkebakke (Bregninge Church Hill) at 73.75 m a.s.l., crowned by Bregninge Church which is famous for its magnificent views. Although military considerations were largely ignored when the castle was built, it is actually situated on the tip of a low peninsula and is thereby protected by water on three sides. At the same time, Valdemars Slot presents an unmistakable and dominant position by the approach to Svendborg Sound. This was an important strategic position at a time when seafaring was far swifter and more important than land traffic.
The hillfort Kærstrup is still visible in the terrain 1.75 km southwest of the current castle. Kærstrup can be traced as far back as 1298 when it was a considerable fortification. Today, you can only see the protected main embankment, but on modern elevation charts you can clearly see the other embankments that have been ploughed over: a front embankment, a farm embankment, and an additional embankment. You can also see all of these features in Resen’s illustration of the installation from the 1600s.