The castle was built in 1598 near Cape Higuer [or Higer] to ward off pirates. It had a rectangular plan, and the seaward side contained a battery with capacity for five cannon. The rest of the castle consists of two rectangular constructions at right angles to each other, joined on one of the shorter sides to a taller quadrangular building.
In 1755 King Ferdinand VI ordered the construction "...outside the walls of Fuenterrabia of a simple store to which shall be transported the gunpowder currently kept in the stores of said town...". The site of the powder magazine was to be St. Elmo's Castle "... since it is the most suitable site for this purpose and because of the dual benefit... of reestablishing said post which the vagaries of time and a fire have left almost in ruins, its conservation being all the more advisable, given that its battery defends the anchorage in that inlet..."
The only addition to the existing structure was a wall whose purpose was to separate the two curtains of the tower - which formed part of the perimeter of the castle - from the outside. The wall had two sentry boxes and a small crenellated section.
The entrance, located in the southern wall, gave access to a narrow passage which led to the original gate. On the lintel, beneath the royal coat of arms, there is an engraving which reads "PHILIPVS II HISP INDIAR Q REX / AD REPRIMENDA PIRRATARVM I LATROCINIA / HOC SANTERMI CASTELLUM EXTTUERE MANDANIT / ANO DOM -MDXCVIIII / SIENDO DON JVAN VELAZQUEZ CAPN GN DE ESTA PROA". The castle has been a private residence for many years.