At around 1128 (during the medieval times), the owner of the castle founded an abbey within the outer bailey, the sole surviving parts of which are the main body of the church & the drains of the lavatories on the exterior wall.

The monks didn't stay here very long though, finding nearby Southwick more convenient. The Keep then was doubled in height and a chapel & chamber were added to its forebuilding around the middle of the century.

*These historical factoids were derived from the internet:)