The point was originally a small tidal island, Bennelong Island, that largely consisted of rocks with a small beach on the western side. The island was located on the tip of the eastern arm of Sydney Cove and was apparently separated from the mainland at high tide. For a brief period in 1788, this relatively isolated protrusion into Port Jackson (Sydney's natural harbour) was called Cattle Point as it was used to confine the few cattle and horses that had been brought from Cape Town by Governor Phillip with the First Fleet.
The area at that time was also strewn with discarded oyster shells from thousands of years of gathering by the local aboriginal women. Those shells were regathered by the newly arrived convict women and burnt to make lime for cement mortar. The point was called Limeburners' Point for that reason, though those shells only furnished enough lime to make a single building, the two-storey government house.