What is now the Library was built in 1872 and altered several times. It was originally used as a Conservatory for ornamental plants. In 1878 the front (north) wall was remodelled. In 1901 the Conservatory was converted to a Smoking Room for men after dinner. In 1933 the galvanised iron ceiling was removed and a new ceiling installed.
In more recent years the room was used as an office for the Governor's Aides-de-Camp, and then as a general office, but in 1985 the present Library was formed. The shelving and cupboards were constructed by the South Australian Department of Housing and Construction to match those in the Governor's Study. The timber is Queensland cedar.
Most of the books have been gifts to Governors.
It is also used for meetings, such as the weekly Executive Council meeting.