His parents were of African descent and were Roman Catholics. He was educated at public and private schools and by tutors in Washington, New York City, Buffalo, and Detroit. In 1851 he withdrew from the Roman Catholic Church and entered the Protestant Episcopal Church. From 1852 to 1853, he was associate editor of The Voice of the Fugitive, a weekly paper, published at Windsor, Ontario. In 1854, he was principal of a public school in Buffalo.
He studied theology, and was ordained deacon on 17 June 1855 and presbyter on 2 January 1856. He was rector of St. Luke’s Church, New Haven, Connecticut, from 1856 until 1861, when he was sent to Haiti as a missionary. He served as consul for Liberia at Port-au-Prince from 1864 until 1874, in which year he was made missionary bishop of Haiti by the Protestant Episcopal Church.
In 1878 Holly went to England as a delegate to the Lambeth Conference. He received the degree of D.D. from Howard University, Washington, D.C., in 1874, and that of LL.D. from Liberia College, Monrovia, in 1882. He contributed to the Church, the Church Eclectic, and the African Methodist Church reviews.