He was the second son of Infanta Maria Teresa, daughter of King Alfonso XII of Spain and her husband (and cousin) Prince Ferdinand of Bavaria. In 1914, Prince Ferdinand had renounced his claim to the Bavarian throne and was granted the title of Infante of Spain. Infante José married María de la Asunción Solange de Messía y de Lesseps, Condesa de Odiel, and had four children.
A talented musician, he studied piano with Alfred Cortot, one of the most sublime interpreters of his time, and musicology at the Sorbonne. However, he was not encouraged to pursue a musical career and instead became an engineer in the army, before being exiled in France with the rest of the Spanish Royal Family. He continued to occupy advisory positions in the arts until his death.