Before being sworn in by Spain's King Felipe VI on Saturday, Sanchez vowed to tackle the "social emergencies" experienced by Spaniards after years of austerity measures.
All of his allies in the no-confidence motion stressed their vote against Rajoy was not a blank cheque for Sanchez.
Rajoy's no-confidence vote in parliament was prompted by a corruption trial that has involved multiple members of his center-right party, and his departure puts Spain into a precarious political landscape, according to NBC News.
But his majority - the smallest for a Spanish government since the return to democracy following Francisco Franco's death in 1975, makes it unclear how long his administration can last.
His ousting of Mr Rajoy, a 63-year-old EU-friendly veteran politician who had been in power since 2011 and was present during the oath, comes at a time of political instability in Europe as Italy brings in a new eurosceptic anti-establishment government.
He has already committed to respecting a budget passed by Rajoy, and the fragmented parliament means Sanchez will find it hard to row back on structural reforms passed by his predecessor, including new labour laws and cuts in healthcare and education.
Mr Rajoy attended the swearing in ceremony, which was held in the Palace of Zarzuela in Madrid.
The forming of a Catalan government will automatically end the extraordinary takeover by Spain's central powers of the region as part of its crackdown following a failed declaration of independence by Catalonia in October.
"Pedro Sanchez, let us talk, take risks, both you and I, let us sit down at a table and talk, government to government."
He will head up a minority government with the support of a hodgepodge of disparate parties like far-left Podemos and Catalan separatists.
After leading the Socialists to two crushing general election defeats in 2015 and 2016, Sanchez was forced to resign by the party apparatus.
A member of parliament for Catalan pro-independence party Esquerra Republicana de Catalunya (ERC), Joan Tarda, made it clear that his support for Mr Sanchez was motivated by his views on the former PM, rather than the current one.
Sanchez will only be able to implement policy initiatives "that allow him to obtain an easy majority" in parliament, said Fernando Vallespin, political scientist at the Autonomous University of Madrid.