Popes have not always worn white.
Until Pope Saint Pius V (1566–1572), the bishops of Rome wore Cardinal robes of scarlet red, but to distinguish themselves from the other Cardinals, their robes had white ermine trim.
Pius V had been a Dominican friar before he was elevated to cardinal, and Dominicans wear all-white habits. When he became pope, Pius V retained his white wardrobe and modified it for his new assignment as pope.
Since that time, all popes have worn an all-white cassock with white zucchetto (skullcap). Had Pius been a Franciscan friar instead of a Dominican, his brown habit would have resulted in the popes wearing a brown cassock.
Cardinals, by the way, wear scarlet red to represent their willingness to die for the pope.