Robert Kerr was born in Wrington, Somerset, the younger son of Sir Thomas Kerr (or Carr) of Ferniehurst, Scotland. In 1606, he happened to break his leg at a tilting match, at which the king was present. According to Thomas Howard, 1st Earl of Suffolk, the king instantly fell in love with the young man, even helping nurse him back to health all the while teaching him Latin. Carr continued as the King's favourite, and in 1613 the King made him Earl of Somerset, Treasurer of Scotland, and Lord Chamberlain.
In 1615 Carr and James fell out as the King took up with a new favourite, George Villiers, 1st Duke of Buckingham. Later that year Carr was convicted for implication in a murder, and confined to the Tower of London for several years.