The man, known only as H6 and described as a "close confidant" to Prince Andrew, brought an appeal against his initial ban but the decision was upheld by the court.
Judges were told the businessman was attempting to leverage Prince Andrew's influence.
Buckingham Palace declined to comment, saying they do not act for the prince, who is not a working royal.
