The actor—and eventual US ambassador and interior decorator—once had a whole cottage devoted to storing gifts from fans