It took the Queer Eye star and his husband three years to build what they thought would be their “forever home”