Despite confidence holding steady, tariffs, higher prices and inflation are still weighing on the minds of consumers.