Ben,
There is no "one size fits all" answer to this. Most companies will have a policy that governs such situations. So the correct answer is: Know your company policy about receiving/accepting gifts and follow it. Even if that means rejecting a gift from a customer: "Look... I really appreciate your gift and I'd love to accept it, but unfortunately, my company forbids it. I'm sorry.".
If you need a general answer: Small gifts (say up to about $100) should be OK to accept. Above that... 'just say no'.