Perception, then policy

By Phoebe Maio & Curtis Bollinger

Customers send pictures. Across Gladly, 11.9% of conversations now include an image or a PDF, 3.3 times as many as a year ago. A leaking bottle of foundation. A receipt from an in-store purchase. An eyeglasses prescription. The wrong item sitting in the box it shipped in.

Most teams handle this with a model for each kind of picture. They build one for damage, one for receipts, one for prescriptions. We built one tool that reads them all, called AI Vision, and handed it to the AI already working the conversation. That only works because the tool has a single job, describing what's in the picture. What to do about it stays with your company.

You can only afford so many specialists

People usually do this one of two ways, and both run into the same problem.

One is to build it by hand. Your team picks a problem, damage claims say, and writes code that reads photos of damaged products. A few weeks later it works. The next kind of photo takes another few weeks…and now there are two things to maintain.

The other is to train a model on your own photos. You gather up thousands of pictures customers have already sent, pay people to tag what's in each one, and train a model until it's good at understanding your catalog. Then the products change, or customers find a new way to break them, and you do it again.

Both work. Your company can name the photos it sees most, damage and receipts and prescriptions, and build for those. It can probably name thirty more than that. But none of those are worth a few weeks on its own, so nobody builds them, and the AI goes blind the moment a customer sends one picture.

A camera, not a judge

When you train a vision model for something like damage claims, you teach it two things at once. To see, and to judge.

"Severe damage" is an ambiguous line somebody drew. "This looks like a stock photo" becomes "reject this claim" because someone decided to wire those two together. Your company's refund policy ends up inside the model, where nobody can read it and nobody can change it without training a new one.

Refund policy can change frequently. What a bottle looks like when it leaks does not.

So we built AI Vision as a single tool the AI calls whenever a picture shows up, called processAttachment. It reports, and stops. It will tell you a bottle is leaking, that the damage looks moderate for that kind of product, that the picture looks more like a photo from your site rather than something taken on a phone. It will not tell you to refund anyone.

We could have gone the other way and put fraud rules straight into the tool, so a photo that looked too staged blocked a claim by itself. We intentionally decided not to. The tool reports what it sees and never decides what that means.

What's in a picture is the same for every company. What to do about it is not. Keep those apart and the same tool can work a return, a warranty claim, and a prescription refill, because none of the three has to agree about what a bad photo means.

You write the form, the AI fills it in

Think of it as a blank form.

Before any of this happens, someone at your company writes down the questions they want answered about a picture. Is anything leaking? What shade is it? Which of our products is this? Each question is one plain sentence, and that sentence is the whole instruction the AI gets. Adding another one takes about as long as typing, “Is anything leaking?” No data to label, no training run to wait for, no project to staff.

Then a customer sends a photo, processAttachment runs, and the form comes back filled in.

Damage is the question most companies start with, so that one comes ready-made from us along with a few others. Everything else you write yourself, which is the whole point. Nobody at Gladly has to guess what your company cares about.

We left one thing out on purpose. The tool never tries to work out what it's looking at. In an early design we tested sorting receipts from product photos first and picking a path from there. But when you've just asked a customer for a photo of the damage, you already know what's in it. So the tool asks your questions directly. One less model in the path, one less thing to be wrong.

Damage means something different for every company

The same rule made it clear how AI Vision should approach damaged products.

Our first version judged damage against a fixed checklist, written with cosmetics in mind. It's the right checklist for a bottle of foundation. But when we pointed it at a dining chair or a pair of glasses and it had almost nothing to say.

So damage now comes back as the two things everyone can use: (1) how bad it is and (2) what's wrong with it…and then you add your own questions on top. Is it leaking, for the bottle. Is a leg broken, for the chair. And how bad depends on the product. A hairline flaw on a lens is severe. A scuff on a cast-iron pan is not.

A fixed checklist would have to anticipate every kind of product on earth. Questions you write yourself don't.

What one answer looks like

Here's an example. A customer says her foundation arrived leaking, and sends a photo of the bottle.

Here's what came back:

• The photo is sharp enough to trust.

• The product is a NYX Total Control Drop Foundation.

• Something is leaking from the cap, and it reads as moderate rather than ruined.

• The shade is beige.

• What the customer said happened matches what the picture shows.

Every line of that is an observation. The decision comes next, and it belongs to your company. Reship and credit, or hand it to a person, or ask for a clearer photo. The tool said what it saw, but the choice remains yours.

Why this keeps working

A new kind of picture is now a few sentences somebody writes, and it's working in days. The tool that reads damage photos today reads warranty receipts tomorrow, because somebody wrote down the questions they wanted answered.

It also makes what the AI sees something it can act on. Answers come back as observations rather than verdicts, so the AI works with them the way it works with an order lookup. Vision stops being a system off to the side. More conversations with pictures in them finish on their own. The ones that still need a person arrive with the AI's read attached, so that person spends their time on the customer instead of squinting at a photo.

Teach the tool to see. Leave the judging where you can read it, argue with it, and change it.