Returns triage from photo and description

Returns triage is the process by which the ops team at this Shopify-based business assesses and responds to customer return claims. Each claim arrives with a photo and a free-text description from the customer. A team member picks it up, reviews the photo and description against the company's return rules and applicable consumer law, decides whether to approve, reject, or escalate to the founder, writes a response, and logs the outcome in a spreadsheet or email thread. The whole cycle takes around twelve minutes on average, and it happens several times a day on most working days, adding up to roughly 4,200 claims and 840 hours of manual work per year.

The process sits across a two-person ops team and the founder. About 85 to 90 percent of claims are straightforward and follow recognisable patterns that a trained eye can assess quickly. The remaining 10 to 15 percent involve ambiguous photo evidence, edge cases under consumer law, or situations that do not fit the standard rules, and those genuinely need human judgement. Right now, because the rules live mostly in people's heads, different team members can reach different conclusions on similar claims, which creates inconsistency in customer experience and some compliance exposure.

The process is anchored in Shopify, with supporting work done in custom scripts, email, and spreadsheets. The founder has the technical capability to build and maintain integrations via the Shopify API, and some automations are already running, which means the infrastructure for a more sophisticated system is closer than it might appear.

The recommended path is a Hybrid AI workflow with a human-in-the-loop gate, and the case for it is straightforward. The majority of claims are repetitive and rule-driven, and the photo-plus-description input is exactly the kind of multimodal, semi-structured content that a modern vision-capable language model handles well. A multimodal LLM such as GPT-4o or Claude 3 can review the image and the customer's description, check them against a documented set of return rules encoded in the prompt, and return a structured decision with a confidence score and a draft customer response, all in a few seconds. That handles the 85 to 90 percent of claims that are currently consuming most of the ops team's time without adding any value that requires a human to be involved.

The human gate is not a compromise on the automation vision; it is the architecturally correct design for this process. A 10 to 15 percent exception rate is real and meaningful. Consumer law deadlines and GDPR obligations mean that an incorrect automated decision on a borderline claim is not just a customer-experience problem; it can be a legal one. Routing low-confidence and genuinely ambiguous claims to the ops team with the AI's summary and draft response already pre-filled means the human is doing actual judgement work, not triage, and they can confirm or override in a fraction of the time the current manual process takes. That is the right division of labour.

The client's preferred tech stack makes this more achievable than it would be for most businesses at this scale. The Shopify API is well-documented and the founder can build integrations directly, which means there is no need for a middleware vendor or a large external development engagement. A lightweight Python or Node service calling an LLM API via HTTP fits naturally alongside the custom scripts already in place. The integration work is real but it is scoped and manageable, and the founder has the skills to own it.

The alternative paths are worth addressing honestly. A fully automated AI Workflow without a human gate is viable for standard cases but leaves the compliance-sensitive exceptions unguarded, which is the part of the process that matters most when something goes wrong. An autonomous AI Agent adds complexity and assumes a level of documented, clean data infrastructure that this process does not yet have. Traditional Code is feasible for the decision logic but cannot interpret photos, which is a hard technical ceiling for this specific process. RPA does not apply here at all: there is no structured, screen-based repetition for a bot to replicate. Education alone would reduce inconsistency but would not touch the volume burden or the key-person dependency, and staying manual is simply the current problem restated. The Hybrid path is the only one that addresses both the scale opportunity and the compliance obligation at the same time.

One honest tradeoff to name: this recommendation depends on the quality of the rule-documentation work that has to happen before any code is written. If the return rules are not written down clearly and completely, the AI will inherit the same inconsistency the manual process has today, just at higher speed. The first milestone is unglamorous but it is load-bearing. The ops team and founder should treat it as the most important week of the project.

The two closest contenders are the Hybrid path and a fully automated AI Workflow, and the gap between them comes down to one word: compliance. A pure AI Workflow could comfortably handle the 85 to 90 percent of claims that follow clear, rule-based patterns. Photo evidence of a damaged item, a description that matches the return policy criteria, a straightforward approval or rejection: the AI gets those right at scale. The problem is the 10 to 15 percent that don't fit the pattern. Ambiguous photos, customers close to a statutory deadline, edge cases that sit between two policy clauses: these require a human decision, not just a confident-looking probability score. Wiring an automated rejection to a consumer law obligation without a human in the loop is not a risk worth taking for a two-person ops team without a legal function to catch the fallout.

The Hybrid path wins precisely because it doesn't try to automate everything. It routes the straightforward majority through the AI classifier and reserves the human's attention for the cases that genuinely need it. That distinction, routine claims handled automatically and exceptions surfaced with a structured summary, is exactly what the ops team described as their goal. The fully autonomous AI Agent path was considered and scored lower not because the technology is inadequate but because the process doesn't yet have the clean, documented rules infrastructure that a safely autonomous agent requires. Building an agent on top of rules that currently live in people's heads would produce confident-looking wrong answers. The Hybrid path forces the rule-documentation step first, which makes the eventual system more reliable and easier to audit.

Traditional Code was a credible option given that the founder can build integrations and Shopify's API is well-documented. A rules engine would eliminate inconsistency for the textbook cases. But the moment a customer submits a blurry photo of a product with a disputed defect, deterministic code has nothing useful to say. The photo interpretation requirement alone rules out a pure rules-based approach as a complete solution, and layering a vision model on top of it brings you back to the Hybrid framing anyway. RPA was never a serious candidate here: it's built for structured, screen-based repetition, and this process leads with unstructured image and text inputs that RPA cannot interpret without an AI layer doing the heavy lifting.

The first thing to build isn't code, it's a decision matrix. Every approval criterion, every rejection criterion, and every escalation trigger that currently lives in the ops team's heads needs to be written down in a structured format before any AI model is trained or prompted against it. This step takes one to two weeks of focused workshops with the ops team and founder, and the output is a shared document that defines what a valid claim looks like, what photo evidence is considered sufficient, and what conditions automatically route to human review. This document has immediate value independent of any automation: it reduces inconsistency overnight just by existing. It is also the single most important input to the AI system, and if it's skipped or rushed, the automation will inherit the same inconsistency it was built to fix.

With the rules documented, the founder builds a lightweight service, most naturally in Python or Node, that sits alongside the custom scripts already running in the stack. When a return claim comes in through Shopify, the service pulls the claim data including the submitted photo and description via the Shopify API and passes them to a multimodal LLM. GPT-4o and Claude 3.5 Sonnet are both strong candidates here; both accept image and text inputs and return structured outputs, and both have commercial API tiers with data processing agreements available. The model receives a prompt built from the documented decision rules and returns a structured JSON response: a classification of approve, reject, or escalate, a confidence score, a brief reasoning summary, and a draft customer response. The founder validates this against a sample of historical claims before anything touches live customers.

The human-in-the-loop gate doesn't need to be complex infrastructure. A structured email digest, a simple internal web form, or a dedicated Slack channel with formatted summaries can serve as the review interface for escalated and low-confidence claims. The AI's summary and draft response are pre-filled, and the ops team member confirms, edits or overrides. The key design principle is that the human is making a real decision, not approving a suggestion on autopilot. Claims below a defined confidence threshold are always routed here regardless of the classification, and all claims touching statutory deadline windows should be treated as mandatory human review until the system has demonstrated consistent accuracy on those cases.

The final stage is a two to three week parallel run where the hybrid system processes claims alongside the existing manual workflow. Every case where the AI and the human disagree is logged and reviewed. Those disagreements are the calibration data: they reveal where the prompts need tightening, where the decision rules were underspecified, and where the confidence threshold needs adjusting. Once the disagreement rate drops to an acceptable level and the team is confident in the system's behaviour on edge cases, the parallel manual process is retired for standard claims. A lightweight monitoring dashboard, a spreadsheet or a simple Metabase instance connected to the log data, tracks approval rate, escalation rate, response time and the human override rate on a monthly basis. A rising override rate is the early warning signal that the model's behaviour has drifted from the current policy and needs a prompt update.