How to Test an AI Logistics Agent in a Real Demo

Key Takeaways

Test an AI logistics agent by running one real shipment from order to payment and breaking it on purpose at every stage.
Seven failures cover the lifecycle: a missing field, a silent carrier, a rate over the limit, an unconfirmed pickup, a late load, a withheld POD and an invoice mismatch.
Let the failures build on each other, so the invoice is checked against the rate approved earlier in the same run.
Score each stage 0 to 2 on an AI logistics agent demo scorecard, with a few outcomes treated as automatic fails.
Wilson, by Cartage, is an AI logistics agent that executes freight work from quote to payment under the company's rules and approval limits.

How do you test an AI logistics agent in a demo?

Run one real shipment from order to payment and break it on purpose at every stage. The injected failures show whether the agent executes the work under your rules, and a scorecard turns what you saw into a result you can compare.

A clean run proves the interface works, not that the agent can move freight. An AI logistics agent is software that executes freight work rather than reporting on it (see what an AI logistics agent actually does), so the test has to ask it to execute.

Testing single failures is useful, and making the freight AI fail in the demo explains why. A full-lifecycle script adds what isolated tests miss: each failure lands on a shipment that already carries the consequences of the last one. That is where hand-offs between steps tend to break.

The script: one shipment, seven stages, seven failures

Agree on the script with the vendor in advance, but don't say when each failure will fire. Use one shipment your team moved recently and keep it moving the whole way through.

Intake: a missing field. Remove the delivery appointment or the consignee contact. Watch whether the agent flags the gap and asks, or fills it with a guess.
Quote: a carrier that never answers. One approved carrier ignores the quote request. Watch for the follow-up and how the comparison proceeds without that reply.
Rate: over the approval limit. Set the limit so the best quote lands above it. Watch for negotiation against your target, then a hand-off to the approver.
Pickup: no confirmation. The carrier doesn't confirm the window. Watch who the agent contacts and when it escalates instead of assuming the truck is coming.
Transit: a late load. Report a delay or let tracking go stale. Watch for a confirmed new ETA and reason, and updates to the people your rules name.
Delivery: a withheld POD. The carrier holds the proof of delivery. Watch whether the load stays open and how the POD is chased.
Invoice: a mismatch. Send an invoice above the booked rate with an extra charge. Watch whether payment is held when the charge isn't supported.

How to stage the failures so they test real behavior

Someone on your side has to play the carrier, and the failures have to stay connected. Use an inbox or phone line you control for the carrier role, so you can stay silent, reply late or send the inflated invoice when the script calls for it.

For each stage, write down the trigger, what you'll do as the carrier and the response that counts as correct under your rules. If you haven't decided whether a missing appointment should block booking, you can't score whether the agent blocked it.

Don't reset between stages. The rate approved at stage 3 is what the invoice at stage 7 must match, and the delay from stage 5 should be on record when the carrier bills detention. Ask the vendor to stay off the keyboard except where your rules call for a person; every click from their side is a step your coordinators would take.

How should you judge what the agent did at each failure?

Judge each failure by the action taken, the rule behind it and the record it left, not by how the screen looked.

Some outcomes fail the run regardless of the total:

booking a carrier your company hasn't approved;
inventing a missing value instead of asking;
going past an approval limit without a person;
closing the load without a POD on file;
paying an unexplained charge.

A stop is not a failure. At the rate stage, the right result is a clean hand-off: the approver sees the quotes, target and gap, decides, and the workflow continues without anyone rebuilding the shipment.

Close with one question: which steps did a person do, and which of those did your rules require? The difference is work the agent left behind. For buying questions beyond the demo, see how to evaluate an AI logistics agent.

The AI logistics agent demo scorecard

Score each row 0, 1 or 2, for a maximum of 16. Any automatic fail overrides the total.

Stage	0 points	1 point	2 points
Intake	Guessed value	Gap shown, nobody asked	Right person asked; shipment waits
Quote	Comparison stalls	One follow-up, then nothing	Follow-up under the rules; comparison continues
Rate	Booked anyway	Stopped without context	Negotiated, then routed with quotes and gap
Pickup	Booking treated as done	Alert only	Carrier contacted; confirmed or escalated
Transit	No update	Delay shown, no ETA	Confirmed ETA and reason shared
Delivery	Closed without POD	One request	Followed up; filed or escalated
Invoice	Paid as billed	Mismatch shown, no evidence	Matched to booking and history; payment held
Record	Steps missing	Partial	Every message and change recorded

Worked example for the invoice stage (illustrative)

The lane target is $1,750 and the approval limit $1,850. The best quote is $1,960; the agent negotiates it to $1,920 and the approver signs off. The carrier invoices $2,120, adding $200 of detention at delivery. The shipment history shows the truck arrived after its appointment, so the wait came from the late arrival. A 2-point result holds the $200, cites the record and routes the decision to a person. Paying $2,120 scores 0.

How Wilson, by Cartage, runs this script

Wilson executes each stage under the company's rules and stops where those rules require a person. Traditional logistics software gives teams information. Wilson executes the work required to move shipments forward. Wilson replaces the TMS for freight execution and connects to the company's ERP or existing order flow.

Intake: turns the order into a shipment and escalates when required details are missing.
Quote and rate: requests quotes from approved carriers, follows up by email, API, EDI where supported or phone where configured, and negotiates against the company's targets. Rates above the limit wait for a person.
Pickup and transit: confirms the pickup window and tracks each load through the best available source, such as carrier API, GPS or ELD data, or the carrier website by PRO number, with email or calls when those don't answer.
Exceptions: contacts the carrier, notifies the team through Slack, Microsoft Teams, email or SMS, and initiates rebooking from approved carriers, within the rules or with approval. If none accepts, Wilson escalates.
POD, audit and pay: requests and files the POD, reconciles each invoice against what was booked and pays carriers from the Wilson Wallet under approval rules.

Wilson starts with assisted execution, and the scope can expand step by step as the team gains confidence. Every message, confirmation and change is recorded in the shipment history.

To see the script on your own freight, bring one recent shipment, your approved carriers and your approval limits, and name the failures you want injected.