Customs clearance

Customs code classification

Year: 2024
Role: Backend engineer, production owner
Outcomes: 1,400 declarations per minute

Context

A customs operator files declarations for importers. Classifying goods against the tariff nomenclature was done by hand against the agency registry; a mistake meant cargo held at the border and a dispute. The business problem was to take code lookup away from the declarant and leave them only the confirmation.

Constraints

The external registry goes down roughly once a day, with no warning
The registry is updated without versioning and without notifications
A misclassification costs the client a cargo delay at the border

Solution

The registry sits behind a circuit breaker, and parsing runs through a worker pool with bounded concurrency. Thirty-two goroutines came from measurement, not intuition: above that, throughput stopped growing and p99 started drifting.

internal/classifier/pool.go

func (p *Pool) Run(ctx context.Context, in <-chan Declaration) <-chan Result {
	out := make(chan Result, p.size)

	var wg sync.WaitGroup
	for i := 0; i < p.size; i++ {
		wg.Add(1)
		go func() {
			defer wg.Done()
			for decl := range in {
				res, err := p.classify(ctx, decl)
				if err != nil {
					p.metrics.Failures.Inc()
					res = Result{Declaration: decl, Fallback: true}
				}
				out <- res
			}
		}()
	}

	go func() { wg.Wait(); close(out) }()

	return out
}

I hard-coded the classifier's confidence threshold as a constant. A quarter later it was clear that different goods groups need different thresholds, and the constant became a map — that should have been the shape from the start.

Architecture

The registry sits behind a breaker: its outage never reaches the declarant

Key decisions

Behaviour when the registry is unreachable
What I chose
A circuit breaker plus a cache of the last successful registry snapshot. On a break the system answers from yesterday and marks the response stale.
What I rejected
Exponential-backoff retries with no breaker: they turn an hour of registry downtime into an hour of our own API being down.
Why
The declarant needs an answer now, caveats included. Failing with nothing is worse than data labelled "snapshot from yesterday" — in the second case a person can still make the call.
Worker pool size
What I chose
Thirty-two goroutines, chosen by measurement against the production load profile.
What I rejected
A pool sized to CPU count, or an unbounded goroutine per declaration.
Why
Above thirty-two, throughput stopped growing and p99 started drifting: the bottleneck moved into the database connection pool. The number comes from a measurement, not a formula.

What was hard

Problem
The registry changed without versioning: the agency silently edited positions, and the cache served stale codes as current.
Solution
The collector hashes the export and compares it with the previous one. A mismatch marks the affected positions and evicts them individually rather than dropping the whole registry.
Problem
The first incident was debugged from logs: there were no metrics, and six minutes of degradation were reconstructed from memory.
Solution
Prometheus and a dashboard went in immediately after. The next degradation was readable from a graph in half a minute.

Outcomes

Under NDA

This case is under NDA: absolute figures are shown as relative ones.

throughput: ≈ / мин
p95: ≈ ms
classification errors: ≈ %
degradations in 30 days: ≈ · 6 мин