Rebuilding SiriusXM's Campaign File Pipeline for 10x Faster Processing

How Celerity re-engineered the file exchange process behind SiriusXM's Adobe Campaign instance, turning an hours-long single-file bottleneck into a multi-threaded pipeline that lets quick campaigns launch the same day.

The Challenge

SiriusXM's Adobe Campaign environment depended on ACX (Adobe Content Exchange), an out-of-the-box file exchange process, to import recipient updates and manage the daily list imports that determined who should receive each marketing campaign. Rather than integrating directly with SiriusXM's databases, the process ran entirely on flat file drops — as many as 20 to 50 files landing on a shared drive every day, some carrying tens of millions of recipient records.

Every one of those files had to pass through a single, single-threaded workflow before any campaign could go out the door. A file with 25 to 50 million records could occupy the entire pipeline for hours, and while it processed, nothing else could move. Small, simple campaigns that should have taken minutes to launch were instead stuck waiting for most of the day, simply because a much larger file happened to be ahead of them in line.

The Approach

Celerity was already engaged with SiriusXM's Adobe Campaign instance and had deep familiarity with the system when this bottleneck became the priority. The team traced the slowdown to two root causes and rebuilt each piece from the ground up.

Multi-threading recipient validation

The existing process validated recipients sequentially, one file at a time, no matter how many were queued behind it. Celerity redesigned it as a queue-based system: incoming files now register with a centralized processor that load-balances recipient validation across multiple parallel workflows. Layers of custom logic were also replaced with a single, purpose-built SQL script optimized purely for speed — stripping out unnecessary overhead so records could be validated and written back as fast as possible.

Routing files by size

With recipient validation no longer the chokepoint, Celerity turned to the file processor itself. Previously, every file moved through one shared queue regardless of size. The team rebuilt the workflow to classify incoming files as small, medium, or large and route each to its own processing lane — so a handful of small files can move through in minutes while large files run independently in the background, without holding anything else up.

The Results

• ~10x faster processing for the recipient validation step that gates every campaign launch.

• Small campaigns launch same-day instead of queuing behind large files for most of the day.

• Large and small files run independently, so file volume no longer dictates when a marketing message actually reaches its audience.

• The redesign turned a process the marketing team had learned to work around into one that keeps pace with how often — and how quickly — SiriusXM needs to campaign.