Content Processing Pipeline
File processing tasks — image resizing, video transcoding, PDF generation — are naturally long-running and prone to failure. By modeling each processing step as a task, Orcher checkpoints progress automatically. If transcoding a 2-hour video fails at the thumbnail generation step, the workflow resumes from there instead of re-transcoding the entire video.
How it works
The pipeline is a small DAG: a serial spine of download → transcode → thumbnail → upload → record, with two stages fanning out because their branches are independent. Each box is a journaled task, so the graph is also the checkpoint map.
Fan-out stages run their branches concurrently; each branch is checkpointed on its own.
Example: a video-processing pipeline
1import {2 Workflow,3 Tasks,4 Task,5 WorkflowContext,6 createTaskRefs,7} from '@orcher/sdk';89@Tasks()10class VideoTasks {11 @Task()12 async downloadSource(url: string): Promise<string> {13 const bytes = await httpClient.get(url);14 return await storage.saveTemp('source', bytes);15 }1617 @Task()18 async transcodeVideo(input: TranscodeInput): Promise<string> {19 return await ffmpeg.transcode(input.source, input.format, input.quality);20 }2122 @Task()23 async generateThumbnail(videoPath: string): Promise<string> {24 return await ffmpeg.extractFrame(videoPath, 10.0);25 }2627 @Task()28 async uploadToCdn(input: UploadInput): Promise<string> {29 return await cdn.upload(input.path, input.key);30 }3132 @Task()33 async updateDatabase(input: MediaRecord): Promise<void> {34 await db.updateMedia(input);35 }36}3738const tasks = createTaskRefs(VideoTasks);3940@Workflow({ name: 'process_video_upload', version: '1.0.0' })41class ProcessVideoUpload {42 async run(43 ctx: WorkflowContext,44 input: VideoUpload,45 ): Promise<ProcessedVideo> {46 // Step 1: Download source file47 const source = await ctx.executeTask(tasks.downloadSource, input.sourceUrl);4849 // Step 2: Transcode to multiple formats in parallel50 const [mp4, webm] = await Promise.all([51 ctx.executeTask(tasks.transcodeVideo, {52 source, format: 'mp4', quality: '720p',53 }),54 ctx.executeTask(tasks.transcodeVideo, {55 source, format: 'webm', quality: '720p',56 }),57 ]);5859 // Step 3: Generate thumbnail60 const thumbnail = await ctx.executeTask(tasks.generateThumbnail, source);6162 // Step 4: Upload all assets to CDN63 const [mp4Url, webmUrl, thumbUrl] = await Promise.all([64 ctx.executeTask(tasks.uploadToCdn, {65 path: mp4, key: `${input.videoId}/video.mp4`,66 }),67 ctx.executeTask(tasks.uploadToCdn, {68 path: webm, key: `${input.videoId}/video.webm`,69 }),70 ctx.executeTask(tasks.uploadToCdn, {71 path: thumbnail, key: `${input.videoId}/thumb.jpg`,72 }),73 ]);7475 // Step 5: Update database76 await ctx.executeTask(tasks.updateDatabase, {77 videoId: input.videoId,78 mp4Url,79 webmUrl,80 thumbnailUrl: thumbUrl,81 status: 'ready',82 });8384 return { mp4Url, webmUrl, thumbnailUrl: thumbUrl };85 }86}The parallel transcode and upload steps run concurrently for better throughput. Each step is independently checkpointed — the expensive transcode work is never repeated on replay.
The expensive work is done once
This is the payoff for long, costly steps. A crash during upload doesn't re-run the twenty-minute transcode — its output was journaled the moment it finished, and replay hands that result straight back:
Design notes
- Pass file references, never bytes. Every task here returns a
FilePathor URL, not the media itself. Task inputs and outputs are journaled, so threading a multi-gigabyte video through them would bloat the journal and slow every replay. Keep the payload a pointer; keep the bytes in object storage. - Intermediate artifacts must outlive the worker. Replay after a crash may land on a different worker, so a transcode output saved to local
/tmpwon't be there. Write intermediate files to shared storage (S3, GCS) keyed by video ID — then the journaled path resolves no matter which worker resumes. - Make uploads idempotent by key. A resumed upload re-pushes to the same CDN key (
{video_id}/video.mp4), overwriting rather than duplicating. Deterministic keys turn a retry into a harmless no-op. - The DAG's shape is your throughput.
try_joinruns the transcodes concurrently because neither depends on the other; the database write is last because it depends on all of them. When adding steps, parallelize any that don't share inputs — the fan-out is free structure, the same idea as parallel ETL.