Getting Started

Content Processing Pipeline


File processing tasks — image resizing, video transcoding, PDF generation — are naturally long-running and prone to failure. By modeling each processing step as a task, Orcher checkpoints progress automatically. If transcoding a 2-hour video fails at the thumbnail generation step, the workflow resumes from there instead of re-transcoding the entire video.

How it works

The pipeline is a small DAG: a serial spine of download → transcode → thumbnail → upload → record, with two stages fanning out because their branches are independent. Each box is a journaled task, so the graph is also the checkpoint map.

download_source
try_join — parallel
transcode
mp4 · 720p
transcode
webm · 720p
generate_thumbnail
try_join — parallel
upload
mp4
upload
webm
upload
thumb
update_database
status: ready

Fan-out stages run their branches concurrently; each branch is checkpointed on its own.

Example: a video-processing pipeline

1
import {
2
Workflow,
3
Tasks,
4
Task,
5
WorkflowContext,
6
createTaskRefs,
7
} from '@orcher/sdk';
8
9
@Tasks()
10
class VideoTasks {
11
@Task()
12
async downloadSource(url: string): Promise<string> {
13
const bytes = await httpClient.get(url);
14
return await storage.saveTemp('source', bytes);
15
}
16
17
@Task()
18
async transcodeVideo(input: TranscodeInput): Promise<string> {
19
return await ffmpeg.transcode(input.source, input.format, input.quality);
20
}
21
22
@Task()
23
async generateThumbnail(videoPath: string): Promise<string> {
24
return await ffmpeg.extractFrame(videoPath, 10.0);
25
}
26
27
@Task()
28
async uploadToCdn(input: UploadInput): Promise<string> {
29
return await cdn.upload(input.path, input.key);
30
}
31
32
@Task()
33
async updateDatabase(input: MediaRecord): Promise<void> {
34
await db.updateMedia(input);
35
}
36
}
37
38
const tasks = createTaskRefs(VideoTasks);
39
40
@Workflow({ name: 'process_video_upload', version: '1.0.0' })
41
class ProcessVideoUpload {
42
async run(
43
ctx: WorkflowContext,
44
input: VideoUpload,
45
): Promise<ProcessedVideo> {
46
// Step 1: Download source file
47
const source = await ctx.executeTask(tasks.downloadSource, input.sourceUrl);
48
49
// Step 2: Transcode to multiple formats in parallel
50
const [mp4, webm] = await Promise.all([
51
ctx.executeTask(tasks.transcodeVideo, {
52
source, format: 'mp4', quality: '720p',
53
}),
54
ctx.executeTask(tasks.transcodeVideo, {
55
source, format: 'webm', quality: '720p',
56
}),
57
]);
58
59
// Step 3: Generate thumbnail
60
const thumbnail = await ctx.executeTask(tasks.generateThumbnail, source);
61
62
// Step 4: Upload all assets to CDN
63
const [mp4Url, webmUrl, thumbUrl] = await Promise.all([
64
ctx.executeTask(tasks.uploadToCdn, {
65
path: mp4, key: `${input.videoId}/video.mp4`,
66
}),
67
ctx.executeTask(tasks.uploadToCdn, {
68
path: webm, key: `${input.videoId}/video.webm`,
69
}),
70
ctx.executeTask(tasks.uploadToCdn, {
71
path: thumbnail, key: `${input.videoId}/thumb.jpg`,
72
}),
73
]);
74
75
// Step 5: Update database
76
await ctx.executeTask(tasks.updateDatabase, {
77
videoId: input.videoId,
78
mp4Url,
79
webmUrl,
80
thumbnailUrl: thumbUrl,
81
status: 'ready',
82
});
83
84
return { mp4Url, webmUrl, thumbnailUrl: thumbUrl };
85
}
86
}

The parallel transcode and upload steps run concurrently for better throughput. Each step is independently checkpointed — the expensive transcode work is never repeated on replay.

The expensive work is done once

This is the payoff for long, costly steps. A crash during upload doesn't re-run the twenty-minute transcode — its output was journaled the moment it finished, and replay hands that result straight back:

Resume Without Re-Transcoding
✓
transcode mp4 · webm~20 min of ffmpeg — outputs journaled
✓
generate_thumbnailFrame extracted — journaled
✗
upload mp4CDN push fails — network blip
↻
replaytranscode + thumbnail return journaled paths — ffmpeg never runs again
✓
upload mp4 · webm · thumbRetried from the upload stage
The journal stores the file path, not the file — so replay is instant and cheap.

Design notes

  • Pass file references, never bytes. Every task here returns a FilePath or URL, not the media itself. Task inputs and outputs are journaled, so threading a multi-gigabyte video through them would bloat the journal and slow every replay. Keep the payload a pointer; keep the bytes in object storage.
  • Intermediate artifacts must outlive the worker. Replay after a crash may land on a different worker, so a transcode output saved to local /tmp won't be there. Write intermediate files to shared storage (S3, GCS) keyed by video ID — then the journaled path resolves no matter which worker resumes.
  • Make uploads idempotent by key. A resumed upload re-pushes to the same CDN key ({video_id}/video.mp4), overwriting rather than duplicating. Deterministic keys turn a retry into a harmless no-op.
  • The DAG's shape is your throughput. try_join runs the transcodes concurrently because neither depends on the other; the database write is last because it depends on all of them. When adding steps, parallelize any that don't share inputs — the fan-out is free structure, the same idea as parallel ETL.

Next Steps