How I automate 95% of my video workflow with AI coding agents

When you use AI to produce a video, there’s usually a catch. One-shot video generators hand you a finished video file. If a single cut is off, you can’t reach in and fix it. You have to tweak your prompt and regenerate the whole clip, hoping the new render doesn’t break something else.

Code-first video frameworks like Remotion and HyperFrames give you repeatable results in code, and they’re great for programmatic motion graphics. Still, editing a five-minute walkthrough purely in code turns every tiny timing adjustment into a slow loop: edit a timestamp, re-render the composition, watch it back, and repeat. Sometimes you just want to grab a clip with your mouse and drag it where it belongs.

What I wanted sits right in between those two extremes. Let a coding agent handle 95% of the heavy lifting: synthesizing the voiceover, building word-synced animated graphics, cutting retakes, and assembling a first pass of the multi-track timeline. Then hand me a real project inside a desktop video editor so I can finish the last 5% by hand.

That hybrid pattern works well for tutorials and product demos. To test it end to end, I used this exact pipeline to turn one of my recent blog posts (Do agent skills help? I ran 72 trials to find out) into a multi-track video walkthrough. I also tell the story here in this 4-minute video:

https://medium.com/media/e3c4e80d99de4e445f5562f981b5e36c/href

Choosing a stack that works across agents and editors

The main idea behind this workflow isn’t locked to a single vendor or editor. Any coding agent that can read local files and run terminal commands can drive the pipeline, and any video editor that stores project timelines in an open text format can receive the handoff.

For my setup, I use Google Antigravity as the agentic development environment, paired with Gemini as the coding and multimodal model. For voice synthesis, I use gemini-3.8-flash-tts via Google AI Studio. For animated data overlays, I use HyperFrames. And for the desktop timeline editor, I use CapCut paired with capcut-cli, an unofficial open-source project compiler (not affiliated with or endorsed by CapCut).

If you use a different video editor, the same workflow applies through open interchange standards like Pixar and the Academy Software Foundation’s JSON-based OpenTimelineIO (.otio). In every case, the trick is generating the project file on disk rather than trying to automate mouse clicks in a graphical interface.

Inside a timeline project: it’s just JSON on disk

Desktop video editors look like monolithic binary applications, but modern timelines are represented as structured text on your local drive. Generally, the project file comes down to a materials list pointing to your media files on disk and a tracks list defining the horizontal timeline layers and where each clip starts and ends.

If a timeline is plain JSON on disk, a coding agent can write it. And when you open your video editor afterward, every clip and audio track sits on its own unlocked layer, ready to drag around.

Compiling timelines declaratively with CLI tooling

Even though an agent can write raw timeline JSON, you don’t want it generating project files from scratch. A five-minute project spans thousands of lines of cross-referenced IDs and timestamps, and it’s easy to get subtly wrong. If a single clip duration runs past the end of its source video file, the editor will flag the project as corrupted.

Instead of having the agent write raw editor JSON by hand, I use capcut-cli. The agent writes a short, readable spec in seconds, and the CLI builds the project and verifies every clip against the real media files on disk.

Here is a trimmed excerpt from spec.json (the input file for capcut compile) showing how the main camera track and the B-roll overlay track line up:

{
  "name": "AI Video CapCut Walkthrough",
  "width": 1920,
  "height": 1080,
  "fps": 30,
  "tracks": [
    {
      "type": "video",
      "name": "main-visuals",
      "items": [
        {
          "ref": "aroll_intro",
          "path": "assets/talking-head.mov",
          "start": 0.0,
          "duration": 45.9,
          "type": "video"
        }
      ]
    },
    {
      "type": "video",
      "name": "broll-overlays",
      "items": [
        {
          "ref": "three_ways_diagram",
          "path": "assets/nanobanana/01_three_ways_comparison.png",
          "start": 28.8,
          "duration": 9.8,
          "type": "photo"
        }
      ]
    }
  ]
}

Once spec.json is ready, the agent runs capcut compile and capcut lint:

capcut compile spec.json --template auto --check -H
capcut compile spec.json --template auto --drafts "$HOME/Movies/CapCut/User Data/Projects/com.lveditor.draft"
capcut lint "$HOME/Movies/CapCut/User Data/Projects/com.lveditor.draft/AI Video CapCut Walkthrough" -H

Passing –template auto clones the project format from an existing local CapCut draft so the desktop app always recognizes the version. The CLI builds the project folder and checks every clip against its real length on disk before you ever open the editor.

Cutting retakes and cloning narration with Gemini TTS

Every video starts with the audio and main footage. You can record yourself on camera, or you can generate a voiceover track from a script.

When I record on camera, I don’t stop and restart every time I miss a line. I record as many takes as I need in one continuous roll, then have the agent run capcut detect-silence and capcut detect-retakes. The CLI flags the dead air and repeated takes and outputs the clean keep segments, which the agent drops straight into spec.json.

When I’m building a video from an existing blog post (or when I change a sentence during editing and don’t want to set up the microphone again), I use voice replication in gemini-3.8-flash-tts.

In Google AI Studio, you record about 20 seconds of yourself talking along with a short consent statement. You get back a voice ID, and gemini-3.8-flash-tts can read any script in that voice. Using controllable speech generation, you can also direct the delivery style and add pauses right in the text:

import os
from pathlib import Path
from google import genai
 
client = genai.Client()
 
response = client.models.generate_content(
    model="gemini-3.8-flash-tts",
    contents=[
        {
            "role": "user",
            "parts": [
                {
                    "text": (
                        "When you load skills into an AI coding agent, are you making it "
                        "smarter, or just burning tokens on prompt bloat? <short pause> "
                        "I ran 72 isolated sandbox trials to find out."
                    ),
                    "speech_metadata": {
                        "style": "warm, conversational, and direct"
                    },
                }
            ],
        }
    ],
    config={
        "response_modalities": ["AUDIO"],
        "speech_config": {
            "voice_config": {"voice": os.environ["GEMINI_TTS_VOICE"]}
        },
    },
)
 
audio_bytes = response.candidates[0].content.parts[0].inline_data.data
Path("assets/narration-gemini.wav").write_bytes(audio_bytes)

If a line feels clunky when I watch the rough cut, I edit the sentence in Python and regenerate the audio in seconds.

Building word-synced overlays in HyperFrames

Once the voiceover is ready, the captions and animated graphics need to stay in lockstep with the voice. To get the timing of every spoken word, the agent transcribes the audio track with HyperFrames:

npx hyperframes transcribe assets/narration-gemini.wav -o assets/transcript.json

That writes word-level timestamps into assets/transcript.json, which the agent uses to time both the B-roll cuts and the animated HTML and GSAP overlays.

To keep the generated graphics consistent, every HyperFrames project includes a frame.md design spec that sets the colors and typography before the agent writes any HTML.

Inside compositions/walkthrough.html, the agent sizes each caption phrase on an offscreen HTML <canvas> so long technical terms never run off the screen, and animates an orange underline across each word as it's spoken:

const measureCtx = document.createElement("canvas").getContext("2d");
const SAFE_WIDTH_PX = 1664;
 
function measurePhraseWidth(words, fontSize) {
  measureCtx.font = `700 ${fontSize}px Inter, -apple-system, sans-serif`;
  const textWidth = words.reduce(
    (sum, w) => sum + measureCtx.measureText(w.text).width,
    0,
  );
  return textWidth + Math.max(0, words.length - 1) * 14;
}
 
function fitPhraseFontSize(words) {
  for (let size = 38; size >= 22; size -= 2) {
    if (measurePhraseWidth(words, size) <= SAFE_WIDTH_PX) return size;
  }
  return 22;
}

Before rendering the overlay, the agent runs npx hyperframes check to verify that nothing overlaps and that all text passes WCAG contrast checks. Then npx hyperframes render exports the composition to an MP4 that the agent adds to spec.json as an overlay track.

Finishing the last 5% in your video editor

After the overlays render and capcut compile builds the project, the automated 95% is done. I open CapCut Desktop, and the multi-track project is waiting in my drafts list.

This is where the hybrid workflow pays off. Your main footage sits on the bottom track, with the overlays and voiceover stacked on separate tracks. If a diagram from the blog post needs to hold a little longer, I don’t touch code or wait for a re-render. I just grab the clip edge with my mouse and drag it.

Packaging the pipeline as a portable agent skill

The final step is making the workflow repeatable. Everything that worked goes into an Agent Skill, which is a folder of Markdown instructions and helper scripts that the agent picks up automatically next time.

The skill covers four things so the agent doesn’t have to figure them out from scratch on the next video:

1. The step-by-step build order across voiceover, transcription, graphics, and timeline assembly.
2. The CLI commands and input schemas for capcut-cli and HyperFrames.
3. The frame.md visual design rules so every video keeps the same look and typography.
4. Pre-flight checks that verify clip durations and text contrast before anything reaches the video editor.

Because skills are plain Markdown and scripts, the same skill runs across Antigravity, Claude Code, or your favorite coding agent. Next time you make a video, let the agent build the timeline, and save your hands for the final cut.

If you experiment with agent-driven video pipelines or build skills for your own editor, I’d love to see what you come up with. Connect with me on X, LinkedIn, or Bluesky.

How I automate 95% of my video workflow with AI coding agents was originally published in Google Cloud – Community on Medium, where people are continuing the conversation by highlighting and responding to this story.

Source Credit: https://medium.com/google-cloud/how-i-automate-95-of-my-video-workflow-with-ai-coding-agents-90dd4a87c53f?source=rss—-e52cf94d98af—4

About the Author

Deven Goratela

Administrator

Visit Website View All Posts