Live video inside VS Code

FramePort live capture demo in VS Code

Live capture inside VS Code. Click the preview for the full clip.

HDMI capture directly inside VS Code.

View on Open VSX GitHub

0.2.5 · JavaScript · FFmpeg · VS Code API

FramePort is an extension for VS Code. It opens a live HDMI or USB video feed in an editor tab, so a board’s display sits next to the code that drives it. On a Mac it can also open the iPhone camera, so the preview is you, or Desk View, the overhead camera. It can save a PNG screenshot or a silent MP4.

The extension is published on Open VSX. Where this manual and src/ disagree, the source wins.

1. Introduction

This chapter states what FramePort is, what it is not, and how to read the rest of the manual.

1.1. Purpose and scope

FramePort is a VS Code extension that opens an HDMI capture card or other USB video device in an editor tab. On macOS the same picker includes the cameras the system exposes, including the iPhone camera (the preview is you) and iPhone Desk View (the overhead camera). It shows the feed beside your code, inspects individual pixels, saves a PNG screenshot, and records a silent MP4 clip without a separate video application.

What FramePort is not:

  • No audio capture, editing, or broadcasting. The iPhone camera and Desk View are the phone’s cameras. They are not a copy of the iPhone’s screen.

  • No background service or network video upload; capture starts only on an explicit action (see Architecture).

  • Device capture is implemented for macOS (AVFoundation), Windows (FFmpeg dshow, new in 0.2.4), and Linux (V4L2 /dev/videoN, new in 0.2.5). Physical devices are verified on macOS. Linux lists the primary capture node and skips the metadata node that shares its name.

  • Desktop VS Code 1.95+ in a trusted local workspace only. Remote, browser-only, and untrusted workspaces are unsupported.

Section Architecture defines the data path. Section Verification defines what has been checked and how to reproduce it. Section Known limitations states explicit non-claims.

1.2. How to read this manual

  • Chapters, sections, and subsections are numbered (sectnums).

  • The left navigation mirrors the document hierarchy and collapses per chapter; see the tip on the title page.

  • Cross-references such as Interfaces navigate within this single-page manual.

  • External links (for example VS Code webview UX, the design reference for this project) open the referenced site.

1.3. Conventions

Inline code such as frameport.previewFps, FramePort: Take Screenshot, or media/screenshots denotes settings, commands, or paths. File paths such as src/recorder.js are repository-relative.

Three rates are never merged:

  • Requested mode — the resolution/frame rate asked of the device.

  • Observed FPS — frames actually rendered in the preview.

  • Encoded frames — frames the recorder confirmed into the MP4.

1.4. Source map

The editable authorities for this project are:

Question Authority

Capture arguments and JPEG framing

src/capture.js

Capture lifecycle

src/session.js

Recording and finalization

src/recorder.js

Screenshot limits and validation

src/screenshot.js

Commands, dialogs, settings

src/extension.js

Panel markup, rendering, controls

src/view.js, media/preview.js, media/preview.css

Commands and settings contract

package.json

UI constraints

docs/DESIGN.md

Development and test workflow

docs/DEVELOPMENT.md

Release workflow

docs/PUBLISHING.md, CHANGELOG.md

Historical rationale

docs/log/

Generated captures under media/screenshots/ and media/videos/ are derivatives, not editable authorities.

2. Architecture

This chapter describes the running system: where capture happens, where frames flow, and which blocks own what. Authority: src/.

2.1. Overview

FramePort runs in the local UI extension host. It starts no capture until an explicit action (open, select device, test pattern). The frame endpoint binds to loopback with an unpredictable session path behind a restricted webview CSP. There is no background service and no network video upload.

FramePort capture pipeline
Figure 1. Capture pipeline: sources converge on FFmpeg; the bounded parser fans out to preview and recording branches.

Figure provenance: hand-rendered SVG from docs/diagrams/capture-pipeline.mmd. Device sources differ per OS (see Known limitations); the pipeline after FFmpeg is identical.

2.2. Block inventory

Block Responsibility Owns (state / artifacts)

src/capture.js

Source enumeration, FFmpeg capture arguments, JPEG framing

Device list, requested modes, frame parser

src/session.js

Serialized capture lifecycle, stale-callback cancellation

One live capture at a time

src/recorder.js

Bounded encoder input, progress parsing, output confirmation, finalization

MP4 output, encoded-frame count

src/screenshot.js

Screenshot limits, validation, saved-file action selection

PNG output, size refusal

src/extension.js

Commands, dialogs, settings, local server, coordination

Panel lifecycle, save destinations

src/view.js, media/preview.*

Theme-aware markup, rendering, controls, accessibility

Displayed frame, toolbar state

Tip
If a behavior is not in Interfaces, it is not a contract. Change it freely; changing an interface table is a breaking change.

2.3. Data flow

Frames leave the bounded parser on two independent branches:

  • The preview branch serves the latest frame to the webview canvas. Pause freezes this branch only; capture and recording continue.

  • The recording branch feeds the current capture into an on-demand encoder. Encoder progress must confirm frames before Saved is reported; arrival-time timestamps are approximate.

Normal teardown finalizes recording before replacing the source. Interrupted files are retained, never silently deleted. See Design.

3. Design

This chapter records how the blocks in Architecture realize the pipeline. It is an implementation description, not a second contract.

3.1. Capture and session

src/capture.js enumerates sources and builds the FFmpeg capture arguments per OS (AVFoundation on macOS, dshow on Windows, V4L2 on Linux, plus a synthetic test pattern on every platform). It frames the JPEG output stream for the bounded parser, which keeps the latest frame and skips — never queues — under load.

src/session.js serializes the capture lifecycle so restarts never touch the device concurrently. Stale callbacks and frames from a stopped or replaced capture are discarded.

3.2. Recorder

src/recorder.js reuses the current capture feed; it never reopens the device. The encoder input queue is bounded. Success requires all of: accepted input frames, confirmed encoded frames from FFmpeg progress on a separate pipe, a clean encoder exit, and nonempty output. A header-only file with accepted input still reports failure.

3.3. Screenshot

src/screenshot.js saves the displayed frame — including a paused frame — as PNG. Oversized captures fail visibly on client and host instead of disappearing. Reveal destinations are local-file only.

3.4. Panel and preview

src/view.js with media/preview.js and media/preview.css renders the theme-aware panel. Rules from docs/DESIGN.md:

  • Follow VS Code theme colors and typography, including light and high-contrast themes. Only the live-video surface stays black.

  • No brand palette, gradients, glows, glass, hero sections, or slogans in the capture panel. Short operational labels; errors state what failed and what to try.

  • Show real measurements only. Requested settings and observed output are labeled separately.

Preview pacing uses deadline-based scheduling so a 30 fps cap delivers near 30 unique frames; per-frame layout work runs only when dimensions, scaling, or smoothing change. FPS sampling resets on pause and tab visibility changes.

3.5. Error handling

Errors are part of the design:

  • Input errors — unwritable filename, oversized screenshot, missing encoder in the FFmpeg build. Reported with the next action.

  • Capture errors — device held by another app, dropped HDMI, bad mode. Reported with permissions, cable, and lower-mode steps.

  • Toolchain errors — FFmpeg missing or not on VS Code’s PATH. Reported with the Homebrew install, guide, or settings actions.

Each class names its recovery in Programming model.

4. Interfaces

This chapter defines every boundary another user or system touches: commands, settings, capture modes, save destinations, and the FFmpeg tool boundary. Authority: package.json, src/.

4.1. Conventions

  • Requested capture settings and observed preview output are labeled separately everywhere. A requested mode is never a measured result.

  • Screenshot and recording destinations are local files only.

4.2. Commands

Command What it does

FramePort: Open Capture Device

Open the capture panel.

FramePort: Select Device

Pick a source and requested mode.

FramePort: Take Screenshot

Save the displayed frame as PNG.

FramePort: Start / Stop Recording

Record silent H.264 MP4; never overwrites.

FramePort: Restart Stream

Restart capture on the current source.

FramePort: Stop Stream

Release the device.

FramePort: Show Diagnostics

Inspect FFmpeg output on failure.

FramePort: Show Last Capture

Reveal the last PNG/MP4 in the file manager.

4.3. Settings

Setting Default Use

frameport.ffmpegPath

ffmpeg

Executable override (for example /opt/homebrew/bin/ffmpeg).

frameport.previewFps

30

Preview cap: 15, 30, or 60. Source capture and recording are independent.

frameport.recordingEncoder

software

libx264 (smaller files, everywhere) or macOS VideoToolbox hardware (larger files; actionable error when unavailable).

frameport.saveRoot

media

Capture root: workspace-relative, absolute as-is, else ~/FramePort.

frameport.screenshotFolder

screenshots

PNG subfolder under the save root.

frameport.videoFolder

videos

MP4 subfolder under the save root.

frameport.saveMode

auto

auto writes immediately; ask shows the save dialog.

frameport.filenameStyle

timestamp

timestamp (frameport-YYYYMMDD-HHMMSS) or sequential (frameport-0001, …). Existing files are never overwritten.

4.4. Capture modes

Modes are requested per device, not probed capabilities. Screen sources capture at native size with a separate FPS choice; the footer never reports an unapplied camera resolution.

Requested capture modes
Figure 2. Capture-mode picker — requested modes, not probed capabilities.
FramePort source picker. USB video. FaceTime camera. iPhone camera. Desk View. A screen. The test pattern.
Figure 3. Source picker. The iPhone camera shows you. Desk View is the overhead camera.

4.5. Save destinations

Screenshots autosave as PNG under the screenshot folder; recordings as silent H.264 MP4 under the video folder. Closing the panel finalizes an active recording first. Deleted last-capture files report a clear message instead of failing silently.

4.6. FFmpeg boundary

FramePort requires FFmpeg with libx264 on PATH. On macOS it also checks /opt/homebrew/bin/ffmpeg and /usr/local/bin/ffmpeg. On Linux it checks /usr/bin/ffmpeg and /usr/local/bin/ffmpeg. Either platform accepts a frameport.ffmpegPath override. ffprobe next to the configured FFmpeg serves the integration tests (FRAMEPORT_TEST_FFMPEG, FRAMEPORT_TEST_FFPROBE overrides). If FFmpeg is missing, FramePort offers a Homebrew install on macOS, the download guide, or settings — never a dead end.

4.7. Host sandbox

The extension runs in the local UI extension host and is unsupported in remote, browser-only, and untrusted workspaces (untrustedWorkspaces and virtualWorkspaces are false). Camera and screen-recording permission is requested by macOS when needed.

5. Programming model

This chapter shows how to drive FramePort. Every claim here is executable or directly checkable.

5.1. Quick start

  1. Install 0.2.5 from the VS Code Marketplace or Open VSX, or install frameport-0.2.5.vsix via Extensions: Install from VSIX….

  2. Run FramePort: Open Capture Device, then Select device — or Test pattern to verify without hardware. On macOS the list can include the iPhone camera (the preview is you) and Desk View. On Linux the list is V4L2 capture nodes such as /dev/video0.

  3. Pick a capture mode the device supports. Allow camera access if macOS asks.

  4. Use Pause, Screenshot, Record, Focus, and Diagnostics from the panel toolbar or the Command Palette.

Screenshots and recordings autosave to media/screenshots and media/videos in the workspace (or ~/FramePort with no folder open). saveMode: ask shows the save dialog, still defaulting into those folders.

5.2. Settings recipes

// VS Code's PATH differs from the terminal's: pin FFmpeg explicitly.
# Settings → frameport.ffmpegPath → /opt/homebrew/bin/ffmpeg
  • Lower preview work without touching capture or recording: set frameport.previewFps to 15.

  • Smaller files everywhere: keep frameport.recordingEncoder software. Lower recording CPU on macOS at the cost of larger files: switch to hardware (VideoToolbox).

  • Per-project captures: set frameport.saveRoot to a workspace-relative folder; with no folder open, ~/FramePort is used.

5.3. Troubleshooting sequences

  • FFmpeg won’t launch. Use the Install-via-Homebrew / Install-guide actions when offered, or set frameport.ffmpegPath; then Restart Stream.

  • No frames. Check permissions, cables, other apps holding the device, and try a lower mode; then Show Diagnostics.

  • Recording fails. Confirm the encoder exists in the FFmpeg build and pick a new writable filename.

  • Black frames. Some cards emit black when HDMI drops; the video alone cannot always tell. Check the source and cable first.

To see the real missing-FFmpeg modal without uninstalling Homebrew FFmpeg: set frameport.ffmpegPath to /nonexistent-frameport-ffmpeg, run FramePort: Open Capture Device or Test pattern, then restore the setting afterward.

6. Verification

This chapter states what has been checked, with commands that reproduce each claim. Harness results are labeled harness; they never stand in for device permissions or Extension Host behavior.

6.1. Strategy

Layer Method Reproduce

Unit

Node test runner

npm test

Static gate

Checks

npm run check

Panel behavior

Browser harness, mocked VS Code messaging

npm run test:ui

Host behavior

Synthetic Extension Host suite

npm run test:host

Missing-FFmpeg recovery

Nonexistent path + mocked API

npm run test:ffmpeg-missing

Missing-FFmpeg in container

Node image without FFmpeg

npm run test:ffmpeg-missing:docker

Physical device

One app owns the device

Manual, see below

Note
The table above is the verification contract. A passing harness run does not prove device permissions or host behavior.

6.2. Focused checks

npm run check
npm test
npm run test:ui

Browser tests need Playwright Chromium (npx playwright install chromium if absent) and a local port. They generate screenshots under ignored artifacts/; they test the webview with mocked messaging, not device permissions or Extension Host behavior.

npm run test:host opens a real VS Code window in a temporary profile and never installs the candidate into the normal profile. Set VSCODE_TEST_BINARY if the binary lives elsewhere.

Missing-FFmpeg recovery is checked without uninstalling host FFmpeg: the suite uses a nonexistent path and a mocked VS Code API, plus the same checks in a Linux container with Node but no FFmpeg binary.

6.3. Physical device vs harness

  • Browser harness means npm run test:ui with mocked VS Code messaging. Label its screenshots synthetic; the recording-state shots are explicitly injected states.

  • Synthetic host means npm run test:host in a temporary profile. It exercises host wiring, not real user permissions.

  • Physical device means a camera or capture card owned by exactly one application, with device/mode, FFmpeg version, and contention status recorded alongside the measurement.

For every change, report which checks actually ran — unit, real FFmpeg, browser harness, Extension Host, and physical device — with device, mode, FFmpeg version, and contention status. Never infer hardware latency from preview FPS alone.

6.4. Captures in the tree

These stills are the manual’s preview set. The animated demo GIF is not checked in; the full clip is the MP4 linked from the overview.

FramePort preview idle
Figure 4. Idle preview
FramePort preview in the light theme
Figure 5. Light theme while a source is selected
FramePort preview in high contrast
Figure 6. High contrast
FramePort recording with focus indication
Figure 7. Focus while recording
Tomato video captured live in FramePort
Figure 8. Tomato output captured through FramePort

7. Implementation

This chapter describes how the checked-in sources become the installable extension and what the build does and does not prove.

7.1. Build flow

npm ci
npm run check
npm test
npm run test:ui
npm run package

npm ci installs development dependencies; no npm dependencies load in the installed extension. Press F5 to open an Extension Development Host and hack directly. Guides: docs/DEVELOPMENT.md, docs/PUBLISHING.md.

Programming the analogy loosely: building a VSIX does not publish a listing. Re-upload the exact reviewed artifact; Marketplace versions are immutable, so listing-only fixes ship as a patch bump.

7.2. VSIX contents

A good archive holds the manifest, README, CHANGELOG, src/, and runtime media/ only. Excluded via .vscodeignore: tests, scripts, docs/, build/, docker/, Makefile, artifacts/, test-host/, node_modules, and the demo MP4 (media/screenshots/*.mp4 loads from the repository URL, not the package). The demo GIF stays packaged so the Extension Details page plays it offline while the repository is private.

Inspect the actual archive, not only its filename:

unzip -l frameport-0.2.5.vsix
shasum -a 256 frameport-0.2.5.vsix

Install it through Extensions: Install from VSIX… in a clean profile.

7.3. Packaging variants

  • npm run package — standard flow. VSCE rewrites README image paths to repository HTTPS URLs, so publish referenced image files first and check them in the Marketplace preview.

  • npm run package:local — adds --no-rewrite-relative-links so the Details page resolves packaged media/screenshots/ images offline. Private-preview exception, valid only while the repository is private.

  • npm run package:public — standard flow with the license bypass removed. Use after distribution terms are selected.

The manifest preview field is false: the listing is not marked preview. That flag is not the pre-release channel; choose the channel deliberately per the current VS Code publishing guide.

8. Performance

This chapter gives the rate, pacing, and size numbers with exact wording. Each number names its domain; domains are never merged.

8.1. Rates

Quantity Value Meaning

Preview cap

15, 30, or 60

frameport.previewFps; display work only

Source rate

Device requested mode

What was asked of the hardware

Observed FPS

Measured

Frames actually rendered; resets on pause and tab visibility changes

Recording confirmation

Encoder progress

Frames confirmed into the MP4 before Saved is reported

Preview pacing uses deadline-based scheduling so a 30 fps cap delivers near 30 unique frames instead of ~27. One FFmpeg process runs while viewing, one more while recording. Under load, frames are skipped rather than queued, so observed FPS is output throughput, not latency.

8.2. Size and fidelity

  • The transport is JPEG, so PNG screenshots are not lossless.

  • Oversized screenshots are refused with a visible error.

  • Software libx264 recording makes smaller files and works everywhere; macOS VideoToolbox hardware recording makes larger files.

  • Existing files are never overwritten under either filename style.

Important
Source rate, preview cap, observed FPS, and encoded frames are different metrics. State the domain with every number. Arrival-time timestamps are approximate and need validation against the intended source timeline. Never infer hardware latency from preview FPS alone.

9. Known limitations

This chapter lists explicit non-claims so readers do not infer more than the repository proves.

9.1. Platform boundaries

  • Hardware capture is implemented for macOS (AVFoundation), Windows (FFmpeg dshow, new in 0.2.4), and Linux (V4L2, new in 0.2.5). Physical devices are verified on macOS. Linux and Windows device selection are covered by unit tests; a local capture card on those systems still needs a manual check.

  • On Linux, FramePort keeps the sysfs index-0 node (/dev/video0 for a typical grabber) and skips metadata nodes. Opening the device requires read permission, usually the video group. The requested mode has to be one the device accepts.

  • Desktop VS Code 1.95+ in a trusted local workspace is required. Remote, browser-only, and untrusted workspaces are unsupported.

  • macOS camera and screen-recording permission is requested when needed; without it there are no frames.

  • Exactly one application may own a device. Contention looks like missing or frozen frames; see Programming model.

9.2. Capability boundaries

  • No audio, editing, or broadcasting. The iPhone camera and Desk View are the phone’s cameras on the Mac. They are not a copy of the iPhone’s screen.

  • The transport is JPEG: screenshots are frames, not lossless captures.

  • Screen sources capture at native size; resolution presets do not apply to them.

  • Black frames from a dropped HDMI signal are indistinguishable from dark content in the video alone.

9.3. Verification boundaries

  • Browser-harness screenshots are synthetic markup states, not hardware evidence — including the explicitly labeled recording-state shots.

  • A passing unit, harness, or synthetic-host run does not prove device permissions, real Extension Host behavior, or latency.

  • Recording success requires confirmed encoded frames, a clean encoder exit, and nonempty output. Header-only output with accepted input reports failure by design.

9.4. Deployment boundaries

  • While the repository is private, README images resolve through packaged media/screenshots/ paths (npm run package:local). At public release the README points back at docs/images/ with the standard npm run package flow.

  • The Marketplace listing updates only on VSIX re-upload; building alone changes nothing public.

  • Dated entries under docs/log/ preserve superseded behavior as history. Do not use them to establish present status unless a current chapter confirms it.

10. Engineering log

Each note below is the original project log, in chronological order. Open the arrow in the sidebar to jump to an entry.

10.1. 2026-09-13 — Other video sources

FramePort does not need to be limited to HDMI capture cards. The source picker can also show cameras and screen sources exposed by macOS.

Available sources in FramePort

USB video, the FaceTime camera, the iPhone camera, Desk View, a screen source, and the test pattern. The list is whatever macOS exposes.

The iPhone camera is the phone's camera on the Mac, so the preview is you. Desk View is the overhead camera from the same phone. Both appear because macOS presents them as video devices, the same way it presents a capture card.

Seeing the iPhone's own screen, the way a simulator does, is a different source. Those two camera entries are not that. Sending a picture back to the phone would need a receiving application, and that path is not this one. Sources come into FramePort.

10.2. 2026-09-13 — Screenshots and recording

The features I care about most, after the live view, are screenshots and recordings of the FPGA output.

I want as little friction as possible between noticing something and saving it. A screenshot is useful for a display bug or a new screen layout. A short recording captures behavior that a still image cannot: a menu transition, a game, or a problem that appears only briefly.

FramePort now saves the displayed frame as PNG and records the active source as a silent MP4. Pausing the preview lets me inspect a frame; it does not pause recording. That distinction matters when I want to study one image without losing what happens next.

Recording indicator in FramePort focus mode

Focus mode with the recording badge, captured from the synthetic test-pattern harness with an injected recording state — not a hardware recording.

The next concern is responsiveness. Saving a clip should not make the live view unpleasant to use. The recorder shares the existing capture feed and skips frames under load rather than building an ever-growing queue. I still want to judge that behavior on the real hardware, not just by whether a test passes.

The aim is to keep the related steps together: view, inspect, save, and find the saved file. More features can follow when there is a clear need for them.

10.3. 2026-09-13 — Welcome, FramePort!

I built FramePort because I wanted to see my FPGA's HDMI output while working on its code.

At first, that meant keeping an external monitor nearby. A USB capture card was a more convenient option, but I still had to open another application just to view the feed. I couldn't find a VS Code extension that fit the workflow I wanted, so I started building one.

Now the video sits in an editor tab beside the code. That small change makes working on the hardware much easier: I can change something, look at the output, and keep going.

Tomato OS displayed through FramePort

The FPGA running Tomato OS, viewed through the USB capture card inside FramePort.

Tomato, my 32-bit computer project, is the reason FramePort exists. It is also the first use case I want to get right. Other video devices can fit the same workflow, but for now: Tomato first.

About me

I’m Tyrone Marhguy, a Computer Engineering student at the University of Pennsylvania, class of 2028. FramePort is the VS Code extension I wrote for HDMI and USB capture, published on Open VSX.

I grew up between Kumasi and Accra. In 2021, Achimota School denied me admission because I wear dreadlocks as an expression of my Rastafarian faith. Ghana’s High Court ordered the school to admit me; I finished secondary school there, then came to Penn on a full scholarship. BBC Pidgin and Deutsche Welle covered the case.

I design the architecture, build the boards, and write the software. Tomato is a 32-bit computer with a running FPGA implementation and discrete boards in progress. I also build systems software, including a search engine, a compiler, and a database, and I contribute fixes to open-source EDA tools such as LibreLane, OpenROAD, Verilator, and OpenFPGA. That work has included production software at Aragorn AI and battery-management hardware at Vero Electric.

Hardware

Software and systems

Open source

See open-source work for contributions to LibreLane, OpenROAD, Verilator, and OpenFPGA.