What is rustjay-engine?

rustjay-engine is a Cargo workspace of focused crates that handles all the infrastructure of a live visual performance tool — GPU rendering, video I/O, audio analysis, parameter modulation, and network control — so you can focus on the one thing that makes your effect unique: the shader.

When you implement the EffectPlugin trait and call rustjay_engine::run(), you get a complete dual-window application:

• Output window — fullscreen GPU-rendered output
• Control window — a tabbed panel with built-in sections for input sources, audio analysis, LFO modulation, MIDI mapping, output routing, and presets

Running the examples

git clone https://github.com/BlueJayLouche/rustjay-engine
cd rustjay-engine
cargo run -p template    # HSB colour — the simplest effect
cargo run -p delta       # RGB delay / motion extraction
cargo run -p waaaves     # multi-pass feedback pipeline
cargo run -p sputnik     # mesh displacement (Rutt-Etra style)

How to use this guide

Start with Installation, then work through Your First Effect. After that, the chapters are largely independent — jump to whatever is relevant to what you're building.

Installation

Prerequisites

• Rust 1.80+ — install via rustup
• cargo (comes with Rust)
• A C/C++ build toolchain (for linking wgpu's native backends):
  • macOS: Xcode Command Line Tools (xcode-select --install)
  • Windows: Visual Studio Build Tools (MSVC)
  • Linux: gcc, pkg-config, and dev headers (build-essential, libssl-dev, etc.)

Platform requirements

macOS

rustjay-engine renders via Metal. No extra GPU drivers needed.

For Syphon video sharing, install Syphon.framework into /Library/Frameworks/. Many VJ apps (Resolume, VDMX, MadMapper) bundle it — if you've installed any of those, Syphon is already present.

# Verify the framework is installed
ls /Library/Frameworks/Syphon.framework

Windows

rustjay-engine renders via Vulkan or DX12. Ensure your GPU drivers are up to date.

Spout video sharing uses DirectX interop and is included automatically — no separate install needed.

Linux

rustjay-engine renders via Vulkan. Install the Vulkan SDK for your distribution:

# Ubuntu / Debian
sudo apt install vulkan-tools libvulkan-dev

# Arch
sudo pacman -S vulkan-tools vulkan-icd-loader

V4L2 loopback output requires the v4l2loopback kernel module:

sudo modprobe v4l2loopback

NDI (optional)

NDI video-over-IP requires the NDI SDK to be installed. The ndi feature is enabled by default — if you don't have the SDK installed and see linker errors, disable it:

[dependencies]
rustjay-engine = { git = "...", default-features = false }

Ableton Link (optional)

The link feature requires CMake ≥ 3.14:

# macOS
brew install cmake

# Ubuntu
sudo apt install cmake

# Windows — download from https://cmake.org/download/

Enabling the link feature links against Ableton Link, which is GPL-2.0+. This changes the license of your resulting binary.

Creating a new project

cargo new my-effect
cd my-effect

Add rustjay-engine to Cargo.toml:

[dependencies]
rustjay-engine = { git = "https://github.com/BlueJayLouche/rustjay-engine" }
bytemuck = { version = "1.21", features = ["derive"] }
serde = { version = "1.0", features = ["derive"] }
anyhow = "1.0"
env_logger = "0.11"
log = "0.4"

Build it once to pull dependencies (this takes a few minutes the first time):

cargo build

You're ready. Head to Your First Effect.

Your First Effect

We'll build a simple desaturation effect in about 15 minutes. It reads video from your webcam, converts it to greyscale, and exposes an intensity parameter you can control from the UI.

Project layout

my-effect/
├── Cargo.toml
└── src/
    ├── main.rs
    └── shaders/
        └── desaturate.wgsl

The shader

Create src/shaders/desaturate.wgsl:

struct VertexOutput {
    @builtin(position) position: vec4<f32>,
    @location(0) texcoord: vec2<f32>,
};

@vertex
fn vs_main(
    @location(0) position: vec2<f32>,
    @location(1) texcoord: vec2<f32>,
) -> VertexOutput {
    var out: VertexOutput;
    out.position = vec4<f32>(position, 0.0, 1.0);
    out.texcoord = texcoord;
    return out;
}

// ── Bindings ─────────────────────────────────────────────────────────────
// group(0) — video input (always present)
@group(0) @binding(0) var input_tex:     texture_2d<f32>;
@group(0) @binding(1) var input_sampler: sampler;

// group(1) — your uniforms
struct Uniforms { intensity: f32 };
@group(1) @binding(0) var<uniform> u: Uniforms;

// ── Fragment ─────────────────────────────────────────────────────────────
@fragment
fn fs_main(in: VertexOutput) -> @location(0) vec4<f32> {
    let col  = textureSample(input_tex, input_sampler, in.texcoord);
    let grey = dot(col.rgb, vec3<f32>(0.299, 0.587, 0.114));
    let out  = mix(col.rgb, vec3<f32>(grey), u.intensity);
    return vec4<f32>(out, col.a);
}

A few things to notice:

• @group(0) @binding(0/1) — the live video texture and its sampler. These are always provided by the engine; your shader just declares them.
• @group(1) @binding(0) — your uniform block. This is where your per-frame data lives.
• The vertex shader is boilerplate — it passes a full-screen quad through unchanged.

The Rust side

Edit src/main.rs:

use rustjay_engine::prelude::*;

// ── Plugin struct ─────────────────────────────────────────────────────────
// Holds no data itself — state lives in DesaturateState below.
struct DesaturateEffect;

// ── GPU uniforms ──────────────────────────────────────────────────────────
// Must be repr(C) and implement Pod + Zeroable for bytemuck upload.
#[repr(C)]
#[derive(Copy, Clone, bytemuck::Pod, bytemuck::Zeroable)]
struct DesaturateUniforms {
    intensity: f32,
}

// ── App state ─────────────────────────────────────────────────────────────
// Serialisable so the preset system can save/restore it.
#[derive(Default, serde::Serialize, serde::Deserialize)]
struct DesaturateState {
    intensity: f32,
}

// ── Plugin implementation ─────────────────────────────────────────────────
impl EffectPlugin for DesaturateEffect {
    type State    = DesaturateState;
    type Uniforms = DesaturateUniforms;

    fn app_name(&self) -> &str { "my-effect" }

    fn shader_source(&self) -> &'static str {
        include_str!("shaders/desaturate.wgsl")
    }

    fn build_uniforms(&self, s: &DesaturateState, _engine: &EngineState) -> DesaturateUniforms {
        DesaturateUniforms { intensity: s.intensity }
    }

    fn parameters(&self) -> Vec<ParameterDescriptor> {
        vec![
            ParameterDescriptor::float(
                "intensity", "Intensity",
                ParamCategory::Color,
                0.0, 1.0, 0.0, 0.01,
            ),
        ]
    }
}

fn main() -> anyhow::Result<()> {
    env_logger::Builder::from_default_env()
        .filter_module("wgpu_hal", log::LevelFilter::Warn)
        .filter_module("naga",     log::LevelFilter::Warn)
        .filter_module("wgpu_core", log::LevelFilter::Warn)
        .init();

    rustjay_engine::run(DesaturateEffect)
}

Run it

cargo run --release

Two windows appear. Move the Intensity slider in the control window and watch the output change.

Release mode matters for real-time work. Debug builds can be 10–20× slower for GPU-bound effects.

What just happened?

rustjay_engine::run() did a lot for you:

1. Opened a wgpu device on your default GPU
2. Loaded your DesaturateUniforms uniform block and compiled your WGSL shader
3. Opened the webcam (or showed black if none available)
4. Created the control window with all built-in tabs
5. Added an Intensity slider to the built-in parameter list based on your parameters() declaration
6. Starts the render loop, calling build_uniforms() every frame

Read The Two Windows for a tour of what's in the control window, and The EffectPlugin Trait for a deep dive into the full API.

The Two Windows

Every rustjay-engine app opens two windows on startup.

The Output window

The output window is your GPU canvas. It renders your effect full-resolution every frame, with no UI chrome — just pixels.

Keyboard shortcuts (focus on the output window):

When you go fullscreen, the control window stays open on whichever screen it's on — send it to your laptop display while the output window fills a projector.

The Control window

The control window is an ImGui panel with a row of tabs. The built-in tabs are:

Input tab

Select the video source for @group(0) @binding(0):

• Webcam — any connected capture device; pick by name or index
• NDI — receive from any NDI source on the LAN (requires ndi feature)
• Syphon (macOS) — receive from any Syphon server (VDMX, Resolume, other effects)
• Spout (Windows) — same idea, DirectX-based
• V4L2 (Linux) — virtual or physical video device

If no source is selected, the input texture is solid black.

Audio tab

Shows the live audio input waveform and 8-band FFT spectrum, current BPM estimate, beat phase progress bar, and the Tempo & Sync section for tempo source selection.

LFO tab

Three LFO banks (A, B, C), each with:

• Waveform selector (Sine, Triangle, Saw, Square, Noise)
• Rate — Hz or beat division
• Depth — modulation amount
• Target — which declared parameter to modulate

See LFOs for details.

MIDI tab

Shows connected MIDI devices. Click Learn on any parameter to arm CC learn mode — move a knob on your controller to assign it. Shows the current CC mapping for each parameter.

OSC tab

Displays the OSC server address and port (default 0.0.0.0:7770). Parameters are addressable as /rustjay/<param-id>.

Presets tab

Save and load snapshots of the full engine state + your effect state. Eight quick-slots correspond to Shift+F1–Shift+F8 on the output window.

Output tab

Configure where your rendered frames go beyond the output window:

• NDI — publish to the LAN as an NDI source
• Syphon / Spout — share with other VJ apps
• V4L2 — write to a loopback device

Sync tab (optional)

Only visible when the link or prodj feature is enabled. Lets you choose the active tempo source (Audio / Ableton Link / ProDJ Link) and shows the current Link session or ProDJ deck list.

Custom tabs

Your effect can add its own tab to the control window, or replace a built-in tab. See Custom Tabs.

The EffectPlugin Trait

EffectPlugin is the central abstraction of rustjay-engine. It's a trait you implement once per app, and the engine calls its methods at the right times during setup and the render loop.

#![allow(unused)]
fn main() {
pub trait EffectPlugin: Send + Sync + 'static {
    type State:    Default + Send + Sync + Serialize + DeserializeOwned + 'static;
    type Uniforms: bytemuck::Pod + bytemuck::Zeroable;

    // Required
    fn shader_source(&self)                                     -> &'static str;
    fn build_uniforms(&self, state: &Self::State, engine: &EngineState) -> Self::Uniforms;

    // Common optional overrides
    fn app_name(&self)      -> &str                            { "rustjay" }
    fn default_state(&self) -> Self::State                     { Default::default() }
    fn parameters(&self)    -> Vec<ParameterDescriptor>        { vec![] }
    fn hidden_tabs(&self)   -> Vec<GuiTab>                     { vec![] }

    // Dynamic parameter lists
    fn parameters_dirty(&self)        -> bool  { false }
    fn clear_parameters_dirty(&mut self)       {}

    // Lifecycle hooks
    fn init(&mut self, device: &wgpu::Device, queue: &wgpu::Queue) {}
    fn prepare(&mut self, state: &mut Self::State, engine: &EngineState,
               device: &wgpu::Device, queue: &wgpu::Queue) {}

    // Rendering overrides
    fn render_graph(&self)           -> Option<RenderGraph>   { None }
    fn mesh_descriptor(&self, state: &Self::State) -> Option<MeshDescriptor> { None }
    fn vertex_reads_texture(&self)   -> bool                  { false }
    fn compute_shader(&self)         -> Option<&'static str>  { None }
    fn render(&mut self, ...) -> bool                         { false }
}
}

Associated types

State

Your app's mutable runtime state. The engine owns one instance of this, passes &State to build_uniforms() every frame, and passes &mut State to prepare() and to your custom GUI tab's draw() method.

Requirements:

• Default — the engine creates the initial state with default_state() (which calls Default::default() unless you override it)
• Serialize + DeserializeOwned — the preset system serialises this to JSON when saving and restores it when loading

A typical state struct:

#![allow(unused)]
fn main() {
#[derive(Default, serde::Serialize, serde::Deserialize)]
struct MyState {
    intensity: f32,
    hue_shift: f32,
    enabled:   bool,
}
}

Uniforms

The GPU-side data block uploaded to @group(1) @binding(0) every frame. Must be:

• #[repr(C)] — stable field layout for bytemuck
• bytemuck::Pod + bytemuck::Zeroable — safe transmute to bytes

#![allow(unused)]
fn main() {
#[repr(C)]
#[derive(Copy, Clone, bytemuck::Pod, bytemuck::Zeroable)]
struct MyUniforms {
    intensity: f32,
    hue_shift: f32,
    _pad:      [f32; 2],   // pad to 16-byte alignment
}
}

Alignment: wgpu requires uniform buffers to be 16-byte aligned. If your struct's size isn't a multiple of 16 bytes, add padding fields.

Required methods

shader_source()

Returns the WGSL source for your fragment shader. Use include_str! to embed a file at compile time:

#![allow(unused)]
fn main() {
fn shader_source(&self) -> &'static str {
    include_str!("shaders/my_effect.wgsl")
}
}

build_uniforms()

Called every frame. Reads from your State and EngineState to produce the Uniforms value that gets uploaded to the GPU:

#![allow(unused)]
fn main() {
fn build_uniforms(&self, s: &MyState, engine: &EngineState) -> MyUniforms {
    MyUniforms {
        intensity: engine.get_param("intensity").unwrap_or(s.intensity),
        hue_shift: s.hue_shift,
        _pad:      [0.0; 2],
    }
}
}

Call engine.get_param(id) to read a parameter with LFO and audio modulations already applied. See EngineState for the full API.

Lifecycle hooks

init(device, queue)

Called once after the wgpu device is ready. Use this to create extra textures, bind groups, or pipelines that the default single-pass setup can't express.

#![allow(unused)]
fn main() {
fn init(&mut self, device: &wgpu::Device, _queue: &wgpu::Queue) {
    let texture = device.create_texture(&wgpu::TextureDescriptor { /* ... */ });
    self.extra_tex = Some(texture);
}
}

prepare(state, engine, device, queue)

Called every frame, before the render pass. Use this for per-frame GPU resource updates — writing to a texture, updating a compute buffer — that aren't handled by the uniform upload.

parameters_dirty() / clear_parameters_dirty()

For effects whose parameter list can change at runtime (e.g. a shader hot-reloader that swaps inputs when a new file loads), implement these two methods together:

#![allow(unused)]
fn main() {
fn parameters_dirty(&self) -> bool {
    self.params_changed
}

fn clear_parameters_dirty(&mut self) {
    self.params_changed = false;
}
}

When parameters_dirty() returns true, the engine re-calls parameters(), swaps out EngineState::param_descriptors, and resizes the parameter value arrays — preserving existing values for any param IDs that survive the change. clear_parameters_dirty() is called immediately after so the flag is reset before the next frame.

Set the flag inside init() (after a successful pipeline rebuild) so the engine picks up the new list on the very next frame:

#![allow(unused)]
fn main() {
fn init(&mut self, device: &wgpu::Device, _queue: &wgpu::Queue) {
    // ... compile pipeline ...
    self.params_changed = true;
}
}

Declaring parameters

#![allow(unused)]
fn main() {
fn parameters(&self) -> Vec<ParameterDescriptor> {
    vec![
        ParameterDescriptor::float(
            "intensity", "Intensity",      // id, display name
            ParamCategory::Color,          // tab grouping
            0.0, 1.0, 0.5, 0.01,          // min, max, default, step
        ),
        ParameterDescriptor::int(
            "blend_mode", "Blend Mode",
            ParamCategory::Motion,
            0, 7, 0, 1,
        ),
    ]
}
}

Declared parameters:

• Appear as sliders in the built-in control UI
• Can be targeted by LFO banks
• Can be mapped to MIDI CC via learn mode
• Are addressable as OSC messages at /rustjay/<id>
• Receive audio-reactive modulation from the routing matrix

Read them back in build_uniforms() via engine.get_param(id), which returns the base value plus all active modulations.

Hiding built-in tabs

If your effect doesn't use colour parameters, you can hide the Color tab to keep the UI clean:

#![allow(unused)]
fn main() {
fn hidden_tabs(&self) -> Vec<GuiTab> {
    vec![GuiTab::Color]
}
}

Available tabs: GuiTab::Input, GuiTab::Audio, GuiTab::Lfo, GuiTab::Midi, GuiTab::Osc, GuiTab::Output, GuiTab::Presets, GuiTab::Color, GuiTab::Sync.

Entry points

Two engine entry points are available:

#![allow(unused)]
fn main() {
// Simple — no custom tabs
rustjay_engine::run(MyEffect)

// With custom control-window tabs
rustjay_engine::run_with_tabs(MyEffect, vec![Box::new(MyTab)])
}

See Custom Tabs for how to implement AnyGuiTab.

EngineState

EngineState is the engine's live runtime state. It's passed to build_uniforms() every frame and to prepare(). Reading from it is how your effect reacts to audio, LFOs, tempo, and the current parameter values.

Audio state

#![allow(unused)]
fn main() {
let volume     = engine.audio.volume;       // f32 — RMS loudness [0, 1]
let bass       = engine.audio.fft[0];       // f32 — lowest FFT band [0, 1]
let kick       = engine.audio.fft[1];       // f32 — next band up
let beat_pulse = engine.audio.beat_pulse;   // bool — true on beat onset
let bpm        = engine.audio.bpm;          // f32 — estimated BPM (audio only)
let beat_phase = engine.audio.beat_phase;   // f32 — [0, 1) position within the current beat
}

fft is an array of 8 bands covering the audible spectrum from bass to treble. Band 0 is the lowest frequencies; band 7 is the highest.

When tempo sync is active, prefer engine.effective_bpm() and engine.effective_beat_phase() over engine.audio.bpm. See Tempo Sync.

Parameters

The get_param method returns a parameter's effective value — base slider value plus all active modulations (LFO, audio routing):

#![allow(unused)]
fn main() {
let intensity = engine.get_param("intensity").unwrap_or(0.5);
}

Returns None if no parameter with that id is registered. The returned value is already clamped to the parameter's declared [min, max] range.

LFO state

You rarely need to read LFO state directly in build_uniforms() — declare parameters and get_param() includes LFO contributions automatically. But you can read raw LFO values if you need them:

#![allow(unused)]
fn main() {
let lfo_a_value = engine.lfo.banks[0].current_value; // f32 [-1, 1]
}

See LFOs.

Tempo

#![allow(unused)]
fn main() {
// Always dispatches on the active sync source:
let bpm   = engine.effective_bpm();
let phase = engine.effective_beat_phase(); // [0, 1)

// Which source is active:
match engine.sync_source {
    SyncSource::Audio => { /* audio beat detection */ }
    SyncSource::Link  => { /* Ableton Link session  */ }
    SyncSource::ProDj => { /* ProDJ Link            */ }
}
}

effective_bpm() and effective_beat_phase() are the right calls for any tempo-reactive effect. They follow the active source automatically.

MIDI Timecode (optional)

When the mtc feature is enabled:

#![allow(unused)]
fn main() {
if let Some(pos) = &engine.mtc.position {
    let seconds = pos.as_seconds_f64();
    let frame_rate = pos.frame_rate; // 24, 25, 29.97, 30
}
}

MTC is a position reference, not a BPM source — use it for timeline-locked visuals.

Input / output state

#![allow(unused)]
fn main() {
// Current input dimensions
let (w, h) = (engine.input.width, engine.input.height);

// Whether an input source is active
let has_input = engine.input.active;
}

Sending commands

EngineState uses command enums rather than direct mutation to change subsystem state. Commands are typically sent from a custom GUI tab, not from build_uniforms(). The engine processes them at the start of the next frame.

#![allow(unused)]
fn main() {
engine.input_commands.push(InputCommand::SetDevice(DeviceId(0)));
engine.output_commands.push(OutputCommand::EnableNdi(true));
engine.lfo_commands.push(LfoCommand::SetRate { bank: 0, hz: 1.0 });
}

You won't need these in simple effects, but they're how the built-in tabs work internally.

Uniforms & Shaders

The standard binding layout

Every rustjay-engine shader shares a consistent binding layout. If you deviate from it, the engine won't be able to set up the bind groups it needs.

Group 0 — Video input

@group(0) @binding(0) var input_tex:     texture_2d<f32>;
@group(0) @binding(1) var input_sampler: sampler;

The engine always provides these two bindings. input_tex is the current video frame from whatever source is active (webcam, NDI, Syphon, etc.). input_sampler is a bilinear clamp sampler.

When a RenderGraph with feedback is active, two more bindings are added at group 0:

@group(0) @binding(2) var feedback_tex:     texture_2d<f32>; // previous frame output
@group(0) @binding(3) var feedback_sampler: sampler;

Group 1 — Your uniforms

@group(1) @binding(0) var<uniform> u: MyUniforms;

One uniform buffer containing your Uniforms struct. The engine uploads whatever build_uniforms() returns each frame.

Uniform struct rules

#![allow(unused)]
fn main() {
#[repr(C)]
#[derive(Copy, Clone, bytemuck::Pod, bytemuck::Zeroable)]
struct MyUniforms {
    intensity: f32,
    hue:       f32,
    _pad:      [f32; 2],   // pad to 16 bytes
}
}

Three requirements:

1. #[repr(C)] — guarantees stable field ordering and no padding surprises
2. bytemuck::Pod — allows safe transmutation to &[u8] for the GPU upload
3. bytemuck::Zeroable — allows zeroing the buffer before your first build_uniforms() call

16-byte alignment: wgpu requires uniform buffers to be multiples of 16 bytes in size. Structs smaller than 16 bytes need explicit padding fields. The _pad: [f32; N] pattern is idiomatic.

The vertex shader

The engine provides a full-screen quad (two triangles covering the NDC unit square). Your vertex shader receives:

@location(0) position: vec2<f32>  // NDC position [-1, 1]
@location(1) texcoord: vec2<f32>  // UV [0, 1], (0,0) = top-left

The standard vertex shader is boilerplate and almost never changes:

struct VertexOutput {
    @builtin(position) position: vec4<f32>,
    @location(0) texcoord: vec2<f32>,
};

@vertex
fn vs_main(
    @location(0) position: vec2<f32>,
    @location(1) texcoord: vec2<f32>,
) -> VertexOutput {
    var out: VertexOutput;
    out.position = vec4<f32>(position, 0.0, 1.0);
    out.texcoord = texcoord;
    return out;
}

Sampling the video input

@fragment
fn fs_main(in: VertexOutput) -> @location(0) vec4<f32> {
    let col = textureSample(input_tex, input_sampler, in.texcoord);
    // ... process col ...
    return col;
}

in.texcoord is [0,1]² with (0,0) at the top-left. The sampler clamps at the border — sampling outside [0,1] returns the edge pixel colour.

Encoding data in uniforms

Floats

Direct: intensity: f32, bpm: f32, etc.

Booleans

wgpu doesn't support bool in uniform buffers. Use u32:

#![allow(unused)]
fn main() {
struct MyUniforms { enabled: u32, _pad: [f32; 3] }
// In build_uniforms:
enabled: if s.enabled { 1 } else { 0 },
}

struct MyUniforms { enabled: u32 };
// In shader:
if u.enabled != 0u { /* ... */ }

Enums / modes

Use u32 for discrete choices:

#![allow(unused)]
fn main() {
struct MyUniforms { blend_mode: u32, _pad: [f32; 3] }
}

switch u.blend_mode {
    case 0u: { /* Replace */ }
    case 1u: { /* Add */     }
    default: { /* ... */     }
}

Colours

Use vec4<f32> (RGBA) or pack channels into a [f32; 4] array:

#![allow(unused)]
fn main() {
#[repr(C)]
#[derive(Copy, Clone, bytemuck::Pod, bytemuck::Zeroable)]
struct MyUniforms {
    values: [f32; 4],  // e.g. [hue_shift, saturation, brightness, unused]
}
}

struct MyUniforms { values: vec4<f32> };
let hue = u.values.x;
let sat = u.values.y;

UV helpers

Common UV transforms:

// Flip V (some sources come upside-down)
let flipped = vec2<f32>(in.texcoord.x, 1.0 - in.texcoord.y);

// Centre coordinates ([-0.5, 0.5]² )
let centred = in.texcoord - vec2<f32>(0.5);

// Aspect-correct coordinates (account for non-square input)
// Requires width/height in uniforms
let aspect = f32(u.width) / f32(u.height);
let uv = vec2<f32>((in.texcoord.x - 0.5) * aspect, in.texcoord.y - 0.5);

Built-in Tabs

rustjay-engine's control window comes with a set of tabs that cover the most common live-performance needs. You don't have to implement any of these yourself — they work out of the box.

What the built-in tabs do

Hiding tabs you don't need

If your effect doesn't use colour parameters, hiding the Color tab keeps the UI focused:

#![allow(unused)]
fn main() {
fn hidden_tabs(&self) -> Vec<GuiTab> {
    vec![GuiTab::Color]
}
}

The full list of hide-able tabs mirrors the constants in GuiTab.

Parameter sliders

When you declare parameters in parameters(), they appear automatically in the built-in control UI — no extra code needed. The engine groups them by ParamCategory:

Each declared parameter gets:

• A slider in the appropriate tab
• An entry in the MIDI learn list
• An OSC address (/rustjay/<id>)
• Availability as an LFO target in the LFO tab
• Slot in the audio routing matrix

Next: Custom Tabs — adding your own panel to the control window.

Custom Tabs

Custom tabs let you add your own panel to the control window. Use them when you need controls that don't fit the standard parameter-slider model: toggle groups, waveform previews, complex layouts, or widgets that act on multiple parameters at once.

Implementing AnyGuiTab

#![allow(unused)]
fn main() {
use rustjay_engine::prelude::*;

struct MyTab;

impl AnyGuiTab for MyTab {
    fn name(&self) -> &str { "My Effect" }

    fn draw(
        &mut self,
        ui:         &imgui::Ui,
        app_state:  &mut dyn std::any::Any,
        engine:     &mut EngineState,
    ) {
        // Downcast app_state to your concrete type
        let state = app_state
            .downcast_mut::<MyState>()
            .expect("MyTab: wrong state type");

        // Draw ImGui widgets
        ui.slider_config("Intensity", 0.0_f32, 1.0_f32)
            .build(&mut state.intensity);

        if ui.button("Reset") {
            state.intensity = 0.0;
        }
    }
}
}

Register it with run_with_tabs:

fn main() -> anyhow::Result<()> {
    rustjay_engine::run_with_tabs(MyEffect, vec![Box::new(MyTab)])
}

The tab appears at the right end of the tab bar.

Replacing a built-in tab

If your effect has its own colour controls and you want one custom "Effect" tab instead of the built-in Color tab:

#![allow(unused)]
fn main() {
impl AnyGuiTab for MyTab {
    fn name(&self) -> &str { "Effect" }

    fn replaces(&self) -> Option<BuiltinTab> {
        Some(BuiltinTab::Color)  // hides the built-in Color tab
    }

    fn draw(&mut self, ui: &imgui::Ui, app_state: &mut dyn std::any::Any, _engine: &mut EngineState) {
        let state = app_state.downcast_mut::<MyState>().unwrap();
        ui.slider_config("Hue Shift", -180.0_f32, 180.0_f32).build(&mut state.hue_shift);
        ui.slider_config("Saturation", 0.0_f32, 2.0_f32).build(&mut state.saturation);
    }
}
}

ImGui widget reference

A quick reference of frequently used ImGui widgets:

#![allow(unused)]
fn main() {
// Sliders
ui.slider_config("Label", min, max).build(&mut state.value);

// Drag (finer control, no visible range)
imgui::Drag::new("Label").speed(0.01).build(ui, &mut state.value);

// Checkbox
ui.checkbox("Enabled", &mut state.enabled);

// Combo (dropdown)
let items = ["Replace", "Add", "Multiply", "Screen"];
let mut current = state.blend_mode as usize;
if ui.combo_simple_string("Blend Mode", &mut current, &items) {
    state.blend_mode = current as u32;
}

// Button
if ui.button("Randomise") { /* ... */ }

// Colour picker (returns [f32; 4])
ui.color_edit4_config("Tint", imgui::ColorEditFlags::NO_ALPHA)
  .build(&mut state.tint);

// Text
ui.text(format!("BPM: {:.1}", engine.effective_bpm()));

// Separator + header
ui.separator();
ui.text_colored([1.0, 0.8, 0.2, 1.0], "-- Section --");
}

Using the egui backend

If you're building with the egui feature, use EguiAnyTab instead and draw with egui::Ui. The pattern is identical — implement the trait, downcast app_state, draw widgets — but the widget API differs.

// Cargo.toml: rustjay-engine = { ..., features = ["egui"] }

use rustjay_engine::prelude::*;

struct MyEguiTab;

impl EguiAnyTab for MyEguiTab {
    fn name(&self) -> &str { "Effect" }

    fn draw(
        &mut self,
        ui:        &egui::Ui,
        app_state: &mut dyn std::any::Any,
        engine:    &mut EngineState,
    ) {
        let state = app_state.downcast_mut::<MyState>().unwrap();
        ui.add(egui::Slider::new(&mut state.intensity, 0.0..=1.0).text("Intensity"));
    }
}

fn main() -> anyhow::Result<()> {
    rustjay_engine::run_with_tabs(MyEffect, vec![Box::new(MyEguiTab)])
}

See examples/delta-egui for a complete egui-backend example.

Audio Analysis

rustjay-engine captures audio from the system default input and analyses it every frame. The results are available in EngineState::audio from build_uniforms() and prepare().

The audio state

#![allow(unused)]
fn main() {
pub struct AudioState {
    pub volume:      f32,       // RMS loudness, normalised [0, 1]
    pub fft:         [f32; 8],  // 8-band spectrum, [0, 1] each
    pub beat_pulse:  bool,      // true on beat onset this frame
    pub bpm:         f32,       // estimated BPM
    pub beat_phase:  f32,       // position within current beat [0, 1)
}
}

Volume

volume tracks the RMS loudness of the current audio frame, normalised to [0, 1]. It responds quickly — useful for direct amplitude-reactive effects.

FFT bands

fft splits the audio spectrum into 8 bands from bass to treble. Band 0 captures the lowest frequencies (sub-bass / kick drum energy), band 7 captures the highest (hi-hat / air).

#![allow(unused)]
fn main() {
let bass   = engine.audio.fft[0]; // kick, sub-bass
let mid    = engine.audio.fft[3]; // midrange, snare
let treble = engine.audio.fft[7]; // hi-hat, brightness
}

All band values are normalised to [0, 1].

Beat detection

beat_pulse is true for exactly one frame when a beat onset is detected. Use it to trigger instantaneous events:

#![allow(unused)]
fn main() {
fn build_uniforms(&self, s: &MyState, engine: &EngineState) -> MyUniforms {
    let flash = if engine.audio.beat_pulse { 1.0_f32 } else { 0.0_f32 };
    // fade flash out in the shader using a time uniform
    MyUniforms { flash, .. }
}
}

Beat phase

beat_phase is a sawtooth [0, 1) that resets to 0 on each detected beat. It's useful for smooth beat-locked animations.

#![allow(unused)]
fn main() {
let phase = engine.audio.beat_phase; // ramps 0→1 between beats
}

Using audio in your effect

Direct audio reactivity

Read FFT bands or volume directly in build_uniforms():

#![allow(unused)]
fn main() {
fn build_uniforms(&self, s: &MyState, engine: &EngineState) -> MyUniforms {
    let bass = engine.audio.fft[0];
    MyUniforms {
        displacement: s.displacement_amount * bass,
        brightness:   0.5 + engine.audio.volume * 0.5,
        // ...
    }
}
}

Via the routing matrix

For a more flexible setup — letting the user choose which FFT band drives which parameter at runtime — use the Routing Matrix. Routing contributions are already included when you call engine.get_param().

Tap tempo

The Audio tab has a tap-tempo button. The user can also tap Shift+T on the output window. Tap tempo overrides the audio BPM estimate but is itself overridden by Ableton Link or ProDJ Link if those sources are active.

For tempo-reactive effects, always use engine.effective_bpm() rather than engine.audio.bpm — it accounts for whichever source is active. See Tempo Sync.

LFOs

rustjay-engine provides 8 LFO slots. Each slot is an independent oscillator that can be assigned to any declared parameter at runtime — from the Modulation tab (desktop) or the Modulation web panel (headless).

Configuration

Each slot exposes:

Waveforms

Depth scales the [-1, 1] oscillator output before it is added to the parameter's base value. A depth of 0.5 means the LFO swings ±0.5 units from whatever the base value is set to.

Beat-sync mode

When Tempo Sync is on, the division field replaces the rate field. The engine converts it using effective_bpm():

cycle_duration_seconds = (60 / bpm) * beats_per_cycle

At 120 BPM, a 1/4 division (0.25 beats) runs at 8 Hz; a 1 beat division runs at 2 Hz; a 4 beat division runs at 0.5 Hz.

The LFO phase advances freely based on wall-clock delta time — it does not directly track beat_phase. When using Ableton Link or ProDJ Link, the phase snaps to the quantum boundary on each beat crossing to stay musically in phase. With audio beat detection or tap tempo, the phase resets freely and may drift relative to the actual beat.

Tap tempo

On headless Pi setups without Ableton Link, use Tap Tempo in the Modulation web panel to set the BPM manually. Tap twice for an immediate estimate; subsequent taps refine the average over up to 8 intervals. The BPM display updates after each tap.

Tap tempo writes to audio.bpm, which effective_bpm() returns when the active sync source is Audio.

Reading LFO values in code

When you call engine.get_param(id), LFO modulation is already included — you don't need to read the LFO state directly.

If you need the raw oscillator output (e.g. to drive something outside the parameter system):

#![allow(unused)]
fn main() {
fn build_uniforms(&self, s: &MyState, engine: &EngineState) -> MyUniforms {
    let lfo_0 = engine.lfo.bank.lfos[0].output; // f32 in [-amplitude, amplitude]
    let lfo_1 = engine.lfo.bank.lfos[1].output;
    MyUniforms {
        wobble: lfo_0,
        // ...
    }
}
}

output is in [-amplitude, amplitude] — it is the raw waveform value multiplied by depth.

Phase continuity on config update

When an LFO's configuration is changed via the web Modulation panel, the engine preserves the current phase, output, and last_beat_phase values from the existing slot before applying the new config. A running LFO does not snap back to phase 0 when you adjust its waveform or depth mid-cycle.

Targeting effect-declared parameters

LFO targets fall into two groups:

Built-in targets — HueShift, Saturation, Brightness. These modulate the HSB colour correction layer common to all effects.

Custom targets — any parameter declared by the effect via ParameterDescriptor. In the web Modulation panel, all current effect parameters appear in the Target dropdown under their category name (e.g. Flux / Flow Scale). Internally, these are stored as LfoTarget::Custom("flow_scale") using the bare parameter ID — the category prefix is stripped before storage.

Multiple LFOs on one parameter

If two or more enabled slots target the same parameter, their outputs are summed before being applied. The summed modulation is still clamped to the parameter's [min, max] range.

Routing Matrix

The routing matrix lets the user map FFT bands to parameters at runtime. When a band is routed to a parameter, its value is added to the parameter's base slider value each frame (after LFO modulation).

How it works

At runtime, the user opens the Audio tab and assigns one of the 8 FFT bands to any declared parameter. They also set a gain and a response curve (linear, squared, etc.) for that mapping.

When engine.get_param("intensity") is called, the engine computes:

effective = base_value + lfo_contribution + (fft_band * gain * curve)

The result is clamped to the parameter's declared [min, max] range.

Your plugin code sees only the final value and doesn't need to know about the routing.

In practice

Declare your parameters, use get_param() in build_uniforms(), and the routing matrix is available automatically:

#![allow(unused)]
fn main() {
fn parameters(&self) -> Vec<ParameterDescriptor> {
    vec![
        ParameterDescriptor::float("brightness", "Brightness", ParamCategory::Color, 0.0, 2.0, 1.0, 0.01),
        ParameterDescriptor::float("hue_shift",  "Hue Shift",  ParamCategory::Color, -180.0, 180.0, 0.0, 1.0),
    ]
}

fn build_uniforms(&self, s: &MyState, engine: &EngineState) -> MyUniforms {
    MyUniforms {
        brightness: engine.get_param("brightness").unwrap_or(s.brightness),
        hue_shift:  engine.get_param("hue_shift").unwrap_or(s.hue_shift),
    }
}
}

With this in place, the user can go to the Audio tab and set "bass band → brightness" with a gain of 0.8, and the brightness will pulse with the kick drum.

Bypassing the routing matrix

If you want to read a raw FFT value and apply your own response curve, read from engine.audio.fft directly:

#![allow(unused)]
fn main() {
let kick_energy = engine.audio.fft[0].powf(0.5); // square-root curve
}

This skips the routing matrix entirely — useful for hard-coded audio reactivity where the user shouldn't need to configure anything.

Tempo Sync

rustjay-engine can lock to three external tempo sources. Enabling them is a feature-flag choice; selecting between them at runtime is a user choice from the Audio tab.

Sources

Audio beat detection (always available)

The default. The engine analyses the audio input to estimate BPM and track beat phase.

Tap tempo is also available to set BPM manually:

• Desktop: Shift+T on the output window, or the tap button in the Audio tab
• Headless / web: Tap button in the Modulation panel — two taps give an immediate estimate; up to 8 taps refine the average. The BPM display updates after each tap.

No features needed. Works offline. Quality depends on the audio signal.

Ableton Link (link feature)

Joins the local Link session — synchronises BPM and beat phase with Live, Serato, Traktor, or any other Link-enabled app on the same network.

[dependencies]
rustjay-engine = { git = "...", features = ["link"] }

Requires CMake (brew install cmake / apt install cmake). Linking against Ableton Link makes the resulting binary GPL-2.0+.

When Link peers are present, the engine joins the session automatically once the user activates the Link source. The Sync tab shows the current peer count, beat phase bar, and quantum (loop length) slider.

ProDJ Link (prodj feature)

Receives BPM, beat phase, and track metadata from Pioneer CDJ/XDJ/DJM gear on the same LAN.

[dependencies]
rustjay-engine = { git = "...", features = ["prodj"] }

No extra system dependencies. Binds UDP ports 50000 and 50002 — get operator approval before using on a production DJ network. The Sync tab shows the connected decks and which is the master.

MIDI Timecode (mtc feature)

Decodes SMPTE timecode from any connected MIDI device. MTC is a position reference, not a BPM source — use it for timeline-locked visuals rather than beat-reactive effects.

[dependencies]
rustjay-engine = { git = "...", features = ["mtc"] }

No extra system dependencies. Listens on all MIDI ports simultaneously.

Using multiple features

Any combination works:

rustjay-engine = { git = "...", features = ["link", "prodj", "mtc"] }

The user selects the active source at runtime from the Audio tab.

Writing sync-aware plugins

Use effective_bpm() and effective_beat_phase() instead of the audio.* fields:

#![allow(unused)]
fn main() {
fn build_uniforms(&self, s: &MyState, engine: &EngineState) -> MyUniforms {
    MyUniforms {
        bpm:        engine.effective_bpm(),
        beat_phase: engine.effective_beat_phase(),
        // ...
    }
}
}

These dispatch on engine.sync_source automatically — your plugin works correctly regardless of which source the user has selected.

Beat-locked animation

A common pattern: animate a value based on beat phase so it resets on each beat.

#![allow(unused)]
fn main() {
let phase = engine.effective_beat_phase(); // [0, 1)
let flash = (1.0 - phase).powf(4.0);      // bright on beat, decays quickly
}

In the shader:

let brightness = u.flash * 2.0 + base_brightness;

BPM-locked oscillation

Use BPM to drive a shader oscillation without LFOs:

#![allow(unused)]
fn main() {
// Pass BPM and time to the shader and oscillate at a beat subdivision
let bpm   = engine.effective_bpm();
let phase = engine.effective_beat_phase();
MyUniforms { bpm, beat_phase: phase, .. }
}

// In shader: oscillate at 2× the beat rate
let osc = sin(u.beat_phase * 2.0 * 3.14159 * 2.0);

Reading MIDI Timecode position

#![allow(unused)]
fn main() {
fn build_uniforms(&self, s: &MyState, engine: &EngineState) -> MyUniforms {
    let time_seconds = engine.mtc.position
        .as_ref()
        .map(|p| p.as_seconds_f64() as f32)
        .unwrap_or(0.0);
    MyUniforms { time_seconds, .. }
}
}

Video I/O

rustjay-engine supports multiple video input and output protocols. All device discovery runs in a background thread — the GPU render loop never blocks waiting for a camera or network source.

Video inputs

Configure the input source from the Input tab at runtime. Your shader always receives the current frame at @group(0) @binding(0) regardless of which source is active.

Webcam

Any V4L2 (Linux), AVFoundation (macOS), or DirectShow (Windows) capture device. The Input tab lists available devices by name; select one and the engine opens it at its native resolution.

No feature flag needed.

NDI

Network Device Interface — receive video from any NDI source on the LAN (OBS, Resolume, another rustjay-engine instance).

Requires the NDI SDK and the ndi feature (on by default).

The Input tab scans the LAN and shows available sources. Low latency, high resolution, suitable for multi-machine setups.

Syphon (macOS only)

Receive frames from any Syphon server on the same machine — VDMX, Resolume, Final Cut Pro, another rustjay-engine app.

Requires Syphon.framework in /Library/Frameworks/. No feature flag (always compiled on macOS).

Zero-copy GPU texture sharing — no CPU round-trip.

Spout (Windows only)

The Windows equivalent of Syphon. Receive frames from Resolume, MadMapper, or any Spout-compatible app.

Uses DirectX texture sharing. No feature flag (always compiled on Windows).

V4L2 (Linux only)

Receive from any /dev/videoN device, including loopback devices created by the v4l2loopback kernel module. Can receive video piped from OBS or ffmpeg.

Video outputs

Configure from the Output tab.

NDI output

Publish the rendered output as an NDI source on the LAN. Other NDI receivers see it immediately.

rustjay-engine = { git = "...", features = ["ndi"] }

Syphon output (macOS)

Publish as a Syphon server. VDMX, Resolume, and other VJ apps on the same machine can receive and further process the output.

Spout output (Windows)

Publish as a Spout sender for DirectX-based apps on the same machine.

V4L2 output (Linux)

Write frames to a V4L2 loopback device (/dev/videoN) so tools like OBS, ffmpeg, or Zoom can receive the output as a virtual camera.

# Create a loopback device first
sudo modprobe v4l2loopback

Resolution and frame rate

The output window renders at the resolution it was created at (typically display DPI-scaled). Output protocols (NDI, Syphon, Spout) capture the rendered frame at that resolution.

Frame rate is uncapped by default and limited by GPU rendering time. Set a target rate in the engine config if you need to control pacing.

No input

If no source is selected, the engine provides a solid-black texture at group 0 binding 0. Your shader still runs — useful for generative effects that don't need video input.

External Control

rustjay-engine supports three protocols for controlling parameters from external devices and software: MIDI, OSC, and a web remote. All three are active by default when the engine starts — no code changes needed.

Web remote

The engine runs a WebSocket + HTTP server (default port 8081) that provides a full browser-based control surface. Any phone or laptop on the same network can open it without installing anything.

Access

On startup, the engine logs a URL with an embedded bearer token:

Web server ready:
  Local:   http://127.0.0.1:8081/flux?token=a1b2c3d4...
  Network: http://192.168.1.42:8081/flux?token=a1b2c3d4...

Open the network URL in any browser. The token is regenerated each launch.

LAN trust mode

For headless operation (Pi, rack unit) where typing a token is inconvenient, enable LAN trust mode in the app config:

{
  "web_host": "0.0.0.0",
  "web_port": 8081,
  "web_lan_trust": true
}

With LAN trust active, all requests from the local subnet pass through without authentication. Disable it on untrusted networks.

Panels

Five panels are available from the toolbar on the main page:

Each panel opens in its own tab. All panels share a single WebSocket connection and receive live state updates.

Command protocol

Panels write commands via POST /<app>/cmd with a JSON body. The outer type field selects the subsystem; the action field selects the operation within it.

Parameter value:

{"type":"set","id":"flux/flow_scale","value":1.8}

Input (webcam):

{"type":"input","action":"select_device","index":0,"width":720,"height":576,"fps":25}
{"type":"input","action":"stop"}
{"type":"input","action":"refresh_devices"}

OSC / MIDI control:

{"type":"control","action":"osc","enabled":true}
{"type":"control","action":"osc_set_port","port":9001}
{"type":"control","action":"midi_learn","param_id":"flux/flow_scale"}
{"type":"control","action":"midi_learn_cancel"}
{"type":"control","action":"midi_unlearn","cc":14,"channel":0}
{"type":"control","action":"midi_select_device","device":"Arturia BeatStep"}
{"type":"control","action":"midi_disconnect"}

LFO / modulation:

{"type":"modulation","action":"lfo_enable","slot":0,"enabled":true}
{"type":"modulation","action":"lfo_set","slot":0,"config":{"index":0,"enabled":true,"target":"HueShift","waveform":"Sine","amplitude":0.5,"tempo_sync":true,"division":4,"rate":1.0,"phase_offset":0.0}}
{"type":"modulation","action":"tap_tempo"}

Presets:

{"type":"preset","action":"save","name":"Deep Blue Session"}
{"type":"preset","action":"load","index":0}
{"type":"preset","action":"delete","index":0}
{"type":"preset","action":"list"}

State broadcasts

The WebSocket pushes JSON messages to all connected panels. The type field identifies the message:

MIDI

Device setup

Connect a USB MIDI controller. Available devices appear in the Control panel's MIDI Devices section. Click Connect to start receiving.

Alternatively, connect from the GUI MIDI tab (desktop) or the web Control panel (headless).

CC learn mode

Desktop (GUI): open the MIDI tab, click Learn next to a parameter, move a knob or fader.

Headless (web Control panel):

1. Open http://<pi>:8081/<app>/control
2. Scroll to the Parameters section
3. Click Learn next to any parameter
4. Move a CC on the controller — the mapping appears immediately

The LEARNING… badge at the top of the Parameters section shows when learn mode is active. Click Cancel to abort.

Persistence

MIDI device selection and all CC mappings are saved automatically to the per-app config file every time a mapping changes. They are restored on the next launch, including reconnecting to the same device if it is present.

OSC

The engine runs an OSC server on 0.0.0.0:9000 by default (configurable from the Control panel or config file).

Parameter addresses

Every declared parameter is addressable as:

/<app-base-address>/<category>/<param-id>  f32

For example, with the default base address /rustjay and a parameter declared as:

#![allow(unused)]
fn main() {
ParameterDescriptor::float("flow_scale", ...)
    .category(ParamCategory::Flux)
}

The address is /rustjay/flux/flow_scale. Send a float value in the parameter's [min, max] range.

The OSC tab in the desktop GUI (or a plain cat /proc/<pid>/net/udp6 on the Pi) shows the active port.

Changing the port

From the Control web panel, update the Listen Port field. The server restarts on the new port immediately. The new port is saved to config.

Or set it directly in the config JSON:

{
  "osc": { "host": "0.0.0.0", "port": 9001, "enabled": true }
}

Presets

Presets are full-state snapshots. A saved preset captures the current engine state (LFO settings, audio routing, MIDI mappings, input/output config) plus your effect's State struct.

Saving and loading

Use the Presets tab in the control window. Presets are stored as JSON files in:

~/.config/rustjay/<app-name>/presets/

Each effect's presets are isolated by app_name(). Running template and delta from the same machine gives them separate preset banks.

Quick-slots

Eight quick-slot buttons appear at the top of the Presets tab. Press Shift+F1–Shift+F8 on the output window to recall them instantly, without switching to the control window.

Save a preset to a quick-slot: right-click the slot button and choose Save here, or drag-and-drop from the preset list.

Including plugin state

By default, presets only save engine state (LFO, audio, MIDI, etc.). To save your effect's own state — extra textures, ring-buffer configuration, anything beyond EngineState — implement three preset hooks:

#![allow(unused)]
fn main() {
impl EffectPlugin for MyEffect {
    // ...

    fn serialize_preset_state(&self, state: &MyState) -> Option<String> {
        // Return a JSON string, or None to skip
        serde_json::to_string(state).ok()
    }

    fn deserialize_preset_state(&self, data: &str, state: &mut MyState) {
        if let Ok(loaded) = serde_json::from_str::<MyState>(data) {
            *state = loaded;
        }
    }

    fn on_preset_applied(&self, state: &mut MyState, engine: &mut EngineState) {
        // Called after both engine state and plugin state are restored.
        // Use to push any required commands back to the engine.
        // e.g. engine.input_commands.push(InputCommand::SetDevice(...));
    }
}
}

Because State already derives serde::Serialize + DeserializeOwned, serde_json::to_string / from_str are the standard approach.

State initialisation vs presets

Preset loading calls deserialize_preset_state() and then on_preset_applied(). The initial state at launch comes from default_state(). These are separate paths — don't rely on on_preset_applied() for startup initialisation.

File format

Preset files are plain JSON. You can hand-edit them, version-control them, or share them. The top-level keys are:

{
  "engine": { /* EngineState fields */ },
  "plugin": "{ /* your serialized State as a JSON string */ }"
}

Single-Pass Effects

The default rendering mode is a single fullscreen pass: the engine draws two triangles covering the screen, runs your fragment shader once per pixel, and writes to the output window.

This covers a large class of effects:

• Colour grading (HSB, LUT, tinting)
• Kaleidoscope, mirror, tile
• Blur, sharpen, edge detection
• Displacement, wave distortion
• Noise overlays, grain, glitch
• Generative patterns (no video input)

How it works

Each frame the engine:

1. Calls build_uniforms() and uploads the result to @group(1) @binding(0)
2. Binds the current video frame at @group(0) @binding(0/1)
3. Draws the fullscreen quad with your shader
4. Presents the result

You control the output entirely through your fragment shader and your uniform values.

A complete single-pass example

examples/template is the canonical reference: HSB colour adjustment with audio reactivity, LFO targets, and MIDI/OSC/web control in ~80 lines.

cargo run -p template

Key points from its implementation:

#![allow(unused)]
fn main() {
// Declare parameters → auto UI, LFO targets, MIDI learn
fn parameters(&self) -> Vec<ParameterDescriptor> {
    vec![
        ParameterDescriptor::float("hue_shift",  "Hue Shift",  ParamCategory::Color, -180.0, 180.0, 0.0, 1.0),
        ParameterDescriptor::float("saturation", "Saturation", ParamCategory::Color, 0.0, 2.0, 1.0, 0.01),
        ParameterDescriptor::float("brightness", "Brightness", ParamCategory::Color, 0.0, 2.0, 1.0, 0.01),
    ]
}

// get_param() returns base + LFO + audio routing contributions
fn build_uniforms(&self, s: &HsbState, engine: &EngineState) -> HsbUniforms {
    HsbUniforms { values: [
        engine.get_param("hue_shift").unwrap_or(s.hue_shift),
        engine.get_param("saturation").unwrap_or(s.saturation),
        engine.get_param("brightness").unwrap_or(s.brightness),
        0.0, // padding
    ]}
}
}

The WGSL shader does the actual colour work:

@fragment
fn fs_main(in: VertexOutput) -> @location(0) vec4<f32> {
    var col = textureSample(input_tex, input_sampler, in.texcoord);
    // apply hue rotation, saturation scale, brightness scale ...
    return col;
}

When single-pass isn't enough

• Frame history — you need to read from a previous frame, or accumulate across multiple frames → Frame History & Custom Pipelines
• Multiple stages — blur then mix, or feedback loop → Multi-Pass with RenderGraph
• Vertex displacement — displace a mesh in 3D space from a texture → Mesh Displacement

Frame History & Custom Pipelines

Some effects need access to frames from the recent past — RGB delay, motion extraction, echo trails, temporal blur. The single-pass default only gives you the current frame.

The solution is to override render() and manage your own GPU pipeline, including a ring buffer of past frames.

The pattern

1. Store textures for your history ring buffer in the plugin struct
2. Create your own pipeline in init()
3. Override render() to: copy the current input frame into the ring buffer, bind delayed frames, run your pass, and return true

Returning true from render() tells the engine to skip its default draw.

Minimal skeleton

#![allow(unused)]
fn main() {
use rustjay_engine::prelude::*;

struct DelayEffect {
    pipeline:  Option<wgpu::RenderPipeline>,
    bgl:       Option<wgpu::BindGroupLayout>,
    history:   Vec<Texture>,     // ring buffer
    write_idx: usize,
}

impl EffectPlugin for DelayEffect {
    type State    = DelayState;
    type Uniforms = DelayUniforms;

    // ── The engine still requires a shader_source ──────────────────────────
    // It compiles this for its default pipeline, but since render() returns
    // true below, that pipeline is never executed. Provide a minimal stub
    // that matches the standard binding layout.
    fn shader_source(&self) -> &'static str {
        include_str!("shaders/stub.wgsl")
    }

    fn build_uniforms(&self, s: &DelayState, engine: &EngineState) -> DelayUniforms {
        DelayUniforms { /* ... */ }
    }

    // ── Create the real pipeline ───────────────────────────────────────────
    fn init(&mut self, device: &wgpu::Device, queue: &wgpu::Queue) {
        let shader = device.create_shader_module(wgpu::ShaderModuleDescriptor {
            label: Some("delay"),
            source: wgpu::ShaderSource::Wgsl(include_str!("shaders/delay.wgsl").into()),
        });

        // Create bind group layout with N history texture slots
        let bgl = device.create_bind_group_layout(&wgpu::BindGroupLayoutDescriptor {
            label: Some("delay-bgl"),
            entries: &[
                // slot 0: current input, slot 1: sampler,
                // slot 2..N+2: history textures and their samplers
                // ...
            ],
        });

        let pipeline_layout = device.create_pipeline_layout(&wgpu::PipelineLayoutDescriptor {
            bind_group_layouts: &[&bgl],
            push_constant_ranges: &[],
            label: None,
        });

        // create_render_pipeline(...) with your layout
        self.pipeline = Some(/* ... */);
        self.bgl      = Some(bgl);

        // Allocate history textures
        self.history = (0..8).map(|_| {
            Texture::create_render_target(device, 1920, 1080) // or your preferred size
        }).collect();
    }

    // ── Custom render ──────────────────────────────────────────────────────
    fn render(
        &mut self,
        encoder:            &mut wgpu::CommandEncoder,
        device:             &wgpu::Device,
        queue:              &wgpu::Queue,
        input_view:         Option<&wgpu::TextureView>,
        input_sampler:      Option<&wgpu::Sampler>,
        render_target_view: &wgpu::TextureView,
        app_state:          &mut DelayState,
        engine_state:       &EngineState,
        _vertex_buffer:     &wgpu::Buffer,
        input_texture:      Option<&wgpu::Texture>,   // raw texture for copies
    ) -> bool {
        // 1. Copy current input frame into history ring buffer
        if let Some(src) = input_texture {
            let dst = &self.history[self.write_idx];
            encoder.copy_texture_to_texture(
                src.as_image_copy(),
                dst.texture().as_image_copy(),
                dst.texture().size(),
            );
            self.write_idx = (self.write_idx + 1) % self.history.len();
        }

        // 2. Build a bind group from the appropriate history frame(s)
        let delay_idx = (self.write_idx + self.history.len() - app_state.delay_frames)
            % self.history.len();
        let bind_group = device.create_bind_group(&wgpu::BindGroupDescriptor {
            layout: self.bgl.as_ref().unwrap(),
            entries: &[/* bind delayed frames ... */],
            label: None,
        });

        // 3. Run the render pass
        let mut rp = encoder.begin_render_pass(&wgpu::RenderPassDescriptor {
            color_attachments: &[Some(wgpu::RenderPassColorAttachment {
                view: render_target_view,
                resolve_target: None,
                ops: wgpu::Operations {
                    load: wgpu::LoadOp::Clear(wgpu::Color::BLACK),
                    store: wgpu::StoreOp::Store,
                },
            })],
            depth_stencil_attachment: None,
            label: Some("delay-pass"),
            timestamp_writes: None,
            occlusion_query_set: None,
        });
        rp.set_pipeline(self.pipeline.as_ref().unwrap());
        rp.set_bind_group(0, &bind_group, &[]);
        rp.draw(0..6, 0..1); // 6 vertices = fullscreen quad

        true // skip the engine's default render pass
    }
}
}

The stub shader

The engine still compiles shader_source() for its default pipeline. The stub must declare the standard binding layout even though the pipeline never runs:

// shaders/stub.wgsl — minimal stub, never actually drawn
struct VertexOutput {
    @builtin(position) position: vec4<f32>,
    @location(0) texcoord: vec2<f32>,
};

@group(0) @binding(0) var input_tex:     texture_2d<f32>;
@group(0) @binding(1) var input_sampler: sampler;

struct Uniforms { _pad: f32 };
@group(1) @binding(0) var<uniform> u: Uniforms;

@vertex
fn vs_main(@location(0) pos: vec2<f32>, @location(1) uv: vec2<f32>) -> VertexOutput {
    var out: VertexOutput;
    out.position = vec4<f32>(pos, 0.0, 1.0);
    out.texcoord = uv;
    return out;
}

@fragment
fn fs_main(in: VertexOutput) -> @location(0) vec4<f32> {
    return textureSample(input_tex, input_sampler, in.texcoord);
}

Per-channel delay

The examples/delta example goes further: it maintains separate delay values for the red, green, and blue channels, producing RGB colour trails. Study it for a complete, production-ready implementation of this pattern.

cargo run -p delta

Features demonstrated:

• 8-frame ring buffer
• Per-channel R/G/B delays (0–7 frames)
• 8 blend modes (Replace, Add, Multiply, Screen, Difference, Overlay, Lighten, Darken)
• Per-channel gain, trail fade, threshold, smoothing

Multi-Pass with RenderGraph

RenderGraph lets you chain multiple shader passes without managing intermediate textures yourself. Each pass writes to a texture that the engine creates automatically; the last pass writes to the render target.

When to use RenderGraph

• Two-stage effects: blur → mix, threshold → edge-detect → composite
• Feedback loops: each frame reads its own previous output
• Post-processing chains where each stage is a self-contained WGSL shader

If you need to manage your own GPU resources or read from a ring buffer of past frames, use Frame History & Custom Pipelines instead.

Defining a graph

Return a RenderGraph from render_graph():

#![allow(unused)]
fn main() {
impl EffectPlugin for MyEffect {
    fn render_graph(&self) -> Option<RenderGraph> {
        Some(
            RenderGraph::new()
                .with_pass(Pass {
                    label: "Blur",
                    shader: include_str!("shaders/blur.wgsl"),
                    input: PassInput::EngineInput,
                })
                .with_pass(Pass {
                    label: "Composite",
                    shader: include_str!("shaders/composite.wgsl"),
                    input: PassInput::PreviousPass,
                }),
        )
    }
}
}

The engine executes passes in declaration order. Intermediate textures are managed automatically.

Pass input sources

Enabling feedback

Add .with_feedback() to the graph to enable the feedback texture:

#![allow(unused)]
fn main() {
RenderGraph::new()
    .with_pass(Pass {
        label: "Distort",
        shader: include_str!("shaders/distort.wgsl"),
        input: PassInput::EngineInput,
    })
    .with_pass(Pass {
        label: "Feedback Mix",
        shader: include_str!("shaders/feedback.wgsl"),
        input: PassInput::PreviousPass,
    })
    .with_feedback()
}

When feedback is enabled, every pass gets two additional bindings:

@group(0) @binding(2) var feedback_tex:     texture_2d<f32>;
@group(0) @binding(3) var feedback_sampler: sampler;

feedback_tex always contains the final output of the previous frame. Passes that don't use feedback simply omit those declarations.

Per-pass uniforms

By default, build_pass_uniforms() delegates to build_uniforms(), so a single uniform struct serves all passes.

Override build_pass_uniforms() to send different values to each pass:

#![allow(unused)]
fn main() {
fn build_pass_uniforms(
    &self,
    pass_index: usize,
    s: &MyState,
    engine: &EngineState,
) -> MyUniforms {
    match pass_index {
        0 => MyUniforms { radius: s.blur_radius, .. },
        1 => MyUniforms { mix: s.feedback_amount, .. },
        _ => self.build_uniforms(s, engine),
    }
}
}

Single-pass fallback

The engine still compiles shader_source() for its default pipeline even when render_graph() returns Some. If the graph isn't available (e.g. the feature is disabled at compile time), the engine falls back to single-pass.

In practice this means shader_source() can be a minimal pass-through shader for RenderGraph effects.

Example: waaaves

examples/waaaves demonstrates a 3-pass feedback pipeline with complex per-pass bind groups and dual ring buffers. Study it for a complete, production-ready multi-pass implementation.

cargo run -p waaaves

Passes:

1. Pipeline A — initial video processing
2. Pipeline B — spatial transformation with feedback
3. Pipeline C — colour mixing and output

Mesh Displacement

Instead of a fullscreen quad, rustjay-engine can generate an indexed cols × rows mesh grid. Your vertex shader displaces each point in 3D space, turning the video frame into a displaced 3D surface.

This is how the classic Rutt-Etra analogue video synthesiser look is achieved — horizontal scanlines pushed out along the Z axis by video luminance.

Enabling a mesh

Return a MeshDescriptor from mesh_descriptor():

#![allow(unused)]
fn main() {
fn mesh_descriptor(&self, _state: &MyState) -> Option<MeshDescriptor> {
    Some(MeshDescriptor {
        cols: 320,
        rows: 240,
        topology: MeshTopology::Scanlines,
    })
}
}

The engine replaces the default two-triangle quad with a 320 × 240 indexed grid and calls your vertex shader for each vertex.

Topologies

Letting the vertex shader sample the texture

The standard binding layout only exposes group 0 to the fragment stage. For displacement effects, you need the vertex shader to sample the video texture to compute the displacement amount.

Add this to your plugin:

#![allow(unused)]
fn main() {
fn vertex_reads_texture(&self) -> bool {
    true
}
}

The engine then adds VERTEX | FRAGMENT visibility to the group-0 bind group entries.

In the vertex shader:

@group(0) @binding(0) var input_tex:     texture_2d<f32>;
@group(0) @binding(1) var input_sampler: sampler;

@vertex
fn vs_main(@location(0) pos: vec2<f32>, @location(1) uv: vec2<f32>) -> VertexOutput {
    // Sample luminance at this vertex's UV
    let col  = textureSample(input_tex, input_sampler, uv);
    let luma = dot(col.rgb, vec3<f32>(0.299, 0.587, 0.114));

    // Displace along Z proportional to luminance
    let displaced = vec4<f32>(pos, luma * u.displacement_scale, 1.0);

    var out: VertexOutput;
    out.position = u.mvp * displaced;
    out.texcoord = uv;
    return out;
}

MVP matrix

For 3D displacement, you need a model-view-projection matrix in your uniforms:

#![allow(unused)]
fn main() {
#[repr(C)]
#[derive(Copy, Clone, bytemuck::Pod, bytemuck::Zeroable)]
struct MeshUniforms {
    mvp:               [[f32; 4]; 4],
    displacement_scale: f32,
    _pad:              [f32; 3],
}
}

Build the MVP matrix each frame from your camera/rotation state. The glam crate is convenient for this:

#![allow(unused)]
fn main() {
fn build_uniforms(&self, s: &MeshState, engine: &EngineState) -> MeshUniforms {
    let rotation = glam::Mat4::from_rotation_y(s.rot_y)
                 * glam::Mat4::from_rotation_x(s.rot_x);
    let view     = glam::Mat4::look_at_rh(
        glam::Vec3::new(0.0, 0.0, 2.0),
        glam::Vec3::ZERO,
        glam::Vec3::Y,
    );
    let proj     = glam::Mat4::perspective_rh(
        std::f32::consts::FOVN_PI_4,
        16.0 / 9.0,
        0.01, 100.0,
    );
    MeshUniforms {
        mvp: (proj * view * rotation).to_cols_array_2d(),
        displacement_scale: engine.get_param("displacement").unwrap_or(s.displacement),
        _pad: [0.0; 3],
    }
}
}

Compute shader option

For very large meshes or complex per-vertex computations, use the compute shader path instead of the vertex shader:

#![allow(unused)]
fn main() {
fn compute_shader(&self) -> Option<&'static str> {
    Some(include_str!("shaders/displace.comp.wgsl"))
}
}

The engine dispatches the compute shader before the render pass. It receives:

• @group(0) @binding(0) — your uniform buffer
• @group(1) @binding(0) — the vertex storage buffer (array<Vertex>, read/write)

Workgroup size must be @workgroup_size(256, 1, 1). The engine dispatches 1D groups to cover all vertices.

Example: sputnik

examples/sputnik is a complete Rutt-Etra-style implementation with:

• Dynamic mesh grid (configurable resolution)
• Per-axis rotation controlled by LFOs
• Ring modulation between mesh position and LFO output
• Audio-reactive displacement depth

cargo run -p sputnik

Template — HSB Colour Adjustment

examples/template is the reference starting point for rustjay-engine. It implements HSB (hue, saturation, brightness) colour grading in the simplest possible way — about 80 lines of Rust and a single WGSL shader — and demonstrates every core feature a real effect needs: parameters, audio reactivity, LFO targets, MIDI/OSC/web control, and presets.

cargo run -p template

Read this page alongside Your First Effect and The EffectPlugin Trait. Template is the canonical example those pages reference.

What it does

Template applies three HSB adjustments to the video input:

• Hue Shift — rotates all hues by ±180°, wrapping at the colour wheel boundary
• Saturation — multiplies saturation; 0 = greyscale, 1 = original, 2 = oversaturated
• Brightness — multiplies value (HSV); 0 = black, 1 = original, 2 = overexposed

All three are live parameters — sliders in the control window, LFO targets, MIDI learnable, OSC addressable, and saved with presets.

The Rust side

The full implementation (src/main.rs) is intentionally minimal:

struct HsbEffect;                         // no fields — all state lives below

#[repr(C)]
#[derive(Copy, Clone, bytemuck::Pod, bytemuck::Zeroable)]
struct HsbUniforms {
    values: [f32; 4],                     // hue_shift, saturation, brightness, _pad
}

#[derive(Default, serde::Serialize, serde::Deserialize)]
struct HsbState {
    hue_shift:  f32,
    saturation: f32,
    brightness: f32,
    enabled:    bool,
}

impl EffectPlugin for HsbEffect {
    type State    = HsbState;
    type Uniforms = HsbUniforms;

    fn app_name(&self) -> &str { "template" }

    fn default_state(&self) -> HsbState {
        HsbState { saturation: 1.0, brightness: 1.0, enabled: true, ..Default::default() }
    }

    fn parameters(&self) -> Vec<ParameterDescriptor> {
        vec![
            ParameterDescriptor::float("hue_shift",  "Hue Shift",  ParamCategory::Color, -180.0, 180.0, 0.0,  1.0),
            ParameterDescriptor::float("saturation", "Saturation", ParamCategory::Color,    0.0,   2.0, 1.0, 0.01),
            ParameterDescriptor::float("brightness", "Brightness", ParamCategory::Color,    0.0,   2.0, 1.0, 0.01),
        ]
    }

    fn shader_source(&self) -> &'static str {
        include_str!("shaders/hsb.wgsl")
    }

    fn build_uniforms(&self, s: &HsbState, engine: &EngineState) -> HsbUniforms {
        if !s.enabled {
            return HsbUniforms { values: [0.0, 1.0, 1.0, 0.0] }; // passthrough
        }
        HsbUniforms { values: [
            engine.get_param("hue_shift").unwrap_or(s.hue_shift),
            engine.get_param("saturation").unwrap_or(s.saturation),
            engine.get_param("brightness").unwrap_or(s.brightness),
            0.0,
        ]}
    }
}

fn main() -> anyhow::Result<()> {
    env_logger::init();
    rustjay_engine::run(HsbEffect)
}

Things to notice

default_state() — HsbState derives Default, which would give saturation: 0.0 and brightness: 0.0 (a black screen). Overriding default_state() sets sensible starting values without requiring a custom Default impl for the whole struct.

engine.get_param() — returns the parameter's base slider value plus any active LFO and audio routing contributions, clamped to the declared range. Falling back to s.hue_shift etc. handles the case where the engine doesn't have a value yet (first frame before the parameter system initialises).

The enabled guard — returning a passthrough uniform ([0, 1, 1, 0]) when disabled lets the user bypass the effect without rebuilding the pipeline. The shader sees unmodified identity values.

ParamCategory::Color — places all three sliders in the built-in Color tab. Changing this to ParamCategory::Motion or ParamCategory::Custom("name") changes where they appear.

The shader

src/shaders/hsb.wgsl does the colour conversion in three steps:

@fragment
fn fs_main(in: VertexOutput) -> @location(0) vec4<f32> {
    var color = textureSample(input_tex, input_sampler, in.texcoord);
    let adjusted = apply_hsb(color.rgb, hsb_params);
    return vec4<f32>(adjusted, color.a);
}

apply_hsb() converts RGB → HSV, applies the three adjustments, and converts back:

fn apply_hsb(rgb: vec3<f32>, params: HsbParams) -> vec3<f32> {
    var hsv = rgb_to_hsv(rgb);
    hsv.x = fract(hsv.x + params.values.x / 360.0); // hue rotation, wrapping
    hsv.y = clamp(hsv.y * params.values.y, 0.0, 1.0); // saturation scale
    hsv.z = clamp(hsv.z * params.values.z, 0.0, 1.0); // brightness scale
    return hsv_to_rgb(hsv);
}

fract() on the hue handles wrap-around — a shift of +350° and a shift of −10° produce the same result. clamp() on saturation and brightness prevents out-of-range values from producing invalid colours when LFO depth pushes a parameter past its declared bounds.

Using template as a starting point

The recommended way to start a new effect:

cp -r examples/template my-effect
cd my-effect
# edit Cargo.toml name, then src/main.rs and src/shaders/

The minimum changes to make it your own:

1. Rename the structs (HsbEffect → MyEffect, etc.)
2. Change app_name() — this isolates config and presets from other effects
3. Replace HsbUniforms with your uniform layout
4. Replace HsbState with your state fields
5. Replace parameters() with your declared parameters
6. Rewrite build_uniforms() to fill your uniform struct
7. Rewrite the shader

Everything else — the control window, all built-in tabs, audio analysis, LFO system, MIDI, OSC, presets — works without any changes.

Delta — RGB Delay / Motion Extraction

examples/delta implements RGB delay — a temporal video effect where the red, green, and blue channels are sampled from independently delayed frames. The result is chromatic motion trails that colour-code the direction and speed of movement in the image.

cargo run -p delta

A related version using the egui backend instead of ImGui is in delta-egui.

What it does

Each channel samples a different point in time from a 16-slot frame-history ring buffer. With red at frame 0, green at frame 2, and blue at frame 4 (the defaults), a moving object leaves a trail: its current position is in all three channels, but its recent positions are visible as distinct red, green, or blue ghosts.

The effect is inspired by Posy's colour delay work and the RGB delay patches found on analogue video synthesisers.

Parameters

Blend modes

Difference mode with matching delays and inverted gains is a clean motion-extraction technique: static areas cancel to black, moving areas glow in the delay colour.

Architecture

Delta overrides render() and manages its own GPU pipeline — see Frame History & Custom Pipelines for the general pattern.

FrameHistory

FrameHistory is a 16-slot ring buffer of GPU textures. Each slot is a full-resolution Bgra8Unorm render target:

Frame N-16  Frame N-15  ...  Frame N-1   Frame N (write head)
     ↑                              ↑
 get_frame(15)               get_frame(0) — most recent completed frame

Each frame:

1. push_frame() copies the current video input (input_texture) into the write slot via copy_texture_to_texture — a pure GPU-side copy, no CPU round-trip
2. The write index advances modulo 16
3. get_frame(n) looks up the slot n steps behind the write head

#![allow(unused)]
fn main() {
fn push_frame(&mut self, source: &wgpu::Texture, encoder: &mut wgpu::CommandEncoder) {
    encoder.copy_texture_to_texture(src, dest, size);
    self.write_index = (self.write_index + 1) % self.max_history;
}

fn get_frame(&self, frames_ago: usize) -> Option<&Texture> {
    let index = if frames_ago < self.write_index {
        self.write_index - 1 - frames_ago
    } else {
        self.max_history - 1 - (frames_ago - self.write_index)
    };
    self.frames.get(index)
}
}

FrameHistory::resize() detects resolution changes and reallocates all slots — this handles window resizes gracefully without a crash.

Bind group layout

The shader receives four textures and one shared sampler on group 0:

@group(0) @binding(0)  red_delayed_frame
@group(0) @binding(1)  green_delayed_frame
@group(0) @binding(2)  blue_delayed_frame
@group(0) @binding(3)  live_input
@group(0) @binding(4)  sampler (shared)

The bind group is rebuilt each frame after looking up the three history slots — this is cheap because it's a struct of TextureView references, not copies.

Uniform layout

#![allow(unused)]
fn main() {
#[repr(C)]
#[derive(Copy, Clone, bytemuck::Pod, bytemuck::Zeroable)]
struct DeltaUniforms {
    delays:       [f32; 4], // red, green, blue, max_history
    settings:     [f32; 4], // intensity, blend_mode, grayscale, unused
    channel_gain: [f32; 4], // red, green, blue, unused
    mix_options:  [f32; 4], // input_mix, trail_fade, threshold, smoothing
}
}

Four vec4<f32> blocks — 64 bytes total, exactly 4× the 16-byte wgpu uniform alignment.

The Motion tab

Delta ships a custom MotionTab that replaces the engine's built-in Motion tab:

#![allow(unused)]
fn main() {
impl AnyGuiTab for MotionTab {
    fn name(&self) -> &str { "Motion" }
    fn replaces(&self) -> Option<GuiTab> { Some(GuiTab::Motion) }
    // ...
}
}

Sliders call engine.set_param_base(id, value) to keep the engine's parameter registry in sync — this ensures LFO and audio routing targets stay consistent with the displayed values.

Preset tips

Because the three delay values interact so strongly with the blend mode, saving named presets for combinations you like is worth doing. Good starting points:

• Motion trails: R=0, G=4, B=8, Blend=Replace, Grayscale=on
• Chroma ghost: R=0, G=8, B=16, Blend=Screen, Grayscale=off, all Gains=1
• Inversion ghost: R=0, G=2, B=4, Blend=Difference, Red Gain=−1, Blue Gain=−1
• Smear: R=0, G=1, B=2, Blend=Add, Trail Fade=0.3, Smoothing=0.2

delta-egui — egui Backend Edition

examples/delta-egui is the same RGB delay effect as delta — same DeltaEffect, same FrameHistory ring buffer, same DeltaState and DeltaUniforms — built with the egui control backend instead of ImGui.

cargo run -p delta-egui

Read the delta page for everything about the effect itself. This page covers only what changes when switching to egui.

Enabling the egui feature

# Cargo.toml
[dependencies]
rustjay-engine = { git = "...", features = ["egui"] }

Entry point

#![allow(unused)]
fn main() {
// ImGui version:
rustjay_engine::run_with_tabs(DeltaEffect::default(), vec![Box::new(MotionTab)])

// egui version:
rustjay_engine::run_with_egui_tabs(DeltaEffect::default(), vec![Box::new(MotionTab)])
}

One function name difference. The rest of the plugin — EffectPlugin impl, state, uniforms, render() — is identical.

Implementing a tab with egui

Implement AnyEguiTab instead of AnyGuiTab. The trait signature is the same, but draw receives &mut egui::Ui instead of &imgui::Ui:

#![allow(unused)]
fn main() {
// ImGui
impl AnyGuiTab for MotionTab {
    fn name(&self) -> &str { "Motion" }
    fn replaces(&self) -> Option<GuiTab> { Some(GuiTab::Motion) }

    fn draw(&mut self, ui: &imgui::Ui, app_state: &mut dyn Any, engine: &mut EngineState) {
        ui.slider_config("Intensity", 0.0_f32, 1.0_f32).build(&mut state.intensity);
    }
}

// egui
impl AnyEguiTab for MotionTab {
    fn name(&self) -> &str { "Motion" }
    fn replaces(&self) -> Option<GuiTab> { Some(GuiTab::Motion) }

    fn draw(&mut self, ui: &mut egui::Ui, app_state: &mut dyn Any, engine: &mut EngineState) {
        param_slider(ui, engine, "intensity", "Intensity", 0.0, 1.0);
    }
}
}

param_slider helpers

The prelude exports two egui-specific helpers that keep the engine's parameter registry in sync without boilerplate:

#![allow(unused)]
fn main() {
// Float slider — reads get_param_base, writes set_param_base on change
param_slider(ui, engine, "intensity", "Intensity", 0.0, 1.0);

// Integer slider
param_slider_int(ui, engine, "red_delay", "Red", 0, 16);
}

These are equivalent to:

#![allow(unused)]
fn main() {
let mut val = engine.get_param_base("intensity").unwrap_or(1.0);
if ui.add(egui::Slider::new(&mut val, 0.0..=1.0).text("Intensity")).changed() {
    engine.set_param_base("intensity", val);
}
}

For types not covered by the helpers (bool checkboxes, enum combo boxes), call engine.get_param_base / engine.set_param_base directly:

#![allow(unused)]
fn main() {
// Bool
let mut grayscale = engine.get_param_base("grayscale_input").unwrap_or(1.0) > 0.5;
if ui.checkbox(&mut grayscale, "Grayscale Input").changed() {
    engine.set_param_base("grayscale_input", if grayscale { 1.0 } else { 0.0 });
}

// Enum / combo
let mut idx = engine.get_param_base("blend_mode").unwrap_or(0.0).round() as usize;
egui::ComboBox::from_id_salt("blend_mode")
    .selected_text(blend_names[idx])
    .show_ui(ui, |ui| {
        for (i, name) in blend_names.iter().enumerate() {
            if ui.selectable_label(idx == i, *name).clicked() { idx = i; }
        }
    });
engine.set_param_base("blend_mode", idx as f32);
}

egui widget quick reference

#![allow(unused)]
fn main() {
// Headings and labels
ui.heading("Section title");
ui.label(egui::RichText::new("Bold label").strong());
ui.separator();

// Sliders (param_slider covers the common case)
ui.add(egui::Slider::new(&mut val, min..=max).text("Label"));

// Checkbox
ui.checkbox(&mut flag, "Label");

// Combo box
egui::ComboBox::from_id_salt("unique_id")
    .selected_text(current_label)
    .show_ui(ui, |ui| { /* selectable_label calls */ });

// Horizontal layout
ui.horizontal(|ui| { ui.label("Key:"); ui.label("Value"); });

// Group scoping (prevents id collisions across tabs)
ui.push_id("my_tab", |ui| { /* widgets */ });
}

Waaaves — Multi-Pass Feedback Pipeline

examples/waaaves is a port of the original rustjay-waaaves hardware effect — a three-pass GPU pipeline with two independent feedback delay lines, ~100+ parameters split across three processing blocks, and a full custom control UI.

cargo run -p waaaves

This is the most complex example in the repository. It demonstrates custom multi-pass rendering, dual ring buffers, per-pass uniform buffers, and a structured parameter/tab system.

What it does

The pipeline processes video through three successive shader passes. Each pass reads from earlier passes and its own feedback history, producing a rich accumulation of geometric distortion, colour manipulation, and temporal delay.

Live Input ──────────────────────────────────┐
                                             ↓
                                    ┌──────────────┐     fb1 ring buffer
                                    │   Block A    │ ←── (up to 30 frames)
                                    └──────┬───────┘
                                           │ intermediate_a
                                           ↓
                                    ┌──────────────┐     fb2 ring buffer
                                    │   Block B    │ ←── (up to 30 frames)
                                    └──────┬───────┘
                                           │ intermediate_b
                                           ↓
                                    ┌──────────────┐
                                    │   Block C    │ ──→ Output
                                    └──────────────┘

Block A

Takes the live input (CH1) plus an optional second channel (CH2) and the fb1 feedback history. Applies:

• Per-channel geometry: X/Y/Z displacement, rotation, zoom
• HSB colour adjustment, posterise, kaleidoscope, blur/sharpen
• Mirror, flip, and overflow modes
• CH2 keying (colour key with threshold and softness)
• FB1 delay mix with configurable delay time (frames)

Output feeds intermediate_a and is also written into the fb1 ring buffer.

Block B

Takes intermediate_a and the fb2 feedback history. Applies the same geometry and colour processing set as Block A, plus its own feedback delay.

Output feeds intermediate_b and is written into the fb2 ring buffer.

Block C

Takes both intermediate_a and intermediate_b. Applies:

• Output geometry and colour transforms for each intermediate
• A colour matrix mixer (R→R, R→G, R→B, G→R, etc.)
• Final HSB and posterise
• A global mix amount between the two intermediates

Output is the final rendered frame.

Parameters

Parameters are split into three blocks reflecting the three passes, each with its own tab in the control window.

Block 1 tab (≈ 60 parameters)

Controls CH1 input processing, the CH2 key, and the FB1 delay line:

Block 2 tab (≈ 40 parameters)

Controls the Block B pass geometry, colour, and the FB2 delay line.

Block 3 tab (≈ 30 parameters)

Controls the final composite: geometry applied to each intermediate before mixing, the colour matrix, and the output blend.

Architecture

Dual ring buffers

The RingBuffer struct (render/ring_buffer.rs) is a circular array of GPU textures, all allocated at the same resolution:

#![allow(unused)]
fn main() {
pub struct RingBuffer {
    textures:   Vec<(wgpu::Texture, wgpu::TextureView)>,
    write_head: usize,
    capacity:   usize,
}

// Reading N frames back (minimum 1 — write head holds incomplete frame)
pub fn read_view(&self, frames_back: usize) -> &wgpu::TextureView {
    let idx = frames_back.max(1).min(self.capacity - 1);
    let i   = (self.write_head + self.capacity - idx) % self.capacity;
    &self.textures[i].1
}

// Advance after each frame
pub fn advance(&mut self) {
    self.write_head = (self.write_head + 1) % self.capacity;
}
}

fb1 and fb2 each hold up to 30 frames (configurable via max_delay_frames in the state). Bind groups that reference ring buffer slots are pre-built per slot and cached — they're looked up by index rather than rebuilt each frame:

#![allow(unused)]
fn main() {
// Pre-built once (or on resize):
fb1_bind_groups: Vec<wgpu::BindGroup>,  // one per slot
fb2_bind_groups: Vec<wgpu::BindGroup>,

// Each frame — just an index lookup:
let delay = state.block1.fb1_delay_time as usize;
let bg = &self.fb1_bind_groups[ring_buffer.read_index(delay)];
}

This is more efficient than rebuilding bind groups every frame and is the recommended pattern for variable-delay feedback effects.

Per-pass uniforms

Each of the three passes has its own uniform buffer and bind group, because the parameter blocks are independent:

#![allow(unused)]
fn main() {
struct WaaavesEffect {
    uniform_buf_a: Option<wgpu::Buffer>,
    uniform_buf_b: Option<wgpu::Buffer>,
    uniform_buf_c: Option<wgpu::Buffer>,
    uniform_bg_a:  Option<wgpu::BindGroup>,
    uniform_bg_b:  Option<wgpu::BindGroup>,
    uniform_bg_c:  Option<wgpu::BindGroup>,
}
}

All three uniform buffers are uploaded every frame inside render() before the passes execute.

Bind group layout per pass

The layouts are created in render/passes.rs and shared across all frames.

Resize handling

When the input resolution changes, all textures are reallocated:

• intermediate_a, intermediate_b — single textures, recreated
• fb1, fb2 — ring buffers, RingBuffer::resize() is called, which reallocates all capacity slots and resets the write head
• All cached bind groups are rebuilt after resize

Dummy texture

Shader slots that don't always have a real texture bound (e.g. CH2 when no second source is active) use a 1×1 black Bgra8Unorm texture (dummy). This avoids validation errors from unbound texture slots without branching in the shader.

Pixel-pick FSM

The PickState field in WaaavesState implements a three-step finite state machine for picking a colour from the output frame to use as a key value:

Idle  →  Armed { target }  →  Pending { target }  →  Idle
         (button clicked)      (next render completes)

target identifies which colour key destination receives the picked value (CH2, FB1, FB2, or Final). The pixel read happens on the Rust side at the completion of the pending render, not on the GPU.

Module layout

examples/waaaves/src/
├── main.rs              # WaaavesEffect struct, EffectPlugin impl
├── state.rs             # WaaavesState, PickState FSM
├── uniforms.rs          # Per-pass uniform structs
├── params/
│   ├── block1.rs        # Block1Params (~60 fields)
│   ├── block2.rs        # Block2Params (~40 fields)
│   ├── block3.rs        # Block3Params (~30 fields)
│   └── descriptors.rs   # ParameterDescriptor declarations for all params
├── render/
│   ├── passes.rs        # Pipeline + bind group layout creation
│   └── ring_buffer.rs   # RingBuffer — circular texture buffer
├── tabs/
│   ├── block1_tab.rs    # ImGui tab for Block 1
│   ├── block2_tab.rs    # ImGui tab for Block 2
│   └── block3_tab.rs    # ImGui tab for Block 3
├── lfo_ui.rs            # Custom LFO UI (hybrid: native controls + engine LFO)
└── legacy_preset.rs     # Preset compatibility with the original waaaves format

Sputnik — Rutt-Etra Mesh Displacement

examples/sputnik is a Rutt-Etra-style effect where video luminance pushes a dense grid of vertices into 3D space. The brighter a pixel, the further its mesh vertex is displaced toward the viewer. With a moving camera and animated LFOs warping the grid before the luminance pass, the result is a continuously morphing 3D wireframe portrait of the video signal.

cargo run -p sputnik

This is the most technically involved single-pass example. It demonstrates MeshDescriptor, vertex_reads_texture, compute_shader, per-axis LFO phase accumulation in prepare(), tempo sync, and an eight-band audio reactivity system.

What it does

Video is not rendered as a flat rectangle. Instead, a dense grid of vertices — 320 columns × 180 rows by default (57,600 vertices) — is displaced in two stages every frame:

1. Compute pass — the LFO system warps the flat grid into undulating 3D shapes before any video is involved
2. Vertex pass — each vertex samples the video texture, converts its pixel to luminance, and displaces further along Y and Z by that brightness value

The fragment shader then samples the same texture at each vertex's UV coordinate, painting the displaced mesh with the live video. The result combines spatial distortion from the LFOs with luminance-driven depth from the image itself.

The two-pass GPU pipeline

Pass 1 — compute shader

The compute shader runs once per vertex (@workgroup_size(256, 1, 1)) before the render pass. It reads and writes a storage vertex buffer:

@group(0) @binding(0) var<uniform>             u:        SputnikUniforms;
@group(1) @binding(0) var<storage, read_write> vertices: array<Vertex>;

For each vertex it:

1. Reconstructs the base XY position from the vertex's UV coordinate, aspect-corrected to match the input texture
2. Evaluates three independent LFO values — X (horizontal), Y (vertical), Z (distance from UV centre)
3. Optionally applies phase modulation (adds a neighbour axis's raw output to the phase argument before re-evaluating the waveform — FM-style cross-axis patterns)
4. Optionally applies ring modulation (multiplies X or Y displacement by the Z value)
5. Applies the Z LFO as a zoom-pulse: scales the base XY position by (1 - z_total), then adds the X/Y displacements
6. Writes the displaced position back to the storage buffer

The base position is always reconstructed from the UV coordinate, never accumulated — this prevents runaway drift across frames.

Pass 2 — vertex + fragment shader

The vertex shader reads the compute-displaced position and adds a second displacement layer from the video:

// textureSampleLevel is required in vertex stage (no screen-space derivatives)
let color = textureSampleLevel(input_tex, input_sampler, texcoord, 0.0);
var bright = dot(color.rgb, vec3<f32>(0.299, 0.587, 0.114));
if u.bright_invert != 0u { bright = 1.0 - bright; }

// Logarithmic scaling — matches the original sputnikMesh feel
bright = 2.0 * log(1.0 + bright);

let displacement = (bright + audio_lift) * u.displacement_scale;
let pos3 = vec3<f32>(position.x, position.y + displacement, displacement * 0.5);

out.position = u.mvp * vec4<f32>(pos3, 1.0);

The logarithmic curve (2 × log(1 + luma)) gives the effect the same feel as the original openFrameworks sputnikMesh: shadow regions stay relatively flat while bright regions punch sharply forward.

Audio adds a second lift to the displacement, mapped across 8 frequency bands — each column of the mesh is biased by the band that corresponds to its horizontal position.

The fragment shader is trivial — it just samples the texture at the vertex UV and returns the colour.

Declaring the mesh

#![allow(unused)]
fn main() {
fn mesh_descriptor(&self, state: &SputnikState) -> Option<MeshDescriptor> {
    let topology = match state.topology {
        0 => MeshTopology::Scanlines,
        1 => MeshTopology::Triangles,
        2 => MeshTopology::Wireframe,
        3 => MeshTopology::Points,
        _ => MeshTopology::Scanlines,
    };
    Some(MeshDescriptor { cols: state.mesh_cols, rows: state.mesh_rows, topology })
}

fn vertex_reads_texture(&self) -> bool { true }

fn compute_shader(&self) -> Option<&'static str> {
    Some(include_str!("shaders/sputnik_compute.wgsl"))
}
}

vertex_reads_texture() returning true tells the engine to bind the input texture at group 0 binding 0 during the vertex stage — by default, vertex stage texture access is not enabled.

compute_shader() returning Some(...) causes the engine to run that compute dispatch before the render pass each frame.

Topology modes

The mesh resolution (columns × rows) can be changed at runtime from the Sputnik tab. Higher values produce finer detail at the cost of vertex count.

The three-axis LFO system

Each axis has an independent LFO that runs at frame-rate-accurate speed regardless of render framerate. Four waveforms are available: Sine, Square, Sawtooth, and Noise.

Each axis has three parameters:

The spatial frequency parameter is key: lfo_freq = 0 produces a uniform wave across the whole mesh, while higher values create many small oscillations across the surface.

Phase accumulation in prepare()

The LFO phase accumulators are advanced in prepare(), which runs once per frame before build_uniforms():

#![allow(unused)]
fn main() {
fn prepare(&mut self, state: &mut SputnikState, engine: &EngineState, ...) {
    let dt  = engine.performance.frame_time_ms / 1000.0;
    let bpm = engine.effective_bpm();

    let xr = if state.x_tempo_sync {
        beat_division_to_hz(state.x_beat_division, bpm)
    } else {
        engine.get_param("x_lfo_rate").unwrap_or(state.x_lfo_rate)
    };
    // ... same for y, z ...

    state.x_lfo_arg += xr * dt;
    state.y_lfo_arg += yr * dt;
    state.z_lfo_arg += zr * dt;
}
}

frame_time_ms / 1000.0 converts the engine's elapsed frame time to seconds. Multiplying by rate (Hz) gives the correct phase increment regardless of how fast or slowly the GPU is rendering.

The phase accumulators are marked #[serde(skip)] in SputnikState — they reset to zero when a preset is loaded (intentional: resuming from a saved snapshot with stale phase values would look wrong).

Tempo sync

Setting x_tempo_sync = true replaces the freerunning x_lfo_rate Hz value with a rate derived from the global BPM:

#![allow(unused)]
fn main() {
beat_division_to_hz(state.x_beat_division, bpm)
}

x_beat_division indexes a table of musical subdivisions (whole note, half, quarter, eighth, etc.). The LFO completes one cycle every N beats, staying locked to the track tempo.

Phase and ring modulation

Phase modulation adds a neighbouring axis's raw output to the LFO's phase argument before evaluating the waveform:

if u.x_phasemod != 0u {
    x_lfo = lfo(u.x_lfo_arg + tc.x * u.x_lfo_freq + y_raw, u.x_lfo_shape);
}

X is phase-modulated by Y, Y by X, Z by X. With both axes at Sine waveform this produces FM-style Lissajous patterns.

Ring modulation multiplies the X or Y displacement by the current Z LFO value. At low Z amplitude this creates a subtle amplitude envelope across the mesh; pushed hard it produces sharp nodal bands.

Audio reactivity

audio_reactivity scales how strongly the audio spectrum lifts the mesh displacement:

#![allow(unused)]
fn main() {
for i in 0..4 {
    bands_a[i] = engine.audio.fft[i]     * audio_reactivity;
    bands_b[i] = engine.audio.fft[i + 4] * audio_reactivity;
}
}

Eight frequency bands (from engine.audio.fft) are passed to the vertex shader in two vec4 uniforms. Each column of the mesh maps to one of the eight bands based on its horizontal UV coordinate — so bass frequencies affect the left side and treble frequencies affect the right side:

let band_idx  = clamp(u32(texcoord.x * 8.0), 0u, 7u);
let audio_lift = bands[band_idx] * u.audio_reactivity;

Combined with video luminance, audio lift gives you a mesh that pulses in time with the music and also reveals the structure of the video.

Camera

The MVP matrix is built each frame in build_uniforms():

#![allow(unused)]
fn main() {
let projection = glam::Mat4::perspective_rh(60.0f32.to_radians(), aspect, 0.1, 100.0);
let eye = glam::Vec3::new(
    0.0,
    camera_tilt.sin() * dist,
    camera_tilt.cos() * dist,
);
let view = glam::Mat4::look_at_rh(eye, glam::Vec3::ZERO, glam::Vec3::Y);
let mvp  = projection * view;
}

Both parameters are exposed to the engine's LFO and audio routing system via ParameterDescriptor, so you can automate a slow orbit or sync a camera tilt to the beat.

Parameters

All parameters live in ParamCategory::Custom("Sputnik") and appear in the Sputnik tab.

The topology (Scanlines/Triangles/Wireframe/Points), mesh resolution (columns/rows), LFO shapes, phase/ring mod flags, invert brightness, and tempo-sync settings are state fields controlled from the Sputnik tab directly — they're not declared as ParameterDescriptor entries because they are discrete choices rather than continuous values.

The Sputnik tab

SputnikTab adds a new tab named "Sputnik" without replacing any built-in tab:

#![allow(unused)]
fn main() {
impl AnyGuiTab for SputnikTab {
    fn name(&self) -> &str { "Sputnik" }
    // no replaces() — adds alongside the existing tabs
    fn draw(&mut self, ui: &imgui::Ui, app_state: &mut dyn Any, engine: &mut EngineState) {
        let s = app_state.downcast_mut::<SputnikState>().unwrap();
        // topology radio buttons, mesh resolution inputs, LFO sliders, ...
    }
}
}

The Motion tab is explicitly hidden since sputnik manages its own motion controls:

#![allow(unused)]
fn main() {
fn hidden_tabs(&self) -> Vec<GuiTab> {
    vec![GuiTab::Motion]
}
}

The tab uses lfo_axis_sliders() — a local helper that draws Rate, Amp, and Freq sliders for one axis in one go — to keep the LFO section compact.

Starting point for mesh effects

To build a different mesh displacement effect from sputnik:

1. Return Some(MeshDescriptor { cols, rows, topology }) from mesh_descriptor()
2. If the vertex shader needs to read the video texture, return true from vertex_reads_texture()
3. If pre-frame vertex transformation is needed, supply a compute shader via compute_shader()
4. Accumulate phase or other per-frame state in prepare() using engine.performance.frame_time_ms
5. The MVP pattern (perspective × look_at) works for any 3D mesh effect — adjust eye, center, and up vectors for your camera behaviour

ISF Shader Viewer

examples/isf-example loads any ISF (Interactive Shader Format) shader at runtime, parses its input declarations, and auto-generates the parameter UI — no Rust code required per shader.

cargo run -p isf-example

The engine starts immediately with the last-loaded shader. On first launch it defaults to the bundled ColorCycle.fs. Use the Load Shader... button inside the control window to pick any .fs or .frag file — the shader swaps within one frame and the control tab updates its name and sliders to match.

What is ISF?

ISF is an open standard that wraps a GLSL fragment shader in a JSON header declaring its inputs:

/*{
    "CREDIT": "by VIDVOX",
    "ISFVSN": "2",
    "CATEGORIES": ["Glitch", "Retro"],
    "INPUTS": [
        { "NAME": "inputImage", "TYPE": "image" },
        {
            "NAME": "noiseLevel",
            "TYPE": "float",
            "MIN": 0.0, "MAX": 1.0, "DEFAULT": 0.5
        },
        {
            "NAME": "rollAmount",
            "TYPE": "float",
            "MIN": -1.0, "MAX": 1.0, "DEFAULT": 0.0
        }
    ]
}*/

void main() {
    // ... GLSL fragment body ...
}

The JSON header is everything between /*{ and }*/. The rest is plain GLSL. ISF is supported natively by Resolume, VDMX, and many other VJ tools — there is a large community library at editor.isf.video.

Bundled shaders

The example ships with ~50 shaders in examples/isf-example/shaders/. They cover a range of styles: generative patterns, video processing, glitch, and feedback effects. Good ones to start with:

How the viewer works

Startup

On launch the viewer:

1. Reads ~/.config/rustjay/isf-last-shader.txt — if it exists and the file is still on disk, that shader is loaded
2. Falls back to the bundled ColorCycle.fs on first launch or if the saved path is gone
3. Parses the ISF JSON header with the isf crate
4. Builds a Vec<ParameterDescriptor> from the declared inputs — these drive the auto-generated UI and the engine's LFO / MIDI / OSC systems
5. Creates an IsfEffect that owns the parsed data and starts the engine

Switching shaders at runtime

The Load Shader... button at the top of the ISF tab opens a native file picker. Picking a new file writes the path into a shared Arc<Mutex<Option<PathBuf>>>. On the next prepare() call the effect picks up the path, rebuilds the pipeline, and signals the engine to refresh the parameter list via parameters_dirty(). The tab label hot-reloads to the new shader's filename within one frame.

The chosen path is saved to ~/.config/rustjay/isf-last-shader.txt so the next launch picks up where you left off.

File hot-reload

While the app is running, edit any .fs file in your editor and save. IsfEffect::prepare() polls the file's mtime each frame and re-transpiles automatically when it changes — no button press needed. Useful for iterating on shader code live.

Transpilation

GLSL can't run on wgpu directly — the engine transpiles it to WGSL at startup inside IsfEffect::init(). The transpiler (isf_transpiler.rs) handles:

• Stripping the ISF JSON comment header to get raw GLSL
• Both entry-point patterns: void main() writing to gl_FragColor, and the Shadertoy-compatible void mainImage(out vec4 fragColor, in vec2 fragCoord) form
• Mapping ISF built-ins to WGSL equivalents:

• Rewriting GLSL type constructors (vec2(x, y) → vec2<f32>(x, y))
• Declaring all scalar ISF inputs as a flat array<f32, 64> uniform buffer

Auto-generated parameters

Each ISF input type maps to a parameter kind:

Parameters declared this way become full first-class engine parameters: they can be targeted by LFOs, mapped to MIDI CC, addressed over OSC, and saved in presets — without any extra code.

Custom pipeline

Because ISF's binding layout differs from the engine's standard layout, the viewer uses a custom render pipeline (see Frame History & Custom Pipelines for the general pattern):

• shader_source() returns a minimal passthrough stub — the engine compiles it but never runs it
• init() compiles the real transpiled WGSL pipeline
• render() uploads uniforms, builds the bind groups, runs the pass, and returns true

Known limitations

The transpiler handles the most common ISF patterns but some shaders won't load:

• Function overloading — GLSL allows multiple functions with the same name and different types; WGSL does not. Shaders that rely on this will fail to compile and show an error in the shader name tab.
• Multi-pass ISF (PASSES array) — not supported; only single-pass shaders work.
• color and point2D inputs — parsed but not yet wired to UI controls (Phase 1 scope).
• audio inputs — not supported.

If a shader fails, the output window shows black and the tab name displays the error message.

Getting more shaders

The ISF community library at editor.isf.video has hundreds of free shaders. Download any .fs file and open it with the file picker. Shaders tagged generator run without video input; shaders tagged filter or transition expect a live video source connected in the Input tab.

Shadertoy shaders often work too if they use the mainImage entry-point pattern — download the GLSL, add a minimal ISF header, and load it:

/*{
    "ISFVSN": "2",
    "INPUTS": []
}*/

void mainImage(out vec4 fragColor, in vec2 fragCoord) {
    // paste Shadertoy code here
    // iTime → TIME, iResolution → RENDERSIZE, iChannel0 → inputImage
}

Web App (WebGPU + WASM)

examples/webapp is a self-contained browser application: the delta RGB delay effect compiled to WebAssembly, rendered via WebGPU, with a React overlay for controls.

This is not a remote-control panel for the native engine — it is the engine running inside the browser.

cargo run -p webapp   # doesn't apply — see build steps below

What it is

The Rust code never runs natively — it targets wasm32-unknown-unknown only (#![cfg(target_arch = "wasm32")]). wgpu's BROWSER_WEBGPU backend talks directly to the browser's WebGPU API (Chrome/Edge 113+).

Browser requirements

WebGPU is required. Check compatibility:

The app also requests webcam access (getUserMedia). Serve it over HTTPS or localhost — browsers block camera access on plain HTTP origins.

Building and running

Prerequisites

# Rust WASM target
rustup target add wasm32-unknown-unknown

# Trunk (WASM bundler)
cargo install trunk

# Node.js (for the React UI build)
node --version  # 18+ recommended

First run

cd examples/webapp

# Build the React UI once (Trunk does this automatically if ui/dist is missing)
cd ui && npm install && npm run build && cd ..

# Start the dev server — opens http://localhost:8080
trunk serve

Trunk compiles the Rust WASM, bundles it, copies the React build output from ui/dist/ into the final bundle, and serves everything at http://localhost:8080.

The React UI only needs a rebuild when you change files under ui/src/. The Trunk hook skips the npm build if ui/dist/ already exists, so subsequent trunk serve calls are fast.

Production build

trunk build --release
# Output in dist/ (configurable in Trunk.toml)

The dist/ directory is self-contained — serve it with any static HTTP server.

Architecture

Startup sequence

Browser loads index.html
  └── Trunk loads WASM module
        └── TrunkApplicationStarted fires
              ├── start() — initialise WebGPU device, textures, pipeline
              ├── getUserMedia() — open webcam
              └── requestAnimationFrame loop begins
                    ├── JS: capture webcam frame → RGBA bytes
                    ├── JS: call update_webcam_frame(data, w, h) → WASM
                    └── Rust: render frame with WebGPU

Camera → GPU

The browser has no direct GPU-texture-from-camera API, so frames travel through a CPU copy each tick:

// index.html — JS side
ctx.drawImage(video, 0, 0, w, h);          // draw video into offscreen canvas
const img = ctx.getImageData(0, 0, w, h);  // read RGBA pixels from canvas
update_webcam_frame(data, w, h);           // call into WASM

#![allow(unused)]
fn main() {
// lib.rs — Rust side
#[wasm_bindgen]
pub fn update_webcam_frame(data: &[u8], width: u32, height: u32) {
    // write_buffer → uploads RGBA bytes to the webcam GPU texture
}
}

This is the main performance ceiling for high-resolution inputs — the getImageData call reads from the GPU back to CPU each frame. For 1280×720 it's fine in practice.

WASM exports (window.rustjay)

After startup, the JS side registers the WASM parameter setters on window.rustjay:

window.rustjay = { set_delay_r, set_delay_g, set_delay_b, set_mix };

The React component calls these directly:

// DelaySliders.tsx
const call = useCallback((fn: string, value: number) => {
    window.rustjay?.[fn]?.(value);
}, []);

// on slider change:
call('set_delay_r', newValue);

There is no network round-trip — the React UI and the WASM renderer run in the same browser tab, communicating through thread_local! state:

#![allow(unused)]
fn main() {
thread_local! {
    static PARAMS: RefCell<Params> = RefCell::new(Params::default());
}

#[wasm_bindgen]
pub fn set_delay_r(v: i32) {
    PARAMS.with(|p| p.borrow_mut().delay_r = v.clamp(-64, 64));
}
}

Render loop

The render loop runs via requestAnimationFrame — no Winit, no event loop, no threads. It reads the current PARAMS, uploads uniforms, and runs the WebGPU render pass:

#![allow(unused)]
fn main() {
fn render_frame(app: &mut App) {
    let params = PARAMS.with(|p| *p.borrow());
    // upload uniforms, run render pass, copy to feedback texture
}
}

The feedback texture (previous frame's output) is updated with copy_texture_to_texture at the end of each frame so the delta shader can read it next tick.

The effect

The effect is a simplified version of examples/delta: per-channel pixel offset with a mix between the live camera and the feedback texture.

┌──────────┐     pixel offset (R, G, B independently)
│  Webcam  │ ──────────────────────────────────────────┐
│  (live)  │                                           ↓
└──────────┘                                    ┌─────────────┐
                                                │    WGSL     │ → output
┌──────────┐     sampled with uv offset         │   shader    │
│ Feedback │ ──────────────────────────────────→│             │
│ (t-1)    │                                    └─────────────┘
└──────────┘         ↑
                     └── copy_texture_to_texture each frame

Parameters exposed to the React UI:

Extending the webapp

Adding a parameter

1. Add to the Params struct and export a setter:

#![allow(unused)]
fn main() {
// lib.rs
pub struct Params {
    pub delay_r: i32,
    // ...
    pub brightness: f32,   // add this
}

#[wasm_bindgen]
pub fn set_brightness(v: f32) {
    PARAMS.with(|p| p.borrow_mut().brightness = v.clamp(0.0, 2.0));
}
}

2. Pass it through uniforms:

#![allow(unused)]
fn main() {
// delta.rs
#[repr(C)]
#[derive(Copy, Clone, bytemuck::Pod, bytemuck::Zeroable)]
pub struct DeltaUniforms {
    // existing fields ...
    pub brightness: f32,
    pub _pad: [f32; 3],
}
}

3. Use it in the shader (src/shaders/delta.wgsl):

out = out * u.brightness;

4. Add a slider in React (ui/src/components/DelaySliders.tsx):

<Slider
    label="Brightness"
    value={brightness}
    min={0} max={2} step={0.01}
    onChange={(v) => { setBrightness(v); call('set_brightness', v); }}
    color="#ffee44"
/>

Then rebuild the React UI (cd ui && npm run build) and restart trunk serve.

Replacing the effect

The rendering logic lives in src/delta.rs and src/shaders/delta.wgsl. Swap in a different WGSL shader and update DeltaUniforms to match. The startup, camera capture, and React overlay are all independent of the specific effect.

Project layout

examples/webapp/
├── src/
│   ├── lib.rs          # WASM entry point, render loop, wasm-bindgen exports
│   ├── delta.rs        # Pipeline creation, DeltaUniforms
│   ├── webcam.rs       # update_webcam helper (CPU→GPU texture upload)
│   └── shaders/
│       └── delta.wgsl  # Fragment shader
├── ui/
│   ├── src/
│   │   ├── App.tsx
│   │   └── components/
│   │       └── DelaySliders.tsx   # React control overlay
│   └── package.json
├── index.html          # Trunk entry — wires WASM init, webcam loop, React mount
└── Trunk.toml          # Build config — port 8080, React pre-build hook

Raspberry Pi

rustjay-engine targets four Pi generations with different GPU paths:

Pi 4/5 use the Vulkan path. Pi 2/3 use the OpenGL ES backend via EGL — wgpu selects the best available backend automatically, but GLES must be compiled in (see below).

Pi 4 / Pi 5 — Raspberry Pi OS Bookworm

Runtime packages

sudo apt update && sudo apt install \
    mesa-vulkan-drivers libvulkan1 vulkan-tools \
    libv4l-dev v4l-utils \
    libasound2-dev \
    libwayland-dev libxkbcommon-dev

Verify Vulkan:

vulkaninfo --summary   # should show a V3DV device

Cross-compiling from macOS / Linux

cross build --release --no-default-features --features webcam \
    --target aarch64-unknown-linux-gnu -p sputnik

Pi 2 / Pi 3 — Arch Linux ARM (armv7)

This section documents deploying to a Raspberry Pi 2 Model B running Arch Linux ARM cross-compiled from macOS Apple Silicon. Most steps also apply to Pi 3.

OS (Arch Linux ARM)

Install runtime libraries on the Pi:

sudo pacman -Sy alsa-lib v4l-utils libv4l

For a display server, install a minimal X11 stack:

sudo pacman -Sy xorg-server xorg-xinit xf86-video-fbdev \
    mesa mesa-utils libx11 libxext libxrandr libxinerama \
    libxcursor libxkbcommon-x11

Allow non-console users to run the X server:

sudo bash -c 'echo -e "allowed_users=anybody\nneeds_root_rights=yes" \
    > /etc/X11/Xwrapper.config'

For Wayland (needed for a reliable wgpu EGL display connection — see Known issue):

sudo pacman -Sy weston seatd libdisplay-info
sudo systemctl enable --now seatd
sudo usermod -aG seat alarm

Cross-compiling from Apple Silicon Mac

1. Install cross from git

The published cross 0.2.5 assumes an x86-64 Linux Docker host and tries to install stable-x86_64-unknown-linux-gnu on the macOS ARM host before Docker starts, which fails. Install the latest git HEAD:

cargo install --git https://github.com/cross-rs/cross cross --locked

2. Cross.toml — Docker image + system libs

Cross.toml (workspace root) must specify the armv7 Docker image and install the ALSA/V4L/udev headers inside the container:

[target.armv7-unknown-linux-gnueabihf]
image = "ghcr.io/cross-rs/armv7-unknown-linux-gnueabihf:edge"
pre-build = [
    "dpkg --add-architecture armhf",
    "apt-get update && apt-get install -y libasound2-dev:armhf libv4l-dev:armhf libudev-dev:armhf",
]

3. Workspace feature isolation

Cargo feature resolution is workspace-wide. Without explicit default-features = false at the workspace definition level, all workspace members (delta, waaaves, etc.) contribute their default features — including ndi — to every package's compilation even when building with -p sputnik --no-default-features.

The workspace Cargo.toml must have:

[workspace.dependencies]
rustjay-engine = { path = "crates/rustjay-engine", version = "0.1.0", default-features = false }
rustjay-io     = { path = "crates/rustjay-io",     version = "0.1.0", default-features = false }

Note: setting default-features = false at the package level ({ workspace = true, default-features = false }) does not override the workspace definition in Cargo 1.95. It must be set in [workspace.dependencies].

Examples that need NDI/webcam must opt in explicitly via their own feature flags (delta, waaaves, etc.) or in their dep declaration (delta-egui: features = ["egui", "ndi", "webcam"]).

4. wgpu GLES feature

The Pi 2 has no Vulkan. wgpu must be compiled with the gles feature so it can use Mesa's OpenGL ES via EGL:

# workspace Cargo.toml
wgpu = { version = "29.0", features = ["spirv", "gles"] }

5. Build command

# sputnik (software rendering on Pi 2 — needs llvmpipe for compute shaders)
cross build --release --no-default-features --features webcam \
    --target armv7-unknown-linux-gnueabihf -p sputnik

# flux with DRM/KMS hardware path (no compositor required)
cross build --release --no-default-features --features webcam,drm-gles2 \
    --target armv7-unknown-linux-gnueabihf -p flux

The drm-gles2 feature includes gles2 and adds DRM/KMS + GBM surface support. On Pi 4/5 you can omit these features — the standard wgpu Vulkan path is used instead.

6. Deploy

One-time setup — persistent config on the boot partition:

The Pi root filesystem is remounted read-only by the ro script. Without intervention, config writes (MIDI mappings, presets, OSC port, etc.) silently fail. The /boot partition is always mounted writable and has ample free space (~973 MB).

Auto-detection: flux detects a read-only root at startup and automatically redirects writes to /boot/rustjay-data for that session, creating the directory if needed. You will see Config dir … is read-only; redirecting saves to /boot/rustjay-data in the journal. The steps below make this permanent so no detection is needed on every boot.

Create the directory and migrate existing settings:

ssh alarm@<pi-ip> '
    sudo mkdir -p /boot/rustjay-data/rustjay
    if [ -d /home/alarm/.config/rustjay ]; then
        sudo cp -r /home/alarm/.config/rustjay/. /boot/rustjay-data/rustjay/
    fi
    ls -la /boot/rustjay-data/rustjay/
'

Copy the binary and restart the service:

scp target/armv7-unknown-linux-gnueabihf/release/flux alarm@<pi-ip>:/home/alarm/flux.new
ssh alarm@<pi-ip> '
    sudo systemctl stop flux
    sleep 1
    mv /home/alarm/flux.new /home/alarm/flux
    chmod +x /home/alarm/flux
    sudo systemctl start flux
'

Running on Pi 2

Pi 2's VideoCore IV GPU supports OpenGL ES 2.0 hardware. wgpu requires GLES 3.0 (specifically for Uniform Buffer Objects), so it cannot use the VC4 GPU directly. The two options are:

flux --gles2 --drm: bypasses wgpu AND the Wayland compositor entirely. Opens /dev/dri/card0 directly via KMS, creates a GBM surface, and renders using GLES 2.0 with GLSL ES 1.00 shaders on VC4 hardware. No weston, no X11, no LIBGL_ALWAYS_SOFTWARE.

sputnik uses compute shaders (mesh deformation) which VC4 does not support in hardware at any GLES version, so it remains on llvmpipe.

# Run flux directly on DRM — no compositor at all
RUST_LOG=warn ./flux --nogui --gles2 --drm

# Half display resolution (preserves aspect ratio, good default for Pi 2)
RUST_LOG=warn ./flux --nogui --gles2 --drm --render-scale 0.25

# Run sputnik — still requires software rendering (compute shaders)
XDG_RUNTIME_DIR=/run/user/1000 WAYLAND_DISPLAY=wayland-1 \
    LIBGL_ALWAYS_SOFTWARE=1 RUST_LOG=warn \
    ./sputnik --nogui

Render resolution flags

By default flux renders at the full display resolution. On Pi 2 you almost always want to reduce this:

--render-scale is preferred because it always matches the display's aspect ratio. Using a fixed --render-width/--render-height that differs from the display's aspect ratio will stretch the optical-flow feedback loop and change the visual character of the effect.

Typical values for Pi 2:

• HDMI output (16:9): --render-scale 0.25 → 512×288 at 2048×1152
• Composite NTSC (4:3, 720×480): --render-scale 0.5 → 360×240
• Composite PAL (4:3, 720×576): --render-scale 0.5 → 360×288

Why does flux work but sputnik doesn't? flux uses three plain fragment-shader passes — no UBOs visible to the GLES 2.0 path, no compute, no mesh. sputnik requires compute shaders (GLES 3.1 feature) that VC4 hardware never supports.

Pi 4 / Pi 5 support Vulkan natively. Use the standard ./flux --nogui (no flags needed) and drop LIBGL_ALWAYS_SOFTWARE=1 from sputnik as well.

DRM presentation on vc4: drmModePageFlip returns EBUSY on Pi 2's VC4 driver regardless of flags. flux works around this by calling drmModeSetCrtc each frame. This is not vblank-locked but eglSwapInterval(1) is set so eglSwapBuffers gates on vsync, keeping tearing minimal.

Running headless (--nogui)

Pass --nogui to suppress the control window and open the output fullscreen:

RUST_LOG=warn ./sputnik --nogui

When --nogui is active:

• Only the output window is created — no imgui control panel.
• The output opens fullscreen immediately.
• target_fps is capped at 30 (configurable via the Web UI after launch).
• Audio, MIDI, OSC, and the Web UI all remain fully functional.
• The last-used webcam (stored as startup_webcam_device in the app's config JSON) is attached automatically before the first frame renders.

Webcam auto-attach

Any effect that uses a webcam input will re-attach the same webcam on the next --nogui launch without any user interaction.

How it works: when the engine shuts down it saves the active webcam's device index to ~/.config/rustjay/<app-name>.json as startup_webcam_device. On the next launch the webcam is opened synchronously during engine initialisation, before the first frame is rendered.

First-time setup: run the effect once with the GUI, select the webcam from the Input tab, then quit. The index is written automatically. All subsequent --nogui launches will use it.

Manual override: edit the config JSON directly:

{
  "startup_webcam_device": 0
}

A value of 0 opens /dev/video0 (the first V4L2 capture device). Set to null to disable auto-attach.

Why synchronous? On Pi 2 with software rendering (llvmpipe) a single 1080p frame can take 30+ seconds. The two-step RefreshDevices → StartWebcam queue that works on desktop would never dispatch before the first render completes. The engine therefore starts the webcam directly inside the initialisation path, before handing control to the render loop.

Controlling without a GUI

Settings (MIDI mappings, last preset, FPS target, OSC port) persist automatically — to /boot/rustjay-data/rustjay/<app>.json when the root is read-only, or to ~/.config/rustjay/<app>.json otherwise. MIDI mappings are saved the moment a mapping is learned or removed; presets are saved immediately on write.

Web remote on headless Pi 2 / Pi 3

The Web UI starts automatically when the effect launches. On a headless embedded device you typically want LAN trust mode enabled so anyone on the same network can open the page without a bearer token.

Enable it in the app's config:

{
  "web_host": "0.0.0.0",
  "web_port": 8081,
  "web_lan_trust": true
}

With web_lan_trust: true, opening http://<pi-ip>:8081/flux from a phone or laptop on the same network requires no password. The controls affect the shader in real time.

Four control panels open in separate tabs from the toolbar:

Resource budgeting (Pi 2 / Pi 3)

All rendering on Pi 2/3 runs through llvmpipe on the CPU. The dominant cost is pixel count × pass count.

Flux — three full-screen fragment passes (flow, warp, blit). With --gles2 --drm the passes run on VC4 hardware. Use --render-scale to trade pixel count for framerate:

The optical-flow webcam capture always runs at 640×480 regardless of render scale. --render-scale only controls the internal FBO size for the warp and accumulation passes.

Sputnik — Dial back mesh resolution via the Web UI or a preset:

Use Web UI → Sputnik tab → Mesh Resolution to change at runtime, then save as a preset.

Autostart on boot (Arch Linux ARM)

1. Add the user to the required groups

sudo usermod -aG video,audio alarm
# reboot for the change to take effect

video grants access to /dev/dri/card0 and /dev/video0. audio grants ALSA sequencer access for MIDI.

2. Create the flux service (Pi 2 — DRM/KMS, no compositor)

ExecStartPre=/bin/sleep 3 gives the kernel time to enumerate USB devices before flux opens /dev/video0.

# /etc/systemd/system/flux.service
[Unit]
Description=Flux VJ effect (optical-flow webcam warp — DRM/KMS direct)
After=multi-user.target
Wants=dev-video0.device

[Service]
User=alarm
Environment=RUST_LOG=warn
Environment=XDG_CONFIG_HOME=/boot/rustjay-data
ExecStartPre=/bin/sleep 3
ExecStart=/home/alarm/flux --nogui --gles2 --drm --render-scale 0.25
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

No weston service is needed. flux opens /dev/dri/card0 directly.

For sputnik.service on Pi 2, weston is still required (llvmpipe path). Use the Wayland-based setup from a previous section and add Environment=LIBGL_ALWAYS_SOFTWARE=1.

3. Enable

sudo systemctl daemon-reload
sudo systemctl enable --now flux
journalctl -u flux -f

Verify on the next boot:

systemctl is-active flux        # should print "active"
fuser /dev/dri/card0            # should show the flux PID
fuser /dev/video0               # same PID — webcam open
ps aux | grep weston            # should be empty

Pi 4/5: The --gles2 --drm flags are not needed — use the standard wgpu Vulkan path (./flux --nogui) and a normal Wayland or fullscreen setup.

SD card protection (read-only root)

Unexpected power cuts can corrupt the SD card. Keep the root filesystem read-only during normal operation and remount read-write only when you need to deploy updates or edit configs.

1. Configure journald to use RAM

Stop systemd-journald from writing logs to disk:

sudo mkdir -p /etc/systemd/journald.conf.d
cat << 'EOF' | sudo tee /etc/systemd/journald.conf.d/volatile.conf
[Journal]
Storage=volatile
EOF

2. Create ro / rw scripts

/usr/local/bin/ro — stop writers, sync, remount read-only:

sudo tee /usr/local/bin/ro << 'EOF'
#!/bin/bash
set -e
sudo systemctl stop flux 2>/dev/null || true
sudo systemctl stop systemd-timesyncd 2>/dev/null || true
sudo systemctl stop systemd-journald 2>/dev/null || true
sudo sync
sudo mount -o remount,ro /
echo "Root filesystem is now READ-ONLY"
EOF
sudo chmod +x /usr/local/bin/ro

/usr/local/bin/rw — remount read-write:

sudo tee /usr/local/bin/rw << 'EOF'
#!/bin/bash
set -e
sudo mount -o remount,rw /
echo "Root filesystem is now READ-WRITE"
EOF
sudo chmod +x /usr/local/bin/rw

3. Passwordless sudo

Allow the alarm user to run the scripts without a password:

echo "alarm ALL=(ALL) NOPASSWD: /usr/local/bin/ro, /usr/local/bin/rw, /bin/mount" \
    | sudo tee /etc/sudoers.d/alarm-ro-rw
sudo chmod 440 /etc/sudoers.d/alarm-ro-rw

Workflow

# Normal state — SD card is protected
ro

# Deploy a new build — remount rw, copy binary, then go back to ro
rw
scp target/armv7-unknown-linux-gnueabihf/release/flux alarm@pi:/home/alarm/flux.new
sudo systemctl stop flux
mv /home/alarm/flux.new /home/alarm/flux
sudo systemctl start flux
ro

Before unplugging the power: run ro to ensure all writes are flushed and the filesystem is clean.

Keyboard Shortcuts & Feature Flags

Keyboard shortcuts

These work when the output window has focus.

Feature flags

Add any combination to your Cargo.toml:

[dependencies]
rustjay-engine = {
    git = "https://github.com/BlueJayLouche/rustjay-engine",
    features = ["link", "prodj", "mtc", "ndi"]
}

Default features: ndi is on by default. All others are off.

To disable NDI (e.g. SDK not installed):

rustjay-engine = { git = "...", default-features = false }

Config file location

Per-app settings (MIDI mappings, last-used preset, OSC port) are stored in:

<app-name> is the string returned by your app_name() implementation.

OSC parameter paths

All declared parameters are available at:

/rustjay/<param-id>   f32

Default port: 7770.

Example — set intensity to 0.75:

/rustjay/intensity   0.75

Web remote endpoints

Default port: 3000.

GET  /params                   — all parameters (JSON array)
GET  /params/<id>              — single parameter value
POST /params/<id>  {"value": N} — set a parameter
WS   /ws                       — live update stream

Workspace crates