Installation

Rust Toolchain

Volley requires a stable Rust toolchain. Install via rustup:

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
rustup default stable

System Dependencies

Most crates build with a standard Rust toolchain. Some connectors need additional system packages:

Note: libcurl and OpenSSL are vendored and statically linked by rdkafka — no system packages needed.

Run volley doctor to check your environment for missing dependencies.

Install the CLI

curl -fsSL https://raw.githubusercontent.com/volley-streams/volley/main/install.sh | sh

Or with Cargo:

cargo install --git https://github.com/volley-streams/volley volley-cli

Adding Volley to a Project

Scaffold a new pipeline

# Interactive mode — prompts for source, sink, and processing pattern
volley new my-pipeline

# Non-interactive with explicit flags (comma-separate to chain operators)
volley new --source kafka --sink s3 --sink-format delta --processing filter my-pipeline
volley new --source kafka --sink kafka --processing filter,ml_inference my-enriched

# Built-in templates bypass source/sink prompts
volley new --template observability-demo my-demo
volley new --template ml-pipeline my-classifier

cd my-pipeline
volley doctor
cargo build && cargo run

Built-in templates: observability-demo, ml-pipeline. Without --template, the CLI prompts for source (kafka, s3, azure-blob), sink (kafka, s3, azure-blob), and processing operators (filter, keyed_agg, windowed_agg, ml_inference, adbc_enrichment, custom). Select multiple operators to chain them in the generated pipeline.

Add to an existing project

Add Volley crates to your Cargo.toml:

[dependencies]
volley-core = { git = "https://github.com/volley-streams/volley", features = ["state-rocksdb"] }
volley-connector-kafka = { git = "https://github.com/volley-streams/volley" }

Note: Volley currently uses git dependencies. crates.io publishing is planned — see the roadmap.

Feature Flags

The volley-connectors crate uses feature flags to control which connectors are compiled:

Tip: If you don’t need the RocksDB state backend (e.g. you’re only developing a Kafka connector), use default-features = false and enable only the features you need to save ~1.5 GB in build artifacts.

Troubleshooting

cmake not found

error: failed to run custom build command for `rdkafka-sys`
--- stderr
CMake not found

Install cmake: brew install cmake (macOS) or apt install cmake (Ubuntu).

RocksDB compilation slow or failing

RocksDB is a C++ dependency that takes ~3–5 minutes to compile on first build. If you don’t need state management (e.g., stateless transformations only), disable it:

volley-core = { git = "...", default-features = false }

protobuf-compiler not found

Only needed if building volley-scheduler. Install: brew install protobuf (macOS) or apt install protobuf-compiler (Ubuntu).

volley doctor reports issues

volley doctor checks for system dependencies, Docker availability, and toolchain versions. Address each reported issue individually. Common fixes:

• Docker not running: Start Docker Desktop or the Docker daemon
• Rust toolchain outdated: rustup update stable
• Missing system packages: See the System Dependencies table above

Verify Your Setup

# Build the workspace
cargo build --workspace

# Run tests
cargo test --workspace --exclude volley-python

# Run a quick example
cargo run --example in_memory_pipeline -p volley-examples