High-Performance Event Streaming for Go Developers
StreamFlow is a lightweight, type-safe stream processing framework built in Go. Ingest, aggregate, and join event streams from Kafka, NATS, and RabbitMQ with sub-microsecond latency, embedded LSM-tree state, and zero JVM operational overhead.
go install github.com/santiagolertora/streamflow/cmd/streamflow@latest
Static Go binary with zero external dependencies. Eliminates Java heap sizing, GC tuning, and 1GB+ container baselines.
The Problems We Solve
Real-time event processing is plagued by operational overhead, latency bottlenecks, and fragile code.
⚠️ The JVM Resource Tax
The Pain: Running Apache Flink or Kafka Streams microservices requires 1GB to 4GB of baseline RAM per container, inflating cloud bills.
✨ The Cure: StreamFlow runs natively in Go, using less than 50MB of baseline RAM per instance, allowing you to run dozens of services on minimal hardware.
⚠️ High-Latency External State
The Pain: Relying on Redis, DynamoDB, or PostgreSQL for stateful aggregations (window averages, session counters) introduces 2ms to 10ms network roundtrip latency per event.
✨ The Cure: Embedded PebbleDB (an LSM-tree key-value store modeled after RocksDB) holds state locally inside the application process for 0ms network latency.
⚠️ Silent Runtime Type Panics
The Pain: Raw byte stream pipelines ([]byte) fail silently or panic in production when upstream teams push unexpected schema changes.
✨ The Cure: StreamFlow's strongly-typed Generics DSL ensures that schema incompatibilities fail at compile-time, not at 3 AM.
⚠️ Fragile and Flaky Tests
The Pain: Testing session window timeouts or sliding joins requires setting up Docker environments, using mock stubs, or inserting fragile time.Sleep() statements.
✨ The Cure: The streamflow/testing harness provides a deterministic memory store and time-travel simulation (AdvanceTime), allowing you to test time-based windows instantly and reliably in standard unit tests.
⚠️ Complex Polyglot Deployment
The Pain: Running custom Python or Rust models inside your pipeline requires deploying separate microservices or using slow, insecure gRPC sidecars.
✨ The Cure: Load polyglot logic dynamically as secure, sandboxed WASM plugins powered by wazero with sub-microsecond in-process invocation.
Why StreamFlow?
🚀 Maximum Cost Efficiency
Reduce your streaming infrastructure costs by up to 90%. By replacing memory-heavy Java runtimes with compiled Go static binaries, you can pack more streaming microservices on smaller virtual machines.
🚀 Sub-Millisecond SLAs
Garbage collection pauses in JVM frameworks cause random latency spikes. StreamFlow's zero-copy, low-allocation Go processing loop ensures predictable sub-millisecond p99 latencies under peak traffic.
🚀 Exceptional Developer Velocity
Write pipelines faster with a clean, fluent Go DSL. Benefit from full IDE autocompletion, type safety, refactoring support, and instant compilation feedback.
🚀 Operational Simplicity
No JVM clusters, ZooKeeper dependencies, or complicated configuration frameworks to maintain. Build, compile, and deploy a single static binary (< 30MB) using standard container runtimes.
The Metrics That Matter
Measured executing in-memory pipeline steps on standard cloud hardware.
Core Features
🔹 Type-Safe Go Generics DSL
No Runtime Panics: Elevate binary streams into strongly typed Go structs. Schema changes fail at compile time.
IDE Autocomplete: Full autocomplete, refactoring, and static analysis across filters, maps, and windows.
🔹 Embedded Stateful LSM Engine
No External Database Hops: Aggregate totals and session averages locally via embedded PebbleDB without Redis/Postgres roundtrips.
Active Disk Compaction: Auto-purges expired window states and join indexes from storage.
🔹 Declarative Streaming SQL
In-Process SQL Queries: Run SELECT category, amount FROM orders WHERE amount BETWEEN 100 AND 200 directly over live streams.
Dynamic Source Routing: Auto-configures broker subscriptions based on the FROM clause.
🔹 Sandboxed WASM Plugins
Polyglot Event Pipelines: Run custom enrichers written in Rust, C++, Python, or Go compiled to WebAssembly.
WASM Instance Pooling: Pre-instantiates module pools to scale with parallel engine threads without lock contention.
🔹 Effectively-Once Semantics (EOS)
Achieve transactional consistency via idempotent database upserts and transaction-aligned offset commits.
🔹 Fault-Tolerant DLQ
Tag failing messages with error metadata and route them to a Dead Letter Queue (DLQ) without halting stream ingestion.
Build Pipelines Your Way
Select a paradigm to see how simple it is to build with StreamFlow.
Architectural Blueprint
Key-Partitioned Worker Pools
To scale throughput across CPU cores while guaranteeing message order per key, StreamFlow hashes keys using FNV-1a. Messages with the same key are assigned to the same worker goroutine, bypassing CPU-thrashing mutex locks.
Zero-Mock Time-Travel Testing
StreamFlow includes a dedicated testing harness (streamflow/testing). Write unit tests for session windows, sliding window joins, and database sink recovery, and fast-forward time instantly (AdvanceTime) without flaky sleeps or fragile mock interfaces.
Get Started in 60 Seconds
pipeline.toml)[pipeline.source]
type = "kafka"
brokers = "localhost:9092"
topic = "raw-events"
group_id = "analytics-consumer"
[pipeline.sink]
type = "postgres"
dsn = "postgres://user:pass@localhost:5432/db"
table = "analytics_metrics"
Ready to Build Sub-Microsecond Streaming Pipelines?
Check out the StreamFlow GitHub repository or consult with Binlogic's data engineers for enterprise pipeline design.