PeregrineQueue

Overview

An elixir/phoenix based queue that uses gRPC to call workers in a language agnostic way.

I wanted to play around with gRPC and had an idea to make a task pool type thing. You can use any language you want to register workers through gRPC. The workers can be configured as either push or pull. Pull workers call and endpoint and get whatever is ready off the top of the stack. Push workers register with a heartbeat and the job orchestrator sends jobs to the available workers in the pool.

What I learned

I like gRPC. It’s a really neat system. You basically get to define your contract, and the gRPC package in your chosen language generates your API clients. I had a lot of struggles with internal networking between workers both on flyio and my own local coolify deployment. But I think a lot of that comes from trying to deploy the workers and queue as one unit. In a real use case the workers would be their own seperate deployment and system.

Stack

Elixir, Phoenix, LiveView, gRPC