Get Started with Sidekiq and Upstash Redis on Fly.io
Sidekiq is a popular background processing library in Ruby language. With this library, we can distribute our processes to what is called Sidekiq workers
instead of doing the heavy work on the server itself.
What these workers do basically is they fetch the enqueued or scheduled tasks from the database. Then, execute them whenever the time comes and the worker is idle.
This way, we can make sure that our server remains responsive for incoming requests, without being blocked by heavy or recurring processes.
We will create a sample app using Sidekiq on Fly.io using Upstash Redis. It will enqueue and/or schedule some notification for users, depending on the inputs provided.
Fly.io Setup
Upstash Redis Setup on Fly
- Run
flyctl redis create
. - Set your connection link (after the
.toml
creation):flyctl secrets set UPSTASH_REDIS_LINK=<upstash-redis-connection-link>
Application Code
bundle init
andbundle add sidekiq sinatra
to initiate Gemfile and install dependencies. `- Create
sendEmail.rb
file:
require "sidekiq"
require "sidekiq/api"
connection_url = ENV['UPSTASH_REDIS_LINK']
Sidekiq.configure_client do |config|
config.redis = {url: connection_url}
end
Sidekiq.configure_server do |config|
config.redis = {url: connection_url}
end
class EmailService
include Sidekiq::Worker
def perform(id, type)
# Logic goes here. Let's assume sending email by printing to console.
puts "Emailed to: " + id + ": " + "'Congrats on " + type + " plan.'"
end
end
def updateEmail(id, newType)
jobFound = false
a = Sidekiq::ScheduledSet.new
a.each do |job|
if job.args[0] == id
job.delete
jobFound = true
end
end
if jobFound
EmailService.perform_async(id, ("starting using our service and upgrading it to " + newType))
else
EmailService.perform_async(id, ("upgrading to " + newType))
end
end
def sendEmail(id, type)
case type
when "free"
# if free, delay for 10 seconds.
EmailService.perform_in("10", id, "free")
when "paid"
# if paid, delay for 5 seconds.
EmailService.perform_in("5", id, "paid")
when "enterprise"
# if enterprise, immediately queue.
EmailService.perform_async(id, "enterprise")
when "enterprise10k"
EmailService.perform_async(id, "enterprise10k")
else
puts "Only plans are: `free`, `paid` and `enterprise`"
end
end
def clearSchedules()
Sidekiq::ScheduledSet.new.clear
Sidekiq::Queue.new.clear
end
require "sidekiq"
require "sidekiq/api"
connection_url = ENV['UPSTASH_REDIS_LINK']
Sidekiq.configure_client do |config|
config.redis = {url: connection_url}
end
Sidekiq.configure_server do |config|
config.redis = {url: connection_url}
end
class EmailService
include Sidekiq::Worker
def perform(id, type)
# Logic goes here. Let's assume sending email by printing to console.
puts "Emailed to: " + id + ": " + "'Congrats on " + type + " plan.'"
end
end
def updateEmail(id, newType)
jobFound = false
a = Sidekiq::ScheduledSet.new
a.each do |job|
if job.args[0] == id
job.delete
jobFound = true
end
end
if jobFound
EmailService.perform_async(id, ("starting using our service and upgrading it to " + newType))
else
EmailService.perform_async(id, ("upgrading to " + newType))
end
end
def sendEmail(id, type)
case type
when "free"
# if free, delay for 10 seconds.
EmailService.perform_in("10", id, "free")
when "paid"
# if paid, delay for 5 seconds.
EmailService.perform_in("5", id, "paid")
when "enterprise"
# if enterprise, immediately queue.
EmailService.perform_async(id, "enterprise")
when "enterprise10k"
EmailService.perform_async(id, "enterprise10k")
else
puts "Only plans are: `free`, `paid` and `enterprise`"
end
end
def clearSchedules()
Sidekiq::ScheduledSet.new.clear
Sidekiq::Queue.new.clear
end
- Create a simple Sinatra server
app.rb
:
#!/usr/bin/env ruby
require 'rubygems'
require 'bundler/setup'
require 'sinatra'
require './sendEmail.rb'
get '/register/:id/:plan' do
sendEmail(params[:id], params[:plan])
"<h1>Email is queued.</h1>"
end
get '/update/:id/:plan' do
updateEmail(params[:id], params[:plan])
"<h1>Updated Email is queued.</h1>"
end
get '/clear' do
clearSchedules()
"<h1>Queue and the schedules are cleared.</h1>"
end
#!/usr/bin/env ruby
require 'rubygems'
require 'bundler/setup'
require 'sinatra'
require './sendEmail.rb'
get '/register/:id/:plan' do
sendEmail(params[:id], params[:plan])
"<h1>Email is queued.</h1>"
end
get '/update/:id/:plan' do
updateEmail(params[:id], params[:plan])
"<h1>Updated Email is queued.</h1>"
end
get '/clear' do
clearSchedules()
"<h1>Queue and the schedules are cleared.</h1>"
end
- Create
config.ru
file:
# config.ru
require './app.rb'
require 'sidekiq'
require "sidekiq/api"
run Sinatra::Application
# config.ru
require './app.rb'
require 'sidekiq'
require "sidekiq/api"
run Sinatra::Application
Launch App
- First, run
flyctl launch
to create.toml
template. - There, update
internal_port = 9292
under[[services]]
. It should look something like this:
app = "sidekiq-example"
kill_signal = "SIGINT"
kill_timeout = 5
processes = []
[build]
builder = "heroku/buildpacks:20"
[env]
PORT = "8080"
[[services]]
internal_port = 9292
protocol = "tcp"
[services.concurrency]
hard_limit = 25
soft_limit = 20
type = "connections"
[[services.ports]]
handlers = ["http"]
port = 80
[[services.ports]]
handlers = ["tls", "http"]
port = 443
app = "sidekiq-example"
kill_signal = "SIGINT"
kill_timeout = 5
processes = []
[build]
builder = "heroku/buildpacks:20"
[env]
PORT = "8080"
[[services]]
internal_port = 9292
protocol = "tcp"
[services.concurrency]
hard_limit = 25
soft_limit = 20
type = "connections"
[[services.ports]]
handlers = ["http"]
port = 80
[[services.ports]]
handlers = ["tls", "http"]
port = 443
- Create a file named
Procfile
so that we can run both the Sinatra server and the Sidekiq Worker:
web: bundle exec rackup & bundle exec sidekiq -r ./sendEmail.rb
web: bundle exec rackup & bundle exec sidekiq -r ./sendEmail.rb
- run
flyctl deploy
to deploy.
App Deployed
-
send
get register/id1/enterprise
viacurl <fly-domain>/register/:id/:plan
-
send
get register/id2/paid
viacurl <fly-domain>/register/:id/:plan
-
send
get register/id3/free
and thenget update/id3/paid
viacurl <fly-domain>/register/:id/:plan
andcurl <fly-domain>/update/:id/:newPlan
Closing Words
In this example, we have implemented an app, using Sidekiq background processing library. In the sample, we have demonstrated a simple usecase which you can customize depending on your needs. This way, we have decreased the workload of our main server and with that enabled for more responsive experience for users.
For the complete source code, you can look at our repo.
You can give your feedbacks on any of our services via twitter or discord, which will be greatly appreciated.