Powered By GitBook
appengine-ruby-rails-activejob-sidekiq
Learn how to run background jobs using Ruby on Rails ActiveJob.
Jeff Ching | Software Engineer | Google
Contributed by Google employees.
This tutorial shows how to create and configure a Ruby on Rails application to run background processing jobs on the App Engine flexible environment using ActiveJob and Sidekiq.

Objectives

    Create a background processing job.
    Deploy your application to the App Engine flexible environment.
    Verify that background jobs are running.

Before you begin

You'll need the following:
    A Google Cloud project. You can use an existing project or create a new project.
    A Rails 4.2+ application. Follow the
    A Redis instance running in your project. Follow this guide​
    to set up Redis on Compute Engine. This tutorial assumes the Redis instance is running in the default
    network so that the App Engine services can access it without restriction. Save or copy the password that you set for the Redis instance,
    which is used later in this tutorial.

Costs

This tutorial uses billable components of Google Cloud including App Engine flexible environment.
Use the pricing calculator to generate a cost estimate based on your projected usage. Google Cloud users might be eligible for a free trial.

Creating your background job

Starting with version 4.2, Ruby on Rails includes an abstraction layer around background job processing called ActiveJob. This abstraction allows you to write the queuing and execution logic independently of queue and job runner implementations.
Ruby on Rails provides command-line tools for generating templated skeletons for things such as database migrations, controllers, and even background jobs.
You create a job named HelloJob that accepts a name argument and prints "Hello #{name}" to standard output.
    1.
    Use the Rails generator feature to create the HelloJob job:
    1
    bin/rails generate job Hello
    Copied!
    Rails creates stub files from templates:
    1
    invoke test_unit
    2
    create test/jobs/hello_job_test.rb
    3
    create app/jobs/hello_job.rb
    4
    create app/jobs/application_job.rb
    Copied!
    2.
    Edit app/jobs/hello_job.rb with the following:
    1
    class HelloJob < ApplicationJob
    2
    queue_as :default
    3
    ​
    4
    def perform(name)
    5
    # Do something later
    6
    puts "Hello, #{name}"
    7
    end
    8
    end
    Copied!

Create a test URL to queue the job

You create a controller named HelloController that provides an action called say that queues the HelloJob job to run in the background.
    1.
    Use the Rails generator feature to create HelloController:
    1
    bin/rails generate controller Hello
    Copied!
    Rails creates stub files from templates:
    1
    create app/controllers/hello_controller.rb
    2
    invoke erb
    3
    create app/views/hello
    4
    invoke test_unit
    5
    create test/controllers/hello_controller_test.rb
    6
    invoke helper
    7
    create app/helpers/hello_helper.rb
    8
    invoke test_unit
    9
    invoke assets
    10
    invoke coffee
    11
    create app/assets/javascripts/hello.coffee
    12
    invoke scss
    13
    create app/assets/stylesheets/hello.scss
    Copied!
    2.
    Add a say action to HelloController by adding the following to your app/controllers/hello_controller.rb file:
    1
    class HelloController < ApplicationController
    2
    def say
    3
    HelloJob.perform_later(params[:name])
    4
    render plain: 'OK'
    5
    end
    6
    end
    Copied!
    This action will queue our HelloJob with the provided request parameter name.
    3.
    Create a route to this action by adding the following to your config/routes.rb file:
    1
    get '/hello/:name', to: 'hello#say'
    Copied!
    When you make an HTTP GET request to /hello/Jeff, the HelloController will handle the request using the say action with parameter :name as "Jeff".

Configuring your background worker to use Sidekiq

ActiveJob can be configured with various different background job runners. This tutorial uses Sidekiq, which requires a Redis instance to manage the job queue.
    1.
    Add sidekiq gem to your Gemfile:
    1
    bundle add sidekiq
    Copied!
    2.
    Configure ActiveJob to use Sidekiq as its queue adapter. In config/application.rb:
    1
    class Application < Rails::Application
    2
    # ...
    3
    config.active_job.queue_adapter = :sidekiq
    4
    end
    Copied!

Deploying to App Engine flexible environment

For Sidekiq, the Redis connection configuration can be provided as an environment variable at run time. You need to obtain the internal address and password of your Redis instance. In the Cloud Console, go to the VM instances page and find the internal IP address of your Compute Engine instance with Redis installed. This IP address and the password that you saved are be provided through environment variables at deployment time to configure Sidekiq.

Option A: Shared worker and web application

For this option, the App Engine service runs both the web server and a worker process through a process manager called foreman. If you choose this method, App Engine scales your web and worker instances together.
    1.
    Add foreman gem to your Gemfile:
    1
    bundle add foreman
    Copied!
    2.
    Create a Procfile at the root of your application:
    1
    web: bundle exec rails server -p 8080
    2
    worker: bundle exec sidekiq
    Copied!
    3.
    Create an app.yaml for deploying the application to Google App Engine:
    1
    runtime: ruby
    2
    env: flex
    3
    ​
    4
    entrypoint: bundle exec foreman start
    5
    ​
    6
    env_variables:
    7
    REDIS_PROVIDER: REDIS_URL
    8
    REDIS_URL: redis://[REDIS_IP_ADDRESS]:6379
    9
    REDIS_PASSWORD: [PASSWORD]
    10
    SECRET_KEY_BASE: [SECRET_KEY]
    Copied!
    Replace [REDIS_IP_ADDRESS] and [PASSWORD] with the internal IP address of your Redis instance and its required password that you gave it, respectively. Replace [SECRET_KEY] with a secret key for Rails sessions.
    4.
    Deploy to App Engine
    1
    gcloud app deploy app.yaml
    Copied!

Option B: Separate worker and web application

For this option, you create two App Engine services: one runs the web server and one runs worker processes. Both services use the same application code. This configuration allows you to scale background worker instances independently of your web instances at the cost of potentially using more resources. To pass the App Engine health checks and keep your background worker instance alive, you use the sidekiq_alive gem to enable the Sidekiq server to respond to each liveness and readiness request with a 200 HTTP status code.
    1.
    Add sidekiq_alive to your Gemfile:
    1
    bundle add sidekiq_alive
    Copied!
    2.
    Create a sidekiq_alive.rb initializer. In config/initializers:
    1
    SidekiqAlive.setup do |config|
    2
    # ==> Server port
    3
    # Port to bind the server.
    4
    config.port = 8080
    5
    ​
    6
    # ==> Server path
    7
    # HTTP path to respond to.
    8
    config.path = '/health_check'
    9
    ​
    10
    # ==> Rack server
    11
    # Web server used to serve an HTTP response.
    12
    config.server = 'puma'
    13
    end
    Copied!
    3.
    Create an app.yaml file for deploying the web service to App Engine:
    1
    runtime: ruby
    2
    env: flex
    3
    ​
    4
    entrypoint: bundle exec rails server -p 8080
    5
    ​
    6
    env_variables:
    7
    REDIS_PROVIDER: REDIS_URL
    8
    REDIS_URL: redis://[REDIS_IP_ADDRESS]:6379
    9
    REDIS_PASSWORD: [PASSWORD]
    10
    SECRET_KEY_BASE: [SECRET_KEY]
    Copied!
    Replace [REDIS_IP_ADDRESS] and [PASSWORD] with the internal IP address of your Redis instance and its required password that you gave it, respectively. Replace [SECRET_KEY] with a secret key for Rails sessions.
    4.
    Create a worker.yaml file for deploying the worker service to App Engine:
    1
    runtime: ruby
    2
    env: flex
    3
    service: worker
    4
    ​
    5
    entrypoint: bundle exec sidekiq
    6
    ​
    7
    env_variables:
    8
    REDIS_PROVIDER: REDIS_URL
    9
    REDIS_URL: redis://[REDIS_IP_ADDRESS]:6379
    10
    REDIS_PASSWORD: [PASSWORD]
    11
    SECRET_KEY_BASE: [SECRET_KEY]
    12
    ​
    13
    liveness_check:
    14
    path: '/health_check'
    15
    ​
    16
    readiness_check:
    17
    path: '/health_check'
    18
    ​
    19
    # Optional scaling configuration
    20
    manual_scaling:
    21
    instances: 1
    Copied!
    Replace [REDIS_IP_ADDRESS] and [PASSWORD] with the internal IP address of your Redis instance and its required password that you gave it, respectively. Replace the [SECRET_KEY] with a secret key for Rails sessions.
    The path attribute for both the liveness_check and readiness_check sections has been set to the value of config.path in your sidekiq_alive.rb initializer.
    As mentioned above, you can configure scaling for the worker service independent of the default (web) service. In the manual_scaling section, you have configured the worker service to start with one worker instance. For more information on scaling options, see scaling configuration options in app.yaml. If you choose an automatic_scaling option, be aware that scaling for the background processing is based off of CPU utilization, not queue size.
    5.
    Deploy both services to App Engine
    1
    gcloud app deploy app.yaml worker.yaml
    Copied!

Verify that background queuing works

    1.
    In the Cloud Console, go to the App Engine services page. Locate the service that is running your background workers (if option A, it should be the default service, if option B, it should be the worker service). Click Tools > Logs for that service.
    2.
    In a separate window, navigate to your deployed Rails application:
    1
    https://[YOUR_PROJECT_ID].appspot.com/hello/Jeff
    Copied!
    Replace [YOUR_PROJECT_ID] with your Google Cloud project ID.
    3.
    Navigate back to the Logs dashboard. In the Filter by label or text search field, add "Hello, Jeff" and you should see a logging statement like the following if using foreman:
    1
    13:13:52.000 worker.1 | Hello, Jeff
    Copied!
    Or, if using a second service:
    1
    13:13:52.000 Hello, Jeff
    Copied!
Congratulations! You have successfully set up background job processing on App Engine with Sidekiq.

Cleaning up

After you've finished this tutorial, you can clean up the resources you created on Google Cloud so you won't be billed for them in the future. The following sections describe how to delete or turn off these resources.

Deleting the project

The easiest way to eliminate billing is to delete the project you created for the tutorial.
    1.
    In the Cloud Console, go to the Projects page.
    2.
    Click the trash can icon to the right of the project name.
Warning: Deleting a project has the following consequences:
If you used an existing project, you'll also delete any other work you've done in the project. You can't reuse the project ID of a deleted project. If you created a custom project ID that you plan to use in the future, you should delete the resources inside the project instead. This ensures that URLs that use the project ID, such as an appspot.com URL, remain available.

Deleting App Engine services

To delete an App Engine service:
    1.
    In the Cloud Console, go to the App Engine Services page.
    2.
    Click the checkbox next to the service you wish to delete.
    3.
    Click Delete at the top of the page to delete the service.
If you are trying to delete the default service, you cannot. Instead, do the following:
    1.
    Click on the number of versions which will navigate you to App Engine Versions page.
    2.
    Select all the versions you wish to disable and click Stop at the top of the page. This will free
    all of the Google Compute Engine resources used for this App Engine service.

Next steps

Last modified 7mo ago