PN Tech

Implementing PostgreSQL full-text search in Rails

2022-03-17T16:30:00+00:00

Introduction

At Precision Nutrition, we recently launched an integrated search feature across some of our front-end applications. This feature provides a single search experience for users to be able to find content from across their courses, as well as curated links to resources from our blog, YouTube channel, and elsewhere.

For a first iteration of the integrated search, we wanted to get an MVP in our users’ hands quickly, at a low cost, and without adding devops work. Given that we use PostgreSQL as our primary data store, we decided to implement the initial version of the feature using PostgreSQL full-text search.

In this post, I’m going to walk you through implementing basic PostgreSQL full-text search in a Rails app. By the end, we’ll have a working GET /search endpoint with a user-friendly query syntax, highlighting of matched search terms, and sorting by search relevance. We’ll also set up database triggers to keep our search index up-to-date as source records are created, updated, and deleted.

This is the integrated search interface we needed to support, including results from multiple Rails models.

The walkthrough

Step 0: Setup

For the purposes of this walkthrough, let’s keep things simple. We have one model, **HelpfulLink**. In our app, helpful links are visible to everyone. They have three attributes we care about: title, description, and url.

Step 1: Create the `SearchDoc` model

While we only have one model for now, let’s imagine that we plan to add more in the near future: maybe user-specific Coupon records with URLs we’ll need to generate on the fly, Product listings, and more. We want our search feature to be able to return results for any record type we might come up with.

To provide a single search endpoint that returns links to URLs associated with different record types, let’s create a new record type, **SearchDoc**. Here’s the migration to create the search_docs table:

# db/migrate/20220211225723_create_search_docs.rb

class CreateSearchDocs < ActiveRecord::Migration[7.0]
  def change
    create_table :search_docs do |t|
      t.tsvector :tsv, null: false, index: { using: :gin }
      t.string :title, null: false
      t.string :body, null: false
      # this creates two columns, `searchable_id` and `searchable_type`
      t.references :searchable, polymorphic: true, null: false

      # We manually declare the timestamps here because we will be
      # exclusively creating/updating search docs using postgres
      # functions, and therefore will be skipping ActiveRecord
      # callbacks that would otherwise populate the timestamp fields.
      t.timestamp :created_at, null: false, default: -> { 'now()' }
      t.timestamp :updated_at, null: false, default: -> { 'now()' }

      t.index [:searchable_id, :searchable_type], unique: true
    end
  end
end

Let’s walk through the migration:

t.tsvector :tsv, null: false, index: { using: :gin }

This column, which we name tsv, is where we’ll store the processed text to search against, using the tsvector data type. Since we plan to search against this column frequently, we index it with the gin index type, which is the recommended index for tsvector columns.

t.string :title, null: false
t.string :body, null: false

We store the raw title and body fields in addition to the processed tsv so that we can render the highlighted title and body in the search results.

t.references :searchable, polymorphic: true, null: false

The searchable relationship is a reference to the source record. In the case of this example, we will use the search_docs.searchable_type and search_docs.searchable_id fields to join against the helpful_links table in order to return the url of the HelpfulLink in the API response.

t.timestamp :created_at, null: false, default: -> { 'now()' }
t.timestamp :updated_at, null: false, default: -> { 'now()' }

We manually declare the create_at and update_at timestamps here (rather than using the t.timestamps macro) because we will be using Postgres triggers, rather than ActiveRecord, to manage the lifecycle of SearchDoc records, meaning that ActiveRecord create and update lifecycle hooks that would otherwise populate the timestamp fields are never run.

The initial SearchDoc model is barebones, with only a reference to the searchable source record.

# app/models/search_doc.rb

class SearchDoc < ApplicationRecord
  belongs_to :searchable, polymorphic: true
end

Step 2: `rake` task to add some `SearchDocs` for `HelpfulLink` records

After we’ve run the above migration, let’s create a rake task to seed SearchDoc entries for the existing HelpfulLink records in our database.

First, we’ll write a Postgres function to create a SearchDoc record based on a HelpfulLink. The function takes one argument, the ID of a source HelpfulLink, and upserts (creates or updates) a SearchDoc based on the source record.

/* app/sql/upsert_helpful_link_search_doc.sql */

CREATE OR REPLACE FUNCTION upsert_helpful_link_search_doc(helpful_link_to_reindex_id bigint) RETURNS void AS $$

BEGIN
  INSERT INTO search_docs (searchable_id, searchable_type, title, body, tsv, updated_at)
    SELECT
      helpful_links.id searchable_id,
      'HelpfulLink' searchable_type,
      coalesce(helpful_links.title, '') title,
      coalesce(helpful_links.description, '') body,
      setweight(
        to_tsvector(
          'pg_catalog.english',
          coalesce(helpful_links.title, '')
        ),
        'A'
      ) ||
      setweight(
        to_tsvector(
          'pg_catalog.english',
          coalesce(helpful_links.description, '')
        ),
        'D'
      )
      tsv,
      now() updated_at
    FROM helpful_links
    WHERE id = helpful_link_to_reindex_id

  ON CONFLICT (searchable_id, searchable_type)
  DO UPDATE
  SET title = excluded.title,
      body = excluded.body,
      tsv = excluded.tsv,
      updated_at = excluded.updated_at;
END;
$$ LANGUAGE plpgsql;

With respect to full-text search, the key part is:

/* ... */
setweight(
  to_tsvector(
    'pg_catalog.english',
    coalesce(helpful_links.title, '')
  ),
  'A'
) ||
setweight(
  to_tsvector(
    'pg_catalog.english',
    coalesce(helpful_links.description, '')
  ),
  'D'
)
tsv,
/* ... */

Looking at this snippet from the inside out:

We use the coalesce function on the raw text to provide a default value of '' if the text is NULL. This ensures that the call to to_tsvector always returns a tsvector, since to_tsvector returns NULL if it is passed a NULL text document. (You could argue that the NOT NULL constraint on the title and description columns means this should never happen, but there isn’t really a downside to using coalesce here, so safety first!)
The to_tsvector function converts the coalesced source text into the tsvector data type.
The setweight function takes a tsvector and a character 'A', 'B', ‘C', or 'D' (with 'A' having the highest weight), and returns a tsvector with weighted lexemes.
The two tsvector values (one for title with a weight of 'A', the other for description with a weight of 'D') are then merged together with the || operator, to create a single tsvector value.
- By merging the two tsvector values together instead of first merging the raw title and description text values together, we are able to assign different weights to the two pieces of text, while still storing them in the single resulting tsvector.
The merged tsvector is assigned to the tsv field of the new SearchDoc record.

After writing the function, we need to add it to our database schema. We can do so with a migration:

# db/migrate/20220223221910_add_upsert_helpful_link_search_doc_function.rb

class AddUpsertHelpfulLinkSearchDocFunction < ActiveRecord::Migration[7.0]
  def up
    execute Rails.root.join('app/sql/upsert_helpful_link_search_doc.sql').read
  end

  def down
  end
end

After running the migration, let’s create a rake task to backfill SearchDoc records for all of our existing HelpfulLinks:

# lib/tasks/backfill_search_docs.rake

namespace :search_docs do
  desc 'index helpful links for search'
  task index_helpful_links: :environment do
    execute <<~SQL
      SELECT upsert_helpful_link_search_doc(id) FROM helpful_links;
    SQL
  end

  def execute(string)
    ActiveRecord::Base.connection.execute(string)
  end
end

Run the rake task…

$ rails search_docs:index_helpful_links

…then open the rails console and confirm that the correct SearchDoc records were created:

$ rails c

Loading development environment (Rails 7.0.2)

irb(main):001:0> SearchDoc.all

  SearchDoc Load (1.5ms)  SELECT "search_docs".* FROM "search_docs"
=>
[#<SearchDoc:0x00000001336e2a38
  id: 1,
  title: "Wikipedia: Cat food",
  body: "Starting from first principles",
  tsv: "'cat':2A 'first':6 'food':3A 'principl':7 'start':4 'wikipedia':1A",
  searchable_type: "HelpfulLink",
  searchable_id: 1,
  created_at: Tue, 15 Mar 2022 20:55:13.611931000 UTC +00:00,
  updated_at: Tue, 15 Mar 2022 20:55:13.611931000 UTC +00:00>,
 #<SearchDoc:0x00000001364d7bc8
  id: 2,
  title: "Organic kitty treats",
  body: "For fancy cats who are healthy too",
  tsv: "'cat':6 'fanci':5 'healthi':9 'kitti':2A 'organ':1A 'treat':3A",
  searchable_type: "HelpfulLink",
  searchable_id: 2,
  created_at: Tue, 15 Mar 2022 20:55:13.611931000 UTC +00:00,
  updated_at: Tue, 15 Mar 2022 20:55:13.611931000 UTC +00:00>,
 #<SearchDoc:0x00000001364d7a60
  id: 3,
  title: "Annex Cat Rescue",
  body: "Find a fluffy friend to feed treats to",
  tsv: "'annex':1A 'cat':2A 'feed':9 'find':4 'fluffi':6 'friend':7 'rescu':3A 'treat':10",
  searchable_type: "HelpfulLink",
  searchable_id: 3,
  created_at: Tue, 15 Mar 2022 20:55:13.611931000 UTC +00:00,
  updated_at: Tue, 15 Mar 2022 20:55:13.611931000 UTC +00:00>]

Nice.

Step 3: Implement the basic `GET /search` endpoint

Now that we have some search documents in the database, let’s create an API endpoint so that the user can search for helpful links. We’re going to add a GET /search endpoint that accepts a query query parameter and returns a JSON payload with an array of hits.

# app/models/search_doc.rb

class SearchDoc < ApplicationRecord
  belongs_to :searchable, polymorphic: true

  default_scope { select('search_docs.*') }

  scope :query, lambda { |webquery|
    joins(
      sanitize_sql([
        ", websearch_to_tsquery('english', '%s') query",
        webquery
      ])
    )
      .where('query @@ tsv')
  }
end

# config/routes.rb

Rails.application.routes.draw do
  get 'search', to: 'search#index'
end

# app/controllers/search_controller.rb

class SearchController < ApplicationController
  def index
    records = fetch_results

    payload = { hits: serialize(records) }

    render json: payload
  end

  private

  def fetch_results
    SearchDoc
      .includes(:searchable)
      .query(search_params[:query])
  end

  def search_params
    params.permit(:query)
  end

  def serialize(results)
    results.map { |result|
      {
        body: result.body,
        searchable_id: result.searchable_id,
        searchable_type: result.searchable_type,
        title: result.title,
        url: result.searchable.url,
      }
    }
  end
end

There’s a lot going on here, especially in the SearchDoc model. Let’s walk through the trickiest bits.

default_scope { select('search_docs.*') }

We need to add this default scope because the select clauses added by other scopes override the implicit select search_docs.*.

scope :query, lambda { |webquery|
  joins(
    sanitize_sql([
      ", websearch_to_tsquery('english', '%s') query",
      webquery
    ])
  )
    .where('query @@ tsv')
}

The websearch_to_tsquery Postgres function takes a text query and returns a tsquery value. (tsquery is a data type used to query tsvector documents.) websearch_to_tsquery supports some basic search operators familiar to users of typical websearch interfaces. For example, text inside quotation marks like "quoted text" is treated as an exact phrase; and you can exclude a word by prefixing it with the - character, like cat treats -tuna.
@@ is the match operator. The expression query @@ tsv evaluates to true if the tsvector document matches the tsquery query.

Let’s try out our new endpoint in the terminal. First, start a rails server in the terminal. Then in another terminal window, send a search request with curl (formatting the JSON response with jq):

$ curl -sS http://localhost:3000/search\?query\=treat | jq

{
  "hits": [
    {
      "body": "For fancy cats who are healthy too",
      "searchable_id": 2,
      "searchable_type": "HelpfulLink",
      "title": "Organic kitty treats",
      "url": "https://duckduckgo.com/?q=organic+cat+treats"
    },
    {
      "body": "Find a fluffy friend to feed treats to",
      "searchable_id": 3,
      "searchable_type": "HelpfulLink",
      "title": "Annex Cat Rescue",
      "url": "https://annexcatrescue.ca/"
    }
  ]
}

Heck yeah, we have a working search endpoint!

Step 4: Add highlighting

Now that we have our search endpoint running, let’s give the user some context on why a given search result was returned in response to their query.

The ts_headline PostgreSQL function accepts some text and a tsquery and returns an excerpt from the document with terms from the query highlighted.

By default, “highlighted” means “wrapped in <b></b> tags”, which isn’t very useful these days. To get around this behaviour, we can pass optional StartSel and StopSel parameters to tell ts_headline what we would like to wrap the highlighted text in. For example, we can use StartSel=[HIGHLIGHT], StopSel=[/HIGHLIGHT] to have the highlights returned wrapped in [HIGHLIGHT][/HIGHLIGHT], which we can then parse on the front-end and turn into whatever HTML we want.

First we add a with_highlights scope to the model.

# app/models/search_doc.rb

class SearchDoc < ApplicationRecord
  belongs_to :searchable, polymorphic: true

  default_scope { select('search_docs.*') }

  scope :with_highlights, lambda {
    selection_options = 'StartSel = [HIGHLIGHT], StopSel = [/HIGHLIGHT]'

    scope = select <<~SQL
      ts_headline('english', search_docs.title, query, '#{selection_options}') highlighted_title,
      ts_headline('english', left(search_docs.body, 2500), query, '#{selection_options}') highlighted_body
    SQL

    merge(scope)
  }

  scope :query, lambda { |webquery|
    # ... same as before
  }
end

(Note that we use left() to truncate the body to the first 2500 characters. ts_headline is an expensive function, so we want to avoid passing it large amounts of text. The tradeoff is that if the matching text doesn’t appear in the first 2500 characters of the body, it won’t show up in the highlighted text, which can be confusing to the user.)

Next, we use the with_highlights scope and the new highlighted_body and highlighted_title in the search controller.

# app/controllers/search_controller.rb

class SearchController < ApplicationController
  def index
    # ... same as before
  end

  private

  def fetch_results
    SearchDoc
      .includes(:searchable)
      .with_highlights
      .query(search_params[:query])
  end

  def search_params
    # ... same as before
  end

  def serialize(results)
    results.map { |result|
      {
        body: result.highlighted_body,
        searchable_id: result.searchable_id,
        searchable_type: result.searchable_type,
        title: result.highlighted_title,
        url: result.searchable.url,
      }
    }
  end
end

We now have highlighting in the search results:

$ curl -sS http://localhost:3000/search\?query\=treat | jq

{
  "hits": [
    {
      "body": "For fancy cats who are healthy too",
      "searchable_id": 2,
      "searchable_type": "HelpfulLink",
      "title": "Organic kitty [HIGHLIGHT]treats[/HIGHLIGHT]",
      "url": "https://duckduckgo.com/?q=organic+cat+treats"
    },
    {
      "body": "Find a fluffy friend to feed [HIGHLIGHT]treats[/HIGHLIGHT] to",
      "searchable_id": 3,
      "searchable_type": "HelpfulLink",
      "title": "Annex Cat Rescue",
      "url": "https://annexcatrescue.ca/"
    }
  ]
}

Step 5: Add sorting by `rank`

With our search results highlighted, we’re very close to having a useful, basic search endpoint. The missing piece is to ensure that the most relevant search results are shown at the top of the list of search hits. Currently, if we search for cat -wikipedia (results that match cat, excluding results that match wikipedia), we get:

$ curl -sS http://localhost:3000/search\?query\=cat\%20-wikipedia | jq

{
  "hits": [
    {
      "body": "For fancy [HIGHLIGHT]cats[/HIGHLIGHT] who are healthy too",
      "searchable_id": 2,
      "searchable_type": "HelpfulLink",
      "title": "Organic kitty treats",
      "url": "https://duckduckgo.com/?q=organic+cat+treats"
    },
    {
      "body": "Find a fluffy friend to feed treats to",
      "searchable_id": 3,
      "searchable_type": "HelpfulLink",
      "title": "Annex [HIGHLIGHT]Cat[/HIGHLIGHT] Rescue",
      "url": "https://annexcatrescue.ca/"
    }
  ]
}

However, based on the weights we set in the upsert_helpful_link_search_doc function, results with a match in the title are more relevant and should be displayed before those with a match in the body.

In order to sort our results taking this into account, we can calculate the rank of each search result and sort the results in descending order based on rank.

# app/models/search_doc.rb

class SearchDoc < ApplicationRecord
  belongs_to :searchable, polymorphic: true

  default_scope { select('search_docs.*') }

  scope :with_rank, lambda {
    merge(select('ts_rank_cd(search_docs.tsv, query, 32) as rank'))
  }

  # ... and the rest is the same as before
end

The ts_rank_cd function computes the cover density ranking for a tsvector document and a query. (The ts_rank function can also be used to calculate the rank of documents for a query, but it ignores the position and weight of the lexemes in the document.)

In the search controller, we can now use the with_rank scope, and order the search results in descending order by rank.

# app/controllers/search_controller.rb

class SearchController < ApplicationController
  def index
    # ...
  end

  private

  def fetch_results
    SearchDoc
      .includes(:searchable)
      .with_rank
      .with_highlights
      .query(search_params[:query])
      .order({ rank: :desc })
  end

  # ...
end

The search results are now returned with the relative weighting of title and body taken into account.

$ curl -sS http://localhost:3000/search\?query\=cat\%20-wikipedia | jq

{
  "hits": [
    {
      "body": "Find a fluffy friend to feed treats to",
      "searchable_id": 3,
      "searchable_type": "HelpfulLink",
      "title": "Annex [HIGHLIGHT]Cat[/HIGHLIGHT] Rescue",
      "url": "https://annexcatrescue.ca/"
    },
    {
      "body": "For fancy [HIGHLIGHT]cats[/HIGHLIGHT] who are healthy too",
      "searchable_id": 2,
      "searchable_type": "HelpfulLink",
      "title": "Organic kitty treats",
      "url": "https://duckduckgo.com/?q=organic+cat+treats"
    }
  ]
}

Step 6: Add a SQL trigger to create/update/delete `SearchDoc` records when `HelpfulLink` records change

As a finishing touch, let’s add a database trigger to ensure that new SearchDoc records are created when HelpfulLink records are added to the database, and that any changes to an existing HelpfulLink record are reflected in its associated SearchDoc.

Fortunately, we did all the hard work when we wrote the upsert_helpful_link_search_doc function. We can use it in a trigger, along with a trigger to delete SearchDoc records when HelpfulLink records are deleted.

/* app/sql/helpful_link_search_doc_triggers.sql */

CREATE OR REPLACE FUNCTION upsert_helpful_link_search_doc_on_change() RETURNS TRIGGER AS $$
BEGIN
  perform upsert_helpful_link_search_doc(new.id);

  RETURN NULL;
END;
$$ language plpgsql;

CREATE OR REPLACE FUNCTION delete_helpful_link_search_doc_on_delete() RETURNS TRIGGER AS $$
BEGIN
  DELETE FROM search_docs WHERE searchable_id = old.id AND searchable_type = 'HelpfulLink';

  RETURN NULL;
END;
$$ language plpgsql;

DROP TRIGGER IF EXISTS upsert_helpful_link_search_doc_on_change ON helpful_links;
CREATE TRIGGER upsert_helpful_link_search_doc_on_change AFTER INSERT OR UPDATE ON helpful_links
  FOR EACH ROW EXECUTE PROCEDURE upsert_helpful_link_search_doc_on_change();

DROP TRIGGER IF EXISTS delete_helpful_link_search_doc_on_delete ON helpful_links;
CREATE TRIGGER delete_helpful_link_search_doc_on_delete AFTER DELETE ON helpful_links
  FOR EACH ROW EXECUTE PROCEDURE delete_helpful_link_search_doc_on_delete();

Now we create a migration to install the triggers:

# db/migrate/20220316214532_install_helpful_link_search_doc_triggers.rb

class InstallHelpfulLinkSearchDocTriggers < ActiveRecord::Migration[7.0]
  def up
    execute Rails.root.join('app/sql/helpful_link_search_doc_triggers.sql').read
  end

  def down
  end
end

Run the migration and confirm everything works as expected.

$ rails db:migrate

== 20220316214532 InstallHelpfulLinkSearchDocTriggers: migrating ==============
== 20220316214532 InstallHelpfulLinkSearchDocTriggers: migrated (0.0051s) =====

Finally, we’ll verify that the triggers are working. First we create a new HelpfulLink and check that the SearchDoc is there as expected.

$ rails c

Loading development environment (Rails 7.0.2)

irb(main):001:1* helpful_link = HelpfulLink.create(
irb(main):002:1*   title: 'Treat yourself and your fluffy friend',
irb(main):003:1*   description: 'Cozy blankets for humans and felines alike',
irb(main):004:1*   url: 'https://cat-blankets.example.com'
irb(main):005:0> )

  TRANSACTION (0.1ms)  BEGIN
  HelpfulLink Create (5.0ms)  INSERT INTO "helpful_links" ("title", "description", "url", "created_at", "updated_at") VALUES ($1, $2, $3, $4, $5) RETURNING "id"  [["title", "Treat yourself and your fluffy friend"], ["description", "Cozy blankets for humans and felines alike"], ["url", "https://cat-blankets.example.com"], ["created_at", "2022-03-16 22:03:20.319339"], ["updated_at", "2022-03-16 22:03:20.319339"]]
  TRANSACTION (1.6ms)  COMMIT
=>
#<HelpfulLink:0x000000010804c2f8
 id: 4,
 title: "Treat yourself and your fluffy friend",
 description: "Cozy blankets for humans and felines alike",
 url: "https://cat-blankets.example.com",
 created_at: Wed, 16 Mar 2022 22:03:20.319339000 UTC +00:00,
 updated_at: Wed, 16 Mar 2022 22:03:20.319339000 UTC +00:00>

irb(main):006:0> search_doc = SearchDoc.find_by(searchable_id: helpful_link.id, searchable_type: 'HelpfulLink')

  SearchDoc Load (1.8ms)  SELECT search_docs.* FROM "search_docs" ORDER BY "search_docs"."id" DESC LIMIT $1  [["LIMIT", 1]]
=>
#<SearchDoc:0x000000011a6c8458
 id: 4,
 title: "Treat yourself and your fluffy friend",
 body: "Cozy blankets for humans and felines alike",
 tsv: "'alik':13 'blanket':8 'cozi':7 'felin':12 'fluffi':5A 'friend':6A 'human':10 'treat':1A",
 searchable_type: "HelpfulLink",
 searchable_id: 4,
 created_at: Wed, 16 Mar 2022 22:03:20.319728000 UTC +00:00,
 updated_at: Wed, 16 Mar 2022 22:03:20.319728000 UTC +00:00>

Looks good! Let’s update the HelpfulLink and make sure its SearchDoc changes too.

irb(main):007:0> HelpfulLink.last.update(title: 'Cat blankets are the hot new trend')

  HelpfulLink Load (0.3ms)  SELECT "helpful_links".* FROM "helpful_links" ORDER BY "helpful_links"."id" DESC LIMIT $1  [["LIMIT", 1]]
  TRANSACTION (0.1ms)  BEGIN
  HelpfulLink Update (0.5ms)  UPDATE "helpful_links" SET "title" = $1, "updated_at" = $2 WHERE "helpful_links"."id" = $3  [["title", "Cat blankets are the hot new trend"], ["updated_at", "2022-03-16 22:05:54.071998"], ["id", 4]]
  TRANSACTION (0.7ms)  COMMIT
=> true

irb(main):008:0> search_doc.reload

  SearchDoc Load (1.2ms)  SELECT search_docs.* FROM "search_docs" WHERE "search_docs"."id" = $1 LIMIT $2  [["id", 4], ["LIMIT", 1]]
=>
#<SearchDoc:0x000000011a1add38
 id: 4,
 title: "Cat blankets are the hot new trend",
 body: "Cozy blankets for humans and felines alike",
 tsv: "'alik':14 'blanket':2A,9 'cat':1A 'cozi':8 'felin':13 'hot':5A 'human':11 'new':6A 'trend':7A",
 searchable_type: "HelpfulLink",
 searchable_id: 4,
 created_at: Wed, 16 Mar 2022 22:03:20.319728000 UTC +00:00,
 updated_at: Wed, 16 Mar 2022 22:03:20.319728000 UTC +00:00>

And to top it all off, let’s destroy the HelpfulLink and confirm that the SearchDoc is destroyed along with it.

irb(main):009:0> HelpfulLink.last.destroy!

  HelpfulLink Load (0.9ms)  SELECT "helpful_links".* FROM "helpful_links" ORDER BY "helpful_links"."id" DESC LIMIT $1  [["LIMIT", 1]]
  TRANSACTION (0.2ms)  BEGIN
  HelpfulLink Destroy (1.3ms)  DELETE FROM "helpful_links" WHERE "helpful_links"."id" = $1  [["id", 4]]
  TRANSACTION (0.9ms)  COMMIT
=>
#<HelpfulLink:0x000000011a34f970
 id: 4,
 title: "Cat blankets are the hot new trend",
 description: "Cozy blankets for humans and felines alike",
 url: "https://cat-blankets.example.com",
 created_at: Wed, 16 Mar 2022 22:03:20.319339000 UTC +00:00,
 updated_at: Wed, 16 Mar 2022 22:05:54.071998000 UTC +00:00>

irb(main):010:0> search_doc.reload

  SearchDoc Load (1.1ms)  SELECT search_docs.* FROM "search_docs" WHERE "search_docs"."id" = $1 LIMIT $2  [["id", 4], ["LIMIT", 1]]

Traceback (most recent call last):
/usr/local/var/rbenv/versions/2.7.5/lib/ruby/gems/2.7.0/gems/activerecord-7.0.2/lib/active_record/relation/finder_methods.rb:381:in `raise_record_not_found_exception!':
Couldn't find SearchDoc with 'id'=4 (ActiveRecord::RecordNotFound)

Good stuff! Our triggers all work as expected.

Conclusion

We’ve just scratched the surface of PostgreSQL full-text search in the context of a Rails app, but we already have a useful feature for our users. I hope you’ve enjoyed this whirlwind tour!

To find out more about the features discussed in this post, you can dig into the official documentation for PostgreSQL full-text search.

A huge thanks to colleagues Justin Giancola and Christopher Milne, my fellow devs on the integrated search feature that inspired this post.

Fun with Multiversion Concurrency Control - Part 2

2020-05-15T16:30:00+00:00

Hello again and welcome back! When we last left off, we had just given up on trying to find a “simple” solution to the following problem:

The problem, restated

How can we read events from a table as they are being written without skipping records that haven’t committed at read time?

To provide a concrete example, we might run the following query:

SELECT id FROM events ORDER BY id WHERE id > 184

and get back 185, 186, 189 even though events 187 and 188 would show up in the results if we were to run this query sometime later. Why? Because it’s possible that at the time of our first read, the transaction that inserts 187 and 188 has not yet committed.

We can’t avoid this issue by looking at each id and waiting for id + 1 to show up before proceeding because the ids are generated by a Postgres sequence, and it is possible that there are gaps that will never be filled.

If this problem still seems unclear, please go back and read the first post.

What we want

Ideally, we would have a cleaner sequence of event ids. In particular, we want each id to be 1 greater than the previous id, with no gaps. This would permit us to read sequentially without skipping anything. As discussed in the previous post, the table we happen to be reading from is managed by a third party library, and we can’t dramatically modify how records are written to it.

But what if we could generate a table of surrogate event ids, arranged in the order that the event records were actually committed? And what if we could write these surrogates in such a way that we left no gaps in their id sequence and didn’t cause egregious performance problems? If this were possible, we could reliably process events by looping over the surrogate table to determine which events we still needed to process. This would permit us to process the events roughly in order, and ensure that none are skipped.

An events stream

Before I go into this solution, I want to credit its source. Commanded is an awesome CQRS/ES framework written in Elixir. It includes an event store implementation which is a veritable treasure trove of Postgres tricks. The following is ~~directly lifted~~ adapted from how that event store generates stream records¹.

We will create a table called events_stream. This table will consist of two columns, id and event_id. The id column will be an integer representing event insert/commit order, and the event_id column will be a foreign key to the corresponding event record. While the id column will be a primary key, it will not be associated with a Postgres sequence because, as discussed in the previous post, sequences can have gaps. Instead, we will generate them using a different mechanism.

We will create a second table stream_ids containing a single column, id, which is again an integer primary key not associated with a Postgres sequence. We will only write a single row to this table which will act as the id counter for the events_stream table. Each time we want to insert records into events_stream, we will increment the id of the single record in the stream_ids table by the number of records we are inserting into events_stream. We will do this within the same transaction that inserts the events_stream records.

To give a concerete example, let’s say events 185 and 186 have just been inserted. We would run the following query to record them in the events_stream table:

WITH stream AS (
  UPDATE stream_ids SET id = id + 2
  RETURNING id - 2 AS original_id
),
events (event_id, index) AS (
  VALUES (185, 1), (186, 2)
)
INSERT INTO events_stream (id, event_id)
  SELECT
    stream.original_id + events.index,
    events.event_id
  FROM events, stream;

The stream common table expression in the above query contains an UPDATE which increments the id of our single stream_ids record by the number of event_stream records we are inserting, and returns the original value of the id before it was incremented. The ids of the events_stream records we are inserting are generated relative to this original value. So the first events_stream record is inserted with original id + 1, the second original id + 2, and so on.

The beauty of this query is the relative id incrementing. Even if we are inserting many events_stream records concurrently, we will never get a duplicate or a gap. It forces the sequence of event_stream ids to have just the properties we want.

Some details I’ve glossed over

Okay, this is great, but when does the query to stream the events run? In Commanded, it’s run within the transaction that writes the events in the first place. There, the context is a little bit different because the streams are a first-class resource rather than something which was tacked on later.

Another good place to run it would be inside of a trigger that fires on inserts to the events table. This works well provided that your event inserts are fairly fast because concurrent inserts can now force transactions writing events in other processes to be held open until they complete².

If you have some event writes that take a long time and you don’t want these to interfere with the processing of other events, a third possibility is to use a queue. Here, you use a trigger on event insert to add the newly inserted event id to a queue table. You then have a separate process that dequeues and streams items from this table in a single transaction.

Back to the original problem

Now, we can finally fix the code that caused all of this trouble in the first place. We upgrade the original code to loop over the event_stream records to determine which events to process:

  sql = "SELECT id, event_id FROM events_stream WHERE id > $1 ORDER BY ID"
  streamed_events = fetch_records(sql, last_seen)

  for streamed_event in streamed_events
    sql = "SELECT * FROM events WHERE id = $?"
    event = fetch_record(sql, streamed_event.event_id)
    process(event)
    last_seen = streamed_event.id
  end
end

Conclusion

This approach ended up being somewhat more involved than we were expecting when we ran into the issue in the first place. On the bright side, we were able to implement it without having to make any changes to the 3rd party library that manages the events table, and it has been quite stable over the last few months in production.

And if you’re curious, you can read the implementation right here ↩
The reason for this is that when using the default READ COMMITTED transaction isolation level, an UPDATE will wait for any concurrent transactions open on its target row to commit or rollback before it proceeds. See here for more details. Hat tip again to @drteeth for digging deep enough to understand what was going on with concurrent queries of this type. ↩

Speedy dev environments with Nix

2020-05-03T14:00:00+00:00

The problem

Our stack at PN is composed by one Rails monolith and a plethora of EmberJS frontend apps.

We also use PostgreSQL, Redis and NGINX.

This means that in order to run our development stack each engineer has to at a minimum run one Rails server, two databases and a web server, usually they will also be running at least one frontend application and possibly a REPL to interact with the Rails server and maybe some tests.

The commonly used solution

In the “good old days” developers used to run everything locally on their work laptop, this was done using a bunch of tools:

rbenv - managing Ruby version
nvm - managing NodeJS versions
homebrew - managing system packages (like PostgreSQL, C libraries, Redis)

But running everything locally has several problems:

It required each dev to manage their own environment and dependencies
It is very hard to have different versions of certain dependencies (i.e database) installed at the same time (one may work on multiple projects or want to experiment with a newer version of some dependency)
A system update can easily break the dev environment, causing lost productivity

Docker saves the day (or not?)

In recent years the dev community moved to containerized solutions.

‘Containerization’ usually means docker, although alternatives do exist.

handwaving ahead - The idea behind Docker is to use the “host” system kernel but package applications and dependencies into a single “blob”. This blob can be pushed to the cloud and downloaded for later use. This guarantees that when the application is the deployed no dependencies have to be installed on the remote system because they are all contained within the “docker image” that will be run.

Although initially intended to make deployments easier, the dev community rallied around Docker and extended its use case beyond deployments to include local development.

Docker Compose is typically used to orchestrate the various containers needed for local development. A docker-compose.yml configuration is defined, which includes all services as well as the main application.

To run a Ruby On Rails app similar to ours you would need a docker-compose file that configures a PostgreSQL server, a Redis server, an NGINX server and a Linux image with all the dependencies for the Rails app. Here’s one of the many blogposts on how to do this.

The only real difference between this setup and the image that gets deployed to production is that the development image is usually configured to “mount” the code directory from the host system, allowing developers to edit their code locally and have it reload within the docker image.

Issues with Docker

The main issue we found with using Docker locally is that docker filesharing is extremely slow, especially on MacOS. The interwebs have plenty of resources to address this problem but these approaches simply mitigate rather than resolve the underlying performance issue.

Docker performance is pretty bad for Rails development but it’s even worse for front-end apps that require a gazillion files to be loaded and written (cough cough webpack). Poor Docker performance usually leads developers to give up on Docker for their frontend - and return to a painfully slow backend development process.

Nix-shells - A better way?

Surely there must be a way to use docker for what it is good at (running services like databases) and have a way to manage dev dependencies without having to manually install them like in the “good old days”.

Enter Nix. Nix is “The Purely Functional Package Manager”, you can imagine it as an alternative to Homebrew or apt, yum, etc.

Nix works both on MacOS and Linux and allows userspace installation of packages.

But for our use case the best part of nix is nix-shell.

A nix-shell is a bash console that is loaded starting from the host terminal but is initialized with a pre-defined set of packages which are downloaded the first time you run the shell. The packages are then instantly available for later use. Think about it as a bundle install or a npm install but for your OS dependencies. nix-shells work in isolation, this means that the dependencies available inside the shell cannot leak out to your host system. Nix achieves this by using a symlink structure and by manipulating your bash PATH. If you are curious on how this works try to issue echo $PATH when you start the example shell below.

For example

A shell.nix file with the following contents

let
  basePackages = [
    ruby
  ];

  hooks = ''
    mkdir -p .nix-gems
    export GEM_HOME=$PWD/.nix-gems
    export GEM_PATH=$GEM_HOME
    export PATH=$GEM_HOME/bin:$PATH
    export PATH=$PWD/bin:$PATH
  '';

in
  pkgs.stdenv.mkDerivation {
    name = "your-shell-name";
    buildInputs = basePackages;
    shellHook = hooks;
    hardeningDisable = [ "all" ];
  }

can be invoked by simply running nix-shell in the current directory, you will be moved to a new bash shell that has Ruby installed for you!

An important thing to note is that a nix-shell is just another bash shell, there is no virtualization happening, the only difference is that the nix-shell has access to more dependencies that come from the shell configuration.

The consequence of this is that the shell is not like a docker container and will not run services for you, services are still system level processes.

A small note about packaged dependencies

Sometimes your project will require to install packages that are not available on Nix. An example of this can be ruby gems that you install with gem install or node packages installed with npm install -g.

The Nix ecosystem offers a few solutions for this problem but the shell.nix file we included above shows a simple trick that we found works well.

By setting some exports (i.e GEM_PATH) we manipulate the install paths for RubyGems so that all gems installations are local to the shell. Normally RubyGems would try to install these globally and because Ruby was installed by Nix the commands would fail.

Our solution: use Nix and Docker together

The solution we went with at PN is to take the best of docker and nix-shell and use each one where it shines.

This means using docker to run our databases and NGINX and using nix-shell to manage the dependencies and run ruby and node.

Our main Rails application then ships with

a docker-compose.yml that configures PostgreSQL, Redis, NGINX
a shell.nix that gives the user a nix-shell with the right version of Ruby, NodeJS, OpenSSL etc

The development workflow then becomes

start docker-compose
run nix-shell and from there start bundle exec rails s or bundle exec rails c or any other process you might need to run

This also works great for our EmberJS applications and allows us to avoid using nvm while retaining native performance.

Taking this further

After doing this for a few months and enjoying the greatly improved development speed we decided to take this further and build some more automation around this.

pndev was born - pndev is a command-line tool that automates our dev workflows and will likely be the subject of a future blog post.

Resources

Here are some useful resources to get you started with development in a nix-shell

Some caveats

Apple does their best to mess up 3rd party installs so installing Nix on MacOS is a bit more complicated than one would like
Nix is somewhat difficult programming language to learn but writing nix-shells is fairly easy

Fun with Multiversion Concurrency Control

2020-03-06T14:00:00+00:00

Do you ever have one of those bugs that, when it is first discovered, appears simple to fix, but in the end forces you to accept that any proper fix will require Serious Engineering™ and in the process breaks your mind a bit? I had one of those this week.

The setup

Part of the PN coaching platform is built using the Event Sourcing design pattern. We model changes as a sequence of events, and then have multiple subscribers consuming these events. The subscribers process the events to build up aggregated statistics and various denormalized representations of domain objects which are fast to query. We are using a framework that supports this design pattern, but it has the significant limitation that all event processing happens synchronously. This has tradeoffs in terms of performance and deployment. To mitigate these tradeoffs and allow us to quickly and easily add or remove subscribers, we extended the framework to support asynchronous subscribers.

The core of our asyncronous subscriber mechanism is pretty simple. There is an events table with a sequence primary key that is getting written to by a bunch of processes. There are some other processes that are reading events out of this table as it is being written to. Each reader process looks something like this:

while true
  sql = "SELECT * FROM events WHERE id > $1 ORDER BY ID"
  events = fetch_records(sql, last_seen)

  for event in events
    process(event)
    last_seen = event.id
  end
end

If you spot the problem already, that’s great, but keep it to yourself and don’t spoil the surprise for the others.

The bug

This code was running in production for a few weeks when we started getting many errors related to event processing. At first it appeared as though one of the processors was making invalid assumptions about some event state, but it turned out the problem was worse than that. We were trying to apply an event to an invalid state because some previous events had not been processed. How could this be? Enter Multiversion Concurrency Control (MVCC¹).

MVCC breaks our simple loop

Based on our logs, the events we processed looked like this:

..., 185, 186, 189, ...

However when we later queried the events table, we saw that events 187 and 188 were present…

The missing events weren’t picked up by the processor and somehow dropped on the floor. There wasn’t an error in processing that silently failed and wasn’t retried. The missing events were temporarily invisible because the transaction that wrote them had not yet committed at the time that we fetched the events just following them in the sequence. When we queried afterwards, both transactions had committed and so the missing events were now visible. The event processor completely skipped the missing events because by the time they were visible, the code had already “seen” up to 189 and proceeded on past them.

This is how MVCC works. Even though the transaction that created events 187 and 188 started before the transaction that created 189, it committed afterwards. You can actually see evidence of this if you query xmin² along with the ids:

psql=> SELECT id, xmin FROM events WHERE id BETWEEN 185 AND 189;
 id  | xmin
-----+-------
 185 | 10673
 186 | 10676
 187 | 10679
 188 | 10679
 189 | 10680

The simple fix that doesn’t work

Okay, “no problem!”, we think. The code just needs to account for this out-of-order commit situation. We’ll just watch out for it by verifying that the event we are about to process is the next one in the sequence like so:

while true
  sql = "SELECT * FROM events WHERE id > $1 ORDER BY ID"
  events = fetch_records(sql, last_seen)

  for event in events
    if event.id != last_seen + 1
      break
    end

    process(event)
    last_seen = event.id
  end
end

Not so fast. The problem is that sometimes event last_seen + 1 will never show up because there are gaps in the sequence that will never be filled.³ This is the case even though for this particular table, records are only ever created. There are never any updates or deletes. Why? Sequences in Postgres (and sequential primary key creation mechansims in other RDBMSs) only guarantee that the numbers provided are always increasing, not that the set of values they produce are free from gaps.

Maybe we can fix the sequence?

Why not implement some mechanism on top of event creation that generates a strictly sequential series of event ids? There are a few ways to do this, but all of the simple approaches use pessimistic locks in one form or another. Using pessimistic locks for this purpose destroys concurrency, drags down performance, and introduces the possibility of deadlocks.

There are optimistic locking strategies you can use as well.⁴ However, they are more involved and it helps when your implementation language has good concurrency primitives.

Maybe we can use Postgres magic?

Despite the looming feeling we are grasping at straws, we vaguely recall past instances where esoteric Postgres features and nontrival compromises to our data abstraction layer have saved us.

After poring over countless Postgres mailing list posts⁵ we learn about the commit timestamp feature that was introduced in 9.5. This is a feature which has Postgres track the commit time of every transaction and make it available via the function pg_xact_commit_timestamp(xmin). This is fantastic! All we have to do is:

enable this config parameter on every Postgres instance on production, staging, and everyone’s dev machine
restart every Postgres server in every environment
work around the fact that we don’t have the data available from before enabling the feature
figure out a way to avoid losing commit timestamps due to 32 bit xid wraparound
work around the fact that you can’t index the results of pg_xact_commit_timestamp(xmin) and we are sorting on this value in a table with tens (and eventually hundreds) of millions of records
deal with the fact that commit timestamps aren’t unique so we have to some bookkeeping to record both the last seen timestamps as well as event ids…which are no longer received in order…

Giving up on simplicity

At this point we⁶ have found many simple solutions that don’t work, learned a lot about Postgres internals, and are still stuck with the same problem.

While I would like to end the post by revealing the simple solution that does work, the unfortunate reality is that we had to introduce significant complexity in order to address this problem.

On the bright side, the solution works really well and enabled us to learn about even more things that make Postgres great.

We’ll talk about it in our next post.

Multiversion Concurrency Control is a mechanism that many relational database systems use to enhance performance while preventing clients from ever seeing data in an inconsistent state. This is accomplished by keeping multiple snapshots of data around. Whenever a client queries, they will see some copy of the data that provides a consistent view. This view will exclude pending transactions that have yet to be committed, unless you specify otherwise by locking for read. ↩
the system colum that stores the creation xid ↩
in this particular table, about 0.6% of primary keys are missing. The number will be higher for tables where there is a lot of contention on record creation. ↩
an awesome one can be found here and here ↩
and if we are being honest, StackOverflow ↩
special thanks to drteeth for fighting this fight with me ↩

Welcome to the PN tech blog!

2020-01-28T21:33:59+00:00

Hello, world!

At PN, we have a conversation every year that goes something like this:

“Hey, we should really make a tech blog” “Yeah, that would sure be great”

And then nothing happens for another year. So rather than have that same conversation again, here is the first, shitty version, of our tech blog.

The plan is to make the barrier low enough to encourage our devs to post, and worry about making it actually good when we actually have content.

Let’s see how it goes!

PN Tech

Implementing PostgreSQL full-text search in Rails

Introduction

The walkthrough

Step 0: Setup

Step 1: Create the SearchDoc model

Step 2: rake task to add some SearchDocs for HelpfulLink records

Step 3: Implement the basic GET /search endpoint

Step 4: Add highlighting

Step 5: Add sorting by rank

Step 6: Add a SQL trigger to create/update/delete SearchDoc records when HelpfulLink records change

Conclusion

Fun with Multiversion Concurrency Control - Part 2

The problem, restated

What we want

An events stream

Some details I’ve glossed over

Back to the original problem

Conclusion

Speedy dev environments with Nix

The problem

The commonly used solution

Docker saves the day (or not?)

Issues with Docker

Nix-shells - A better way?

A small note about packaged dependencies

Our solution: use Nix and Docker together

Taking this further

Resources

Some caveats

Fun with Multiversion Concurrency Control

The setup

The bug

MVCC breaks our simple loop

The simple fix that doesn’t work

Maybe we can fix the sequence?

Maybe we can use Postgres magic?

Giving up on simplicity

Welcome to the PN tech blog!

Step 1: Create the `SearchDoc` model

Step 2: `rake` task to add some `SearchDocs` for `HelpfulLink` records

Step 3: Implement the basic `GET /search` endpoint

Step 5: Add sorting by `rank`

Step 6: Add a SQL trigger to create/update/delete `SearchDoc` records when `HelpfulLink` records change