pganalyze Blog - Articles by Leigh Halliday

PostGIS vs. Geocoder in Rails

Leigh Halliday — Thu, 01 Oct 2020 12:00:00 GMT

This article sets out to compare PostGIS in Rails with Geocoder and to highlight a couple of the areas where you'll want to (or need to) reach for one over the other. I will also present some of the terminology and libraries that I found along the way of working on this project and article as I set out to understand PostGIS better and how it is integrated with Rails.

If you are interested in learning how to work with geospatial data with PostGIS in Django I recommend having a look at our blog post Using GeoDjango and PostGIS in Django here.

Installing PostGIS
ActiveRecord PostGIS Adapter
Our Example Data
Building a Geo Helper Class with PostGIS
Finding Nearby Records with PostGIS and Geocoder
Finding Records Within a Bounding Box with PostGIS and Geocoder
Finding Records Within a Polygon with PostGIS and Geocoder
Finding Nearby Related Records with PostGIS and Geocoder
Conclusion
About the Author

Picture via Annie Spratt on Unsplash

I have built a number of Rails applications over the years that show locations on a map, have nearby search functionality, and I had never used PostGIS before! How was this possible? The reason is that there is a Ruby gem named Geocoder which enables you to do these sorts of queries, and it's quite efficient! That said, there is a reason that PostGIS exists. For more complex geo queries I’d recommend reaching beyond Geocoder to PostGIS.

As an example, if you wanted to find homes which have a school within 1km of them, or if you wanted to draw an oddly shaped polygon on a map and search within it, this is the world where PostGIS shines and makes these complex geo queries possible.

In this article we will be covering:

PostGIS in Rails setup
Finding nearby records (Geocoder + PostGIS)
Finding records within a bounding box (Geocoder + PostGIS)
Finding records within a polygon (PostGIS)
Finding nearby related records (PostGIS)

The source code referenced in this article can be found here.

Installing PostGIS

Postgres comes with a number of built-in extensions that you can enable, but unfortunately PostGIS (Spatial and Geographic objects for Postgres) isn't one of them. In order to enable this extension, you will have to use a Postgres install with PostGIS support. I recommend using the official postgis docker image, but luckily many hosted Postgres solutions come with PostGIS already available. If you are not sure, you can query the available extensions with the following query:

select *
from pg_available_extensions
where name like '%postgis%'

If you'd like to see if the extension is already enabled, you can run this query:

select * from pg_extension

And finally, to enable this extension, you can use the command create extension postgis, but since we're working within Rails, there is a Gem that will take care of this step for us as we'll see below.

ActiveRecord PostGIS Adapter

If you have confirmed that your version of Postgres supports the postgis extension, you're ready to integrate it with your Rails application. This can be done by using the activerecord-postgis-adapter gem. Two things need to be done to get up and running. The first is to update the adapter within config/database.yml to be set to postgis. Next, if this is a new application, you can run rails db:create as normal, but if it is an existing one, you'll have to run the command rake db:gis:setup. This command is enabling the postgis extension in your database.

Our Example Data

We'll be working with sample data for a realtor website that allows us to find homes in a variety of ways, including homes that are nearby a local school. There are two models: homes and schools. The Rails migration to create these tables is below:

class CreateHomes < ActiveRecord::Migration[6.0]
  def change
    create_table :homes do |t|
      t.string :name, null: false
      t.string :status, null: false
      t.bigint :price, null: false
      t.integer :beds, null: false, default: 0
      t.integer :baths, null: false, default: 0
      t.st_point :coords, null: false, geographic: true
      t.float :longitude, null: false
      t.float :latitude, null: false
      t.timestamps

      t.index :coords, using: :gist
      t.index %i[latitude longitude]
      t.index :status
      t.index :price
    end

    create_table :schools do |t|
      t.st_point :coords, null: false, geographic: true

      t.index :coords, using: :gist
      t.timestamps
    end
  end
end

By using activerecord-postgis-adapter we are able to define PostGIS columns within our migration file. When working with PostGIS you can store a point (latitude + longitude) as a single column of type ts_point, whereas when working with Geocoder the latitude and longitude are stored as floats in separate columns. Because we are comparing the two approaches, we will store the data both ways, but typically you would choose one approach or the other.

PostGIS geographic columns can be indexed using GiST style indexes. GiST indexes are required over B-Tree indexes when working with geographic data because coordinates cannot be easily sorted along a single axis (such as numbers, letters, dates, etc...) in a way that would allow the database to speed up common geographic operations.

The example project for this article contains a seeds file (run with rake db:seed) which will generate 100k homes and 100 schools in and around the Atlanta, Georgia area.

Building a Geo Helper Class with PostGIS

The Rails PostGIS adapter is based on a library named RGeo, which while incredibly powerful, I found a little bit confusing due to a lack of documentation. I ended up building a small helper class to generate different geo objects for me. The first thing to point out is what SRID is. Just like the imperial and metric systems are used to measure and weigh amounts using an agreed upon reference point, coordinates also need a coordinate reference system to ensure that the latitude and longitude that one uses means the same thing to different people when referring to a single place on earth. 4326 is the spatial system used for GPS satellite navigation systems and the one we will be using within this article.

One last thing to define is what WKT is. Well-known Text representation of geometry is a string representation of a point, line string, and polygon (among other things) that we will be using in our examples in this article. This is the format Postgres (PostGIS) receives and displays geographic data types in.

class Geo
  SRID = 4326

  def self.factory
    @@factory ||= RGeo::Geographic.spherical_factory(srid: SRID)
  end

  def self.pairs_to_points(pairs)
    pairs.map { |pair| point(pair[0], pair[1]) }
  end

  def self.point(longitude, latitude)
    factory.point(longitude, latitude)
  end

  def self.line_string(points)
    factory.line_string(points)
  end

  def self.polygon(points)
    line = line_string(points)
    factory.polygon(line)
  end

  def self.to_wkt(feature)
    "srid=#{SRID};#{feature}"
  end
end

Finding Nearby Records with PostGIS and Geocoder

One of the most common geo queries used in applications is to find all records within X distance from a known point (the user's location, an event, a search, etc...). Because we installed Geocoder and added reverse_geocoded_by :latitude, :longitude to our Home class, we can use the nearby method to find all homes within 5km of this latitude and longitude (which happens to be Atlanta, Georgia). Geocoder likes to have arrays with latitude and then longitude, as opposed to PostGIS which prefers the exact opposite order!

Home.near([33.753746, -84.386330], 5).count(:all) # ~5ms

This query ran in about 5ms on my computer (searching through 100k records)... pretty fast! The reason it is fast is because we added an index on the latitude and longitude fields, but also because Geocoder applies a bounding box filter which utilises the index. Remember the Spatial Reference System (SRID) that we mentioned above? Because our coordinates do not take place on a Cartesian plane, we can’t use a standard distance formula to calculate the distance between two points. Although we won’t venture further into the math of this query below, it takes into consideration the Earth’s spherical nature when calculating the distance between two coordinates as specified by latitude and longitude. This article dives into more detail on these calculations if you are interested.

SELECT COUNT(*) FROM "homes" WHERE (homes.latitude BETWEEN 33.708779919704064 AND 33.798712080295935 AND homes.longitude BETWEEN -84.44041260768655 AND -84.33224739231345 AND (6371.0 * 2 * ASIN(SQRT(POWER(SIN((33.753746 - homes.latitude) * PI() / 180 / 2), 2) + COS(33.753746 * PI() / 180) * COS(homes.latitude * PI() / 180) * POWER(SIN((-84.38633 - homes.longitude) * PI() / 180 / 2), 2)))) BETWEEN 0.0 AND 5)

We'll have to build our own near query when working with PostGIS, but don't worry, it's pretty straight forward! The g_near method lives within the Home model, and takes advantage of the ST_DWithin function provided by PostGIS. Remember that we have to convert our point into the correct WKT format so that PostGIS understands the data we are passing it.

def self.g_near(point, distance)
  where(
    'ST_DWithin(coords, :point, :distance)',
    { point: Geo.to_wkt(point), distance: distance * 1000 } # wants meters not kms
  )
end

Home.g_near(Geo.point(-84.386330, 33.753746), 5).count # ~5ms

This query performs just about as fast as the Geocoder version (because of our GiST index on the coords column), but is definitely a little easier on the eyes to read.

SELECT COUNT(*) FROM "homes" WHERE (ST_DWithin(coords, 'srid=4326;POINT (-84.38633 33.753746)', 5000))

Finding Records Within a Bounding Box with PostGIS and Geocoder

Geocoder provides us a way to find all records within a bounding box (roughly a rectangle, ignoring projection onto a sphere), and we just have to pass it the bottom left (south west) and top right (north east) coordinates.

Home.within_bounding_box(
  [33.7250057553, -84.4224209302],
  [33.774350796, -84.3570139222]
).count # ~5ms

Because it can use the index on latitude and longitude, it is quite efficient.

SELECT COUNT(*) FROM "homes" WHERE (homes.latitude BETWEEN 33.7250057553 AND 33.774350796 AND homes.longitude BETWEEN -84.4224209302 AND -84.3570139222)

To perform a bounding box query using PostGis, we'll create a method named g_within_box inside of the Home model, and utilize a PostGIS function named ST_MakeEnvelope along with the && operator.

def self.g_within_box(sw_point, ne_point)
  where(
    "coords && ST_MakeEnvelope(:sw_lon, :sw_lat, :ne_lon, :ne_lat, #{
      Geo::SRID
    })",
    {
      sw_lon: sw_point.longitude,
      sw_lat: sw_point.latitude,
      ne_lon: ne_point.longitude,
      ne_lat: ne_point.latitude
    }
  )
end

Home.g_within_box(
  Geo.point(-84.4224209302, 33.7250057553),
  Geo.point(-84.3570139222, 33.774350796)
).count # ~5ms

Again, this version performs at about the same efficiency as the Geocoder version.

SELECT COUNT(*) FROM "homes" WHERE (coords && ST_MakeEnvelope(-84.4224209302, 33.7250057553, -84.3570139222, 33.774350796, 4326))

Finding Records Within a Polygon with PostGIS and Geocoder

We're now into territory that requires PostGIS. To find records inside of a polygon, along with the help of our Geo class helper and the ST_Covers function from PostGIS, we can create a method named g_within_polygon in our Home model. This polygon is a triangle, where the last point is the same as the first one, thereby "closing" the shape of the polygon.

def self.g_within_polygon(points)
  polygon = Geo.polygon(points)
  where('ST_Covers(:polygon, coords)', polygon: Geo.to_wkt(polygon))
end

Home.g_within_polygon(
  Geo.pairs_to_points(
    [
      [-84.39731626974567, 33.75570358345219],
      [-84.33139830099567, 33.86524376001825],
      [-84.25243406759724, 33.770545357734925],
      [-84.39731626974567, 33.75570358345219]
    ]
  )
).count # ~5ms

This query remains efficient due to the use of our GiST index, searching through 100k records in about 5ms.

SELECT COUNT(*) FROM "homes" WHERE (ST_Covers('srid=4326;POLYGON ((-84.39731626974567 33.75570358345219, -84.33139830099567 33.86524376001825, -84.25243406759724 33.770545357734925, -84.39731626974567 33.75570358345219))', coords))

Using PostGIS it is also possible to find related nearby records. What do I mean by that? Let's try to find available homes that are within 1km of a school. This can be done by joining to the schools table and utilizing ST_DWithin for the on clause. Starting with the SQL we'd like to produce:

SELECT
  count(DISTINCT homes.id)
FROM
  homes
  INNER JOIN schools ON ST_DWithin (homes.coords, schools.coords, 1000)
WHERE
  homes.status = 'available'

Within the Home model of our Rails application, we can create two scopes that allow us to find these homes. We're able to join 100k homes to the schools table (100 schools) based on their proximity in approximately 16ms.

class Home < ApplicationRecord
  scope :available, -> { where(status: 'available') }
  scope :near_school,
        lambda {
          select('DISTINCT ON (homes.id) homes.*').joins(
            'INNER JOIN schools ON ST_DWithin (homes.coords, schools.coords, 1000)'
          )
        }
end
# Example using the scopes declared above
Home.available.near_school.count('distinct homes.id') # 16ms

Conclusion

We've only scratched the surface of what you can do with PostGIS, yet we were able to cover a ton of functionality that is common among websites that allow you to filter results based on their location. That said, if PostGIS isn't available as an extension on your version of Postgres, or you aren't requiring the power that PostGIS provides, Geocoder offers you a great alternative.

Share this article: If you liked this article you might want to tweet it to your peers.

About the Author

Leigh Halliday is a guest author for the pganalyze blog. He is a developer based out of Canada who works at FlipGive as a full-stack developer. He writes about Ruby and React on his blog and publishes React tutorials on YouTube.

]]>

Advanced Active Record: Using Subqueries in Rails

Leigh Halliday — Wed, 24 Jun 2020 12:00:00 GMT

Active Record provides a great balance between the ability to perform simple queries simply, and also the ability to access the raw SQL sometimes required to get our jobs done. In this article, we will see a number of real-life examples of business needs that may arise at our jobs.

They will come in the form of a request for data from someone else at the company, where we will first translate the request into SQL, and then into the Rails code necessary to find those records. We will be covering five different types of subqueries to help us find the requested data.

Working with Active Record in Rails
What are Subqueries in Rails
An Overview of our Data
The Where Subquery
- Where Not Exists
The Select Subquery
The From Subquery
The Having Subquery
Conclusion
You might also be interested in
About the author

Let's take a look at why subqueries matter:

In the first case, without subqueries, we are going to the database twice: First to get the average salary, and then again to get the result set. With a subquery, we can avoid the extra roundtrip, getting the result directly with a single query.

Working with Active Record in Rails

Active Record is a little like a walled garden. It protects us as developers (and our users) from the harsh realities of what lies beyond those walls: Differences in SQL between databases (MySQL, Postgres, SQLite), knowing how to properly escape strings to avoid SQL injection attacks, and generally providing an elegant abstraction to interact with our database using the language of our choice, Ruby.

But, SQL is extremely powerful! By understanding the SQL that Active Record is executing, we can open the gate in our walled garden to reach beyond what you may think is possible to accomplish in Rails, taking advantage of optimizations and flexibility that may be difficult to achieve otherwise.

What are Subqueries in Rails

In this article, we will be learning how to use subqueries in Active Record. Subqueries are what their name implies: A query within a query. We will look at how to embed subqueries into the SELECT, FROM, WHERE, and HAVING clauses of SQL, to meet the demands of our business counterparts who are asking to view data in different and interesting ways.

We'll be playing the role of a developer fielding questions from HR. They are asking for reports about our employees at BCE (Best Company Ever), and we'll do our best to find the data they need using Active Record.

The source code for this article is available on GitHub.

An Overview of our Data

Our database has 4 tables:

roles: The job roles of our employees (Finance, Engineering, Sales, HR, etc...)
employees: The people that work for BCE
performance_reviews: Performance reviews carried out by an employee's manager, giving them a score between 0 and 100
vacations: Keeping track of when employees have taken vacation

Using https://dbdiagram.io/ we're able to see how these tables relate to each other:

If you are following along, the rails db:seed command will generate 1,000 employees, 1,000 vacations, and 10,000 performance reviews.

The Where Subquery

Now that we have our data set and we’re ready to go let’s help our HR team with their first request:

Leigh, could you find us all the employees that make more than the average salary at BCE?

Here we will use a subquery within the WHERE clause to find the employees that match HR's request:

SELECT *
FROM employees
WHERE
  employees.salary > (
    SELECT avg(salary)
    FROM employees)

My first attempt at replicating the query above looked like this:

Employee.where('salary > :avg', avg: Employee.average(:salary))

But what it produced was two queries: One to find the average, and a second to query employees with a salary greater than that number. Not technically wrong, but it doesn't line up with the SQL we were going for. There is also a potential performance impact of two round-trip requests to the database server, along with potential inconsistencies if a new employee making $1B/year is hired between queries one and two. Although this is unlikely in this particular scenario, it’s something to consider as a potential risk.

-- find the average
SELECT AVG("employees"."salary") FROM "employees"
-- find the employees
SELECT "employees".* FROM "employees" WHERE (salary > 99306.4)

What we shouldn’t forget about Active Record is that certain methods, such as average(:salary), actually execute the query and return a result, while other methods implement Method Chaining, allowing you to chain multiple Active Record methods together, building up more complex SQL statements prior to their execution.

Employee.where('salary > (:avg)', avg: Employee.select('avg(salary)'))

This produces the SQL we want, but note that we had to wrap the placeholder condition :avg in brackets, because the database wants subqueries wrapped in brackets as well.

Because the seed data is generated randomly, your results will vary from mine, but I am seeing 487 matching employees, getting a result that looks like this:

#<ActiveRecord::Relation [#<Employee id: 4, role_id: 5, name: "Bob Williams", salary: 127053.0, created_at: "2020-04-26 18:42:53", updated_at: "2020-04-26 18:42:53">, #<Employee id: 5, role_id: 4, name: "Bob Florez", salary: 149218.0, created_at: "2020-04-26 18:42:53", updated_at: "2020-04-26 18:42:53">, ...]>

Where Not Exists

Leigh, we would like to encourage employees to have a healthy work-life balance, and were hoping you could provide us with a list of all the employees who have yet to take any vacation time.

For this case, NOT EXISTS is a perfect fit, since it only matches records that do not have a match in the subquery. An alternative is to perform a left outer join, only choosing the records with no matches on the right side. This is referred to as an anti-join, where the purpose of the join is to find records that do not have a matching record.

SELECT *
FROM employees
WHERE
  NOT EXISTS (
    SELECT 1
    FROM vacations
    WHERE vacations.employee_id = employees.id)

If you're interested in the LEFT OUTER JOIN equivalent, it might look like this:

SELECT employees.*
FROM
  employees
  LEFT OUTER JOIN vacations ON vacations.employee_id = employees.id
WHERE vacations.id IS NULL

The subquery depends on a match between the employees.id column and the vacations.employee_id column, making it a correlated subquery. Because Rails follows standard naming conventions when querying (the downcased plural form of our model), we can add the above condition into our subquery without too much difficulty.

Employee.where(
  'NOT EXISTS (:vacations)',
  vacations: Vacation.select('1').where('employees.id = vacations.employee_id')
)

Using my seed data, I am seeing 369 employees that have yet to take any vacations.

#<ActiveRecord::Relation [#<Employee id: 2, role_id: 2, name: "Alice Florez", salary: 86920.0, created_at: "2020-04-26 18:42:53", updated_at: "2020-04-26 18:42:53">, #<Employee id: 5, role_id: 4, name: "Bob Florez", salary: 149218.0, created_at: "2020-04-26 18:42:53", updated_at: "2020-04-26 18:42:53">, ...]>

The Select Subquery

Leigh, could you provide us with a list of employees, including the average salary of a BCE employee, and how much this employee's salary differs from the average?

SELECT
  *,
  (SELECT avg(salary)
    FROM employees) avg_salary,
  salary - (
    SELECT avg(salary)
    FROM employees) above_avg
FROM employees

Because the subquery is repeated, we can save ourselves a little bit of hassle by placing the subquery SQL into a variable that we'll embed into the outer query. The to_sql method is perfect for this, but it's also fantastic to peak into the SQL that Rails is producing without actually executing the query.

avg_sql = Employee.select('avg(salary)').to_sql

Employee.select(
  '*',
  "(#{avg_sql}) avg_salary",
  "salary - (#{avg_sql}) avg_difference"
)

This query does not limit the results in any way, but instead selects two additional columns (avg_salary and avg_difference). Looking at the first three results, I am seeing:

[
  {"id"=>1, "role_id"=>1, "name"=>"Joe Serna", "salary"=>86340.0, "avg_salary"=>99306.4, "avg_difference"=>-12966.399999999994}, 
  {"id"=>2, "role_id"=>2, "name"=>"Alice Florez", "salary"=>86920.0, "avg_salary"=>99306.4, "avg_difference"=>-12386.399999999994}, 
  {"id"=>3, "role_id"=>3, "name"=>"Amanda Florez", "salary"=>93600.0, "avg_salary"=>99306.4, "avg_difference"=>-5706.399999999994}
]

As with any SQL query, there are often many ways to arrive at the same result. In this example we used subqueries to find the average employee salary, but it may have been better to use window functions instead. They give us the same result, but provide a simpler query which is actually more performant as well. Even on a small dataset of 1000 employees, this query takes approximately 12ms vs 18ms for the subquery equivalent.

SELECT
  *,
  avg(salary) OVER () AS avg_salary,
  salary - avg(salary) OVER () AS avg_salary
FROM
  employees

The window function approach is actually easier to write in Rails as well!

Employee.select(
  '*',
  "avg(salary) OVER () avg_salary",
  "salary - avg(salary) OVER () avg_difference"
)

The From Subquery

Leigh, we'd like to know the average performance review score given across all our managers.

After clarifying with HR, they are looking to take the average score each manager has given, and then take the average of those averages. In other words, the average average. When you are dealing with an aggregate of aggregates, it needs to be accomplished in two steps. This can be done using a subquery as the FROM clause, essentially giving us a temporary table to then select from, allowing us to find the average of those averages.

SELECT avg(avg_score) reviewer_avg
FROM (
  SELECT reviewer_id, avg(score) avg_score
  FROM performance_reviews
  GROUP BY reviewer_id) reviewer_avgs

To keep our Ruby code clean, we'll place the subquery into a variable which can then be embedded into the main query.

from_sql =
  PerformanceReview.select(:reviewer_id, 'avg(score) avg_score').group(
    :reviewer_id
  ).to_sql

PerformanceReview.select('avg(avg_score) reviewer_avg').from(
  "(#{from_sql}) as reviewer_avgs"
).take.reviewer_avg

The result of this query is 50.652. This makes sense given that the seed data used a random value between 1 and 100 (rand(1..100)).

The Having Subquery

Leigh, certain reviewers are consistently giving low performance review scores. Could you find us a list of all the managers whose average score is 25% below our company average? We need to find out what is happening.

We will start by joining the employees table to the performance_reviews table where the employee is the reviewer (a manager), and then take their average score. Then we will filter out these managers using a HAVING clause to only include those whose score increased by 25% is still lower than the company average.

SELECT
  employees.*,
  avg(score) avg_score,
  (SELECT avg(score)
    FROM performance_reviews) company_avg
FROM
  employees
  INNER JOIN performance_reviews
    ON performance_reviews.reviewer_id = employees.id
GROUP BY employees.id
HAVING
  avg(score) < 0.75 *
    (SELECT avg(score)
    FROM performance_reviews)

You'll notice that I actually included two subqueries in the above SQL. Because the SQL was saved to a variable (avg_sql), we were able to reuse this both within the SELECT portion of the query, and also within the HAVING clause.

avg_sql = PerformanceReview.select('avg(score)').to_sql

Employee.joins(:employee_reviews).select(
  'employees.*',
  'avg(score) avg_score',
  "(#{avg_sql}) company_avg"
).group('employees.id').having("avg(score) < 0.75 * (#{avg_sql})")

The result of this query gives me 103 employees, and the first three of them look like:

[
  {"id"=>173, "role_id"=>1, "name"=>"Bob Williams", "salary"=>109206.0, "avg_score"=>23.75, "company_avg"=>50.04}, 
  {"id"=>390, "role_id"=>5, "name"=>"Bob Serna", "salary"=>127559.0, "avg_score"=>26.0, "company_avg"=>50.04}, 
  {"id"=>802, "role_id"=>4, "name"=>"Alice Halliday", "salary"=>94956.0, "avg_score"=>35.88, "company_avg"=>50.04}
]

Conclusion

In this article we were able to see a number of (somewhat) real-life examples of real business needs translating first into SQL, and then into the Rails code necessary to find those records. A backend developer's career will consist in most likely hundreds of similar requests!

Active Record gives us the ability to perform simple queries simply, but also lets us access the raw SQL which is sometimes required to get our jobs done. Subqueries are a perfect example of that, and we saw how to create subqueries in Rails and Active Record in the SELECT, FROM, WHERE, and HAVING clauses of an SQL statement. As we have seen in the examples above, with the expressiveness of Active Record, one doesn’t have to resort to writing completely in SQL to use a subquery.

Share this article: If you liked this article we’d appreciate it if you’d tweet it to your peers.

About the author

]]>

Full Text Search in Milliseconds with Rails and PostgreSQL

Leigh Halliday — Thu, 16 Apr 2020 12:00:00 GMT

Imagine the following scenario: You have a database full of job titles and descriptions, and you’re trying to find the best match. Typically you’d start by using an ILIKE expression, but this requires the search phrase to be an exact match. Then you might use trigrams, allowing spelling mistakes and inexact matches based on word similarity, but this makes it difficult to search using multiple words. What you really want to use is Full Text Search, providing the benefits of ILIKE and trigrams, with the added ability to easily search through large documents using natural language.

The Foundations of Full Text Search
Implementing Postgres Full Text Search in Rails
Configuring pg_search
Optimizing Full Text Search Queries in Rails
Conclusion
You might also be interested in
About the author

To summarize, here is a quick overview of popular built-in Postgres search options:

Postgres Feature	Typical Use Case	Can be indexed?	Performance
LIKE/ILIKE	Wildcard-style search for small data	Sometimes	Unpredictable
pg_trgm	Similarity search for names, etc	Yes (GIN/GIST)	Good
Full Text Search	Natural language search	Yes (GIN/GIST)	Good

In this article, we are going to learn about the inner workings of Full Text Search in Postgres and how to easily integrate Full Text Search into your Rails application using a fantastic gem named pg_search. We will learn how to search multiple columns at once, to give one column precedence over another, and how to optimize our Full Text Search implementation, taking a single query from 130ms to 7ms.

The full source code used in this article can be found here. Instructions on how to run this application locally and how to load the sample data referenced within this article can be found in the README.

If you are interested in efficient Full Text Search in Postgres with Django, you can read our article about it.

The Foundations of Full Text Search

Let's break down the basics of Full Text Search, defining and explaining some of the most common terms you'll run into. Taking the text “looking for the right words”, we can see how Postgres stores this data internally, using the to_tsvector function:

SELECT to_tsvector('english', 'looking for the right words');
-- 'look':1 'right':4 'word':5

In the above SQL we have some text; often referred to as a document when talking about Full Text Search. A document must be parsed and converted into a special data type called a tsvector, which we did using the function to_tsvector.

The tsvector data type is comprised of lexemes. Lexemes are normalized key words which were contained in the document that will be used when searching through it. In this case we used the english language dictionary to normalize the words, breaking them down to their root. This means that words became word, and looking became look, with very common words such as for and the being removed completely, to avoid false positives.

SELECT to_tsvector('english', 'looking for the right words') @@ to_tsquery('english', 'words');
-- TRUE

The @@ operator allows us to check if a query (data type tsquery) exists within a document (data type tsvector). Much like tsvector, tsquery is also normalized prior to searching the document for matches.

SELECT
  ts_rank(
    to_tsvector('looking for the right words'),
    to_tsquery('english', 'words')
   );
-- 0.06079271

The ts_rank function takes a tsvector and a tsquery, returning a number that can be used when sorting the matching records, allowing us to sort the results from highest to lowest ranking.

Now that you have seen a few examples, let’s have a look at one last one before getting to Rails. Following, you can see an example of a query which searches through the jobs table where we are storing the title and description of each job. Here we are searching for the words ruby and rails, grabbing the 3 highest ranking results.

SELECT
  id,
  title,
  ts_rank(
    to_tsvector('english', title) || to_tsvector('english', description),
    to_tsquery('english', 'ruby & rails')
  ) AS rank
FROM jobs
WHERE
  to_tsvector('english', title) || to_tsvector('english', description) @@
  to_tsquery('english', 'ruby & rails')
ORDER BY rank DESC
LIMIT 3

The highest ranking result is a job with the title "Ruby on Rails Developer"... perfect! The full results of this query are:

[
  {
    "id": 1,
    "title": "Ruby on Rails Developer",
    "rank": 0.40266925
  },
  {
    "id": 109,
    "title": "Senior Ruby Developer - Remote",
    "rank": 0.26552397
  },
  {
    "id": 151,
    "title": "Team-Lead Developer",
    "rank": 0.14533159
  }
]

This query is actually concatenating (using ||) two tsvector fields together. This allows us to search both the title and the description at the same time. Later, we'll see how to give additional weight (precedence) to the title column.

Implementing Postgres Full Text Search in Rails

With a basic understanding of Full Text Search under our belts, it's time to take our knowledge over to Rails. We will be using the pg_search Gem, which can be used in two ways:

Multi Search: Search across multiple models and return a single array of results. Imagine having three models: Product, Brand, and Review. Using Multi Search we could search across all of them at the same time, seeing a single set of search results. This would be perfect for adding federated search functionality to your app.
Search Scope: Search within a single model, but with greater flexibility.

We will be focusing on the Search Scope approach in this article, as it lets us dive into the configuration options available when working with Full Text Search in Rails. Let's add the Gem to our Gemfile and get started:

# Gemfile
gem 'pg_search', '~> 2.3', '>= 2.3.2'

With that done, we can include a module in our Job model, and define our first searchable field:

class Job < ApplicationRecord
  include PgSearch::Model
  pg_search_scope :search_title, against: :title
end

This adds a class level method to Job, allowing us to find jobs with the following line, which automatically returns them ranked from best match to worst.

Job.search_title('Ruby on Rails')

If we were to append to_sql to the above Ruby statement, we can see the SQL that is being generated. I have to warn you, it’s a bit messy, but that is because it handles not only searching, but also putting the results in the correct order using the ts_rank function.

SELECT
  "jobs".*
FROM
  "jobs"
  INNER JOIN (
    SELECT
      "jobs"."id" AS pg_search_id,
      (ts_rank((to_tsvector('simple', coalesce("jobs"."title"::text, ''))), (to_tsquery('simple', ''' ' || 'Ruby' || ' ''') && to_tsquery('simple', ''' ' || 'on' || ' ''') && to_tsquery('simple', ''' ' || 'Rails' || ' ''')), 0)) AS rank
    FROM
      "jobs"
    WHERE ((to_tsvector('simple', coalesce("jobs"."title"::text, ''))) @@ (to_tsquery('simple', ''' ' || 'Ruby' || ' ''') && to_tsquery('simple', ''' ' || 'on' || ' ''') && to_tsquery('simple', ''' ' || 'Rails' || ' ''')))) AS pg_search_5d9a17cb70b9733aadc073 ON "jobs"."id" = pg_search_5d9a17cb70b9733aadc073.pg_search_id
ORDER BY
  pg_search_5d9a17cb70b9733aadc073.rank DESC,
  "jobs"."id" ASC

Configuring pg_search

There are a number of ways you can configure pg_search: From support for prefixes and negation, to specifying which language dictionary to use when normalizing the document, as well as adding multiple, weighted columns.

By default pg_search uses the simple dictionary, which does zero normalization, but if we wanted to normalize our document using the english dictionary, searching across both the title and description, it would look like:

class Job < ApplicationRecord
  include PgSearch::Model
  pg_search_scope :search_job,
                  against: %i[title description],
                  using: { tsearch: { dictionary: 'english' } }
end

We can perform a search in the same way we did before: Job.search_job("Ruby on Rails"). If we wanted to give higher precedence to the title column, we can add weighting scores to each of the columns, with possible values of: A, B, C, D.

class Job < ApplicationRecord
  include PgSearch::Model
  pg_search_scope :search_job,
                  against: { title: 'A', description: 'B' },
                  using: { tsearch: { dictionary: 'english' } }
end

When you start combining columns, weighting them, and choosing which dictionary provides the best results, it really comes down to trial and error. Play around with it, try some queries and see if the results you get back match with what you are expecting!

Optimizing Full Text Search Queries in Rails

We have a problem! The query that is produced by Job.search_job("Ruby on Rails") takes an astounding 130ms. That may not seem like such a large number, but it is astounding because there are only 145 records in my database. Imagine if there were thousands! The majority of time is spent in the to_tsvector function. We can verify this by running this streamlined query below, which takes almost as much time to execute as the full query which actually finds the matching jobs:

SELECT to_tsvector('english', description) FROM jobs;
-- ~130ms

SELECT description FROM jobs;
-- ~15ms

This tells me that the slowness is in re-parsing and normalizing the document into a tsvector data type every single time the query is executed. The folks at thoughtbot have a great article about Full Text Search optimizations, where they add a pre-calculated tsvector column, keeping it up-to-date with triggers. This is great because it allows us to avoid re-parsing our document for every query and also lets us index this column!

There is a similar but slightly different approach I want to cover today which I learned by reading through the Postgres documentation. It also involves adding a pre-calculated tsvector column, but is done using a stored generated column. This means we don't need any triggers! It should be noted that this approach is only available in Postgres 12 and above. If you are using version 11 or earlier, the approach in the thoughtbot article is probably still the best one.

As we are venturing into the territory of more custom Postgres functionality, not easily supported by the Rails schema file in Ruby, we'll want to switch the schema format from :ruby to :sql. This line can be added to the application.rb file:

config.active_record.schema_format = :sql

Now, let's generate a migration to add a new column to the jobs table which will be automatically generated based on the setweight and to_tsvector functions:

class AddSearchableColumnToJobs < ActiveRecord::Migration[6.0]
  def up
    execute <<-SQL
      ALTER TABLE jobs
      ADD COLUMN searchable tsvector GENERATED ALWAYS AS (
        setweight(to_tsvector('english', coalesce(title, '')), 'A') ||
        setweight(to_tsvector('english', coalesce(description,'')), 'B')
      ) STORED;
    SQL
  end

  def down
    remove_column :jobs, :searchable
  end
end

Note that, as of the writing of this article, Postgres always requires a generated column to be a “Stored” column. That means it actually occupies space in your table and gets written on each INSERT/UPDATE. This also means that when you add a generated column to a table, it will require a rewrite of the table to actually set the values for all existing rows. This may block other operations on your database.

With our tsvector column added (which is giving precedence to the title over the description, is using the english dictionary, and is coalescing null values into empty strings), we're ready to add an index to it. Either GIN or GiST indexes can be used to speed up full text searches, but Postgres recommends GIN as the preferred index due to GiST searches being lossy, which may produce false matches. We'll add it concurrently to avoid locking issues when adding an index to large tables.

class AddIndexToSearchableJobs < ActiveRecord::Migration[6.0]
  disable_ddl_transaction!

  def change
    add_index :jobs, :searchable, using: :gin, algorithm: :concurrently
  end
end

The last thing we need to do is to tell pg_search to use our tsvector searchable column, rather than re-parsing the title and description fields each time. This is done by adding the tsvector_column option to tsearch:

class Job < ApplicationRecord
  include PgSearch::Model
  pg_search_scope :search_job,
                  against: { title: 'A', description: 'B' },
                  using: {
                    tsearch: {
                      dictionary: 'english', tsvector_column: 'searchable'
                    }
                  }
end

With this optimization done, we have gone from around 130ms to 7ms per query... not bad at all!

Conclusion

Let’s have a look at a real-life data set. We can prove the precision of our approach by looking at my database: Out of 145 jobs pulled from the GitHub and Hacker News job APIs, searching for "Ruby on Rails" returns the following results:

[
  "Ruby on Rails Developer",
  "Senior Ruby Developer - Remote",
  "Gobble (YC W14) – Senior Full Stack Software Engineers – Toronto, On",
  "DevOps (Remote - Europe)",
  "CareRev (YC S16) Is Hiring a Senior Back End Engineer in Los Angeles",
  "Software Engineer, Full Stack (Rails, React)",
  "Software Engineer",
  "Technology Solutions Developer"
]

To summarize:

We have shown how to use Postgres' Full Text Search within Rails and also how to customize it both in terms of functionality, but also in terms of performance. We ended up with a performant and flexible solution right inside the database we were already using.

Many use cases for Full Text Search can be implemented directly inside Postgres, avoiding the need to install and maintain additional services such as Elasticsearch.

If you find this article useful and want to share it with your peers you can tweet about it here.

About the author

]]>

Effectively Using Materialized Views in Ruby on Rails

Leigh Halliday — Thu, 16 Jan 2020 12:00:00 GMT

It's every developer's nightmare: SQL queries that get large and unwieldy. This can happen fairly quickly with the addition of multiple joins, a subquery and some complicated filtering logic. I have personally seen queries grow to nearly one hundred lines long in both the financial services and health industries.

Luckily Postgres provides two ways to encapsulate large queries: Views and Materialized Views. In this article, we will cover in detail how to utilize both views and materialized views within Ruby on Rails, and we can even take a look at creating and modifying them with database migrations.

Our example will be a real-world sized dataset of hockey teams and their top scorers. If you'd like to follow along, the source code covered in this article can be found here.

What is a view?
What makes a view materialized?
Creating a materialized view
Utilizing a materialized view
Refreshing a materialized view
When to use views vs. materialized views?
Migrating views
Testing with materialized views
Conclusion
About the Author

We'll also talk a bit about the performance benefits that a Materialized View can bring to your application:

What is a view?

A view allows us to query against the result of another query, providing a powerful way of abstracting away a complex query full of joins, conditions, groupings, and any other clause that can be added to an SQL query. Looking at the query below, it isn't overly complex, but it does include 3 joins, grouping by a number of fields to aggregate the numbers of goals scored for a player each season.

It takes approximately 450ms to execute on my computer. I am using seed data that generates 31 teams, each playing 200 games in a season, scoring 20 goals per game... a little unrealistic, but I wanted the dataset used to be substantial!

SELECT
  players.name AS player_name,
  players.id AS player_id,
  players.position AS player_position,
  matches.season AS season,
  teams.name AS team_name,
  teams.id AS team_id,
  count(goals.id) AS goal_count
FROM goals
  INNER JOIN players ON (goals.player_id = players.id)
  INNER JOIN matches ON (goals.match_id = matches.id)
  INNER JOIN teams ON (goals.team_id = teams.id)
GROUP BY players.id, teams.id, matches.season

A view allows us to take the final result of this query, and query against that as if it were any other table. You can see why views can come in handy in many different scenarios. They allow for the succinct abstraction of a complicated query, and allow us to re-use this logic in a simple to understand way.

Now, we could make a new view by running CREATE VIEW in Postgres. But, as we all know, one-off schema changes are hard to keep track of. Instead, let's try something thats closer to how Rails does things. How does that look like? First things first, we'll create a view using Scenic. Scenic gives us the ability to define migrations that create, update, or drop views, just as you're used to doing with regular tables in Rails.

rails g scenic:view top_scorers

This will generate two files. The first is named db/views/top_scorers_v01.sql, and in it we will paste the SQL for the underlying query (from above). The second is db/migrate/[date]_create_top_scorers.rb, and this is where the migration will live to migrate/rollback the creation of our view:

class CreateTopScorers < ActiveRecord::Migration[6.0]
  def change
    create_view :top_scorers
  end
end

With the view in place, we can now query against it. This query takes approximately 50ms to execute.

SELECT *
FROM top_scorers
WHERE
  team_name = 'Toronto Maple Leafs'
ORDER BY goal_count DESC

By creating a model in Rails, we can interact with it much like we would be able to with a typical model which is backed by a table. First things first, let's define the model, letting Rails know it is read-only. Whilst some views can be updated, this view contains a top-level GROUP BY clause and thus can't be updated.

# app/models/top_scorer.rb
class TopScorer < ApplicationRecord
  def readonly?
    true
  end
end

Now we can perform the same SQL query using the TopScorer model:

TopScorer.where(team_name: 'Toronto Maple Leafs').order(goal_count: :desc)

What makes a view materialized?

A regular view still performs the underlying query which defined it. It will only be as efficient as its underlying query is. This means, if the larger query discussed above takes 450ms to execute, executing SELECT * FROM top_scorers will also take 450ms.

Materialized views take regular views to the next level, though they aren't without their drawbacks. The difference is that they save the result of the original query to a cached/temporary table. When you query a materialized view, you aren't querying the source data, rather the cached result.

This can provide serious performance benefits, especially considering you can index materialized views. But, when the underlying data from the source tables is updated, the materialized view becomes out of date, serving up an older cached version of the data. We can resolve this by refreshing the materialized view, which we'll get to in a bit.

Creating a materialized view

Just like we saw with our regular view, materialized views begin the same way, by executing a command to generate a new view migration: rails g scenic:view mat_top_scorers. This produces two files, the first of which contains the SQL to produce the underlying view of the data. The difference is in the migration, passing in materialized: true to the create_view method. Also notice that we are able to add indexes to the materialized view.

class CreateMatTopScorers < ActiveRecord::Migration[6.0]
  def change
    create_view :mat_top_scorers, materialized: true

    add_index :mat_top_scorers, :player_name
    add_index :mat_top_scorers, :player_id
    add_index :mat_top_scorers, :team_name
    add_index :mat_top_scorers, :team_id
    add_index :mat_top_scorers, :season
  end
end

Utilizing a materialized view

Like a regular view, we are able to define an ActiveRecord model that can query it. Also notice that we can define relationships which point to other ActiveRecord models. If you didn't know, you might not even realize it is pointing to a materialized view, except for the readonly? method which was defined.

class MatTopScorer < ApplicationRecord
  belongs_to :player
  belongs_to :team
  belongs_to :match

  def self.top_scorer_for_season(season)
    where(season: season).order(goal_count: :desc).first
  end

  def readonly?
    true
  end
end

Let's take our new materialized view for a spin! Running the query select * from mat_top_scorers, which took 450ms as a view, takes 5ms as a materialized view, 90x faster! The ruby code below, which took 50ms as a view, takes under 1ms to execute!

MatTopScorer.where(team_name: 'Toronto Maple Leafs').order(goal_count: :desc)

For a side-by-side comparison, this performs the same query on both views:

irb(main):001:0> TopScorer.where(team_name: 'Toronto Maple Leafs').count
   (60.2ms)  SELECT COUNT(*) FROM "top_scorers" WHERE "top_scorers"."team_name" = $1  [["team_name", "Toronto Maple Leafs"]]
=> 30

irb(main):002:0> MatTopScorer.where(team_name: 'Toronto Maple Leafs').count
   (1.3ms)  SELECT COUNT(*) FROM "mat_top_scorers" WHERE "mat_top_scorers"."team_name" = $1  [["team_name", "Toronto Maple Leafs"]]
=> 30

Refreshing a materialized view

As mentioned previously, materialized views cache the underlying query's result to a temporary table. This is what gives us the speed improvements and the ability to add indexes. The downside is that we have to control when the cache is refreshed. Modifying the MatTopScorer model, let's add a refresh method that can be called any time the data is to be refreshed. You will need to figure out how often it makes sense to update the data for your specific use-case, depending on how often the data is changing and how quickly those changes need to be reflected to the end user.

class MatTopScorer < ApplicationRecord
  belongs_to :player
  belongs_to :team
  belongs_to :match

  def self.refresh
    Scenic.database.refresh_materialized_view(table_name, concurrently: false, cascade: false)
  end

  def self.top_scorer_for_season(season)
    where(season: season).order(goal_count: :desc).first
  end

  def readonly?
    true
  end
end

To schedule the refresh, I like to use the whenever gem. Let's call a rake task to refresh the materialized view every hour:

# config/schedule.rb
every 1.hour do
  rake "refreshers:mat_top_scorers"
end

The rake task is simple, only calling the refresh method defined on the MatTopScorer model.

# lib/tasks/refreshers.rake
namespace :refreshers do
  desc "Refresh materialized view for top scorers"
  task mat_top_scorers: :environment do
    MatTopScorer.refresh
  end
end

When to use views vs. materialized views?

Views focus on abstracting away complexity and encouraging reuse. Views allow you to interact with the result of a query as if it were a table itself, but they do not provide a performance benefit, as the underlying query is still executed, perfect for sharing logic but still having real-time access to the source data.

Materialized Views are related to views, but go a step further. You get all the abstraction and reuse of a view, but the underlying data is cached, providing serious performance benefits. Materialized views are especially useful for - for example - reporting dashboards because they can be indexed to allow for performant filtering.

If the purpose of the view is to provide a cleaner interface to complicated joins and query logic, and performance isn't too much of an issue, by all means stick with a regular view. Views have the advantage of always being real-time, since they simply reference the real underlying data rather than a cached copy of it.

If your purpose is to provide a cleaner interface in addition to performance improvements, and you can live with the data being not quite real-time, then creating it as a materialized view can provide some great benefits.

Migrating views

It's easy to migrate views in Scenic. Views are versioned by default in Scenic and generating a view with the same name will create a v2, providing two files, just like it did the first time we generated a view (earlier in this article). Likewise, Scenic also provides a way to drop a view.

Testing with materialized views

Views and materialized views aren't particularly challenging to test, but it does require remembering that both types of views don't contain any original data in and of themselves, they are either a live view of an underlying query, or a cached view of an underlying query, as in the case of materialized views.

Let's see how we would populate and then test our MatTopScorer model in RSpec and factory_bot.

After creating some test data using factory_bot, we'll call a method which is supposed to return the top scorer for a given season. It returns nil, and that is expected. The underlying data exists, but because materialized views must be refreshed, something we haven't done yet, there is no data to be found.

After calling MatTopScorer.refresh, we're now able to retrieve the expected result.

RSpec.describe MatTopScorer, type: :model do
  describe "#top_scorer_for_season" do
    it "finds top scorer" do
      # create some data using factory_bot helper methods
      match = create(:match)
      player = create(:player)
      goal = create(:goal, match: match, player: player)

      # without any data in materialized view
      expect(MatTopScorer.top_scorer_for_season(match.season)).to eq(nil)

      MatTopScorer.refresh

      # with data in materialized view
      top_scorer = MatTopScorer.top_scorer_for_season(match.season)
      expect(top_scorer).to be_present
      expect(top_scorer.player).to eq(player)
      expect(top_scorer.goal_count).to eq(1)
    end
  end
end

Conclusion

With the help of Scenic, using views and materialized views feels right at home in Rails. Truthfully, I haven't used views as much as I have used materialized views. In particular, I've found materialized views incredibly useful when building searchable reporting dashboards.

The ability to group and summarize data by geographic region, category, grouped by date, in combination with adding the correct indexes has provided an efficient way to report on large amounts of data without relying on external reporting systems or causing excessive load on the production database.

Share this article: If you liked this article we’d appreciate it if you’d tweet it to your peers.

About the Author

]]>

Similarity in Postgres and Rails using Trigrams

Leigh Halliday — Tue, 19 Nov 2019 12:00:00 GMT

You typed "postgras", did you mean "postgres"?

Use the best tool for the job. It seems like solid advice, but there's something to say about keeping things simple. There is a training and maintenance cost that comes with supporting an ever growing number of tools. It may be better advice to use an existing tool that works well, although not perfect, until it hurts. It all depends on your specific case.

Postgres is an amazing relational database, and it supports more features than you might initially think! It has full text search, JSON documents, and support for similarity matching through its pg_trgm module.

What are Trigrams?
- Postgres Trigram example
- Ruby Trigram example
Using Trigrams in Rails
Showing the closest matches for a term based on its similarity
Conclusion
About the Author

Today, we will break down how to use pg_trgm for a light-weight, built-in similarity matcher. Why are we doing this? Well, before reaching for a tool purpose-built for search such as Elasticsearch, potentially complicating development by adding another tool to your development stack, it's worth seeing if Postgres suits your application's needs! You may be surprised!

In this article, we will look at how it works under the covers, and how to use it efficiently in your Rails app.

What are Trigrams?

Trigrams, a subset of n-grams, break text down into groups of three consecutive letters. Let's see an example: postgres. It is made up of six groups: pos, ost, stg, tgr, gre, res.

This process of breaking a piece of text into smaller groups allows you to compare the groups of one word to the groups of another word. Knowing how many groups are shared between the two words allows you to make a comparison between them based on how similar their groups are.

Postgres Trigram example

Postgres' pg_trgm module comes with a number of functions and operators to compare strings. We'll look at the show_trgm and similarity functions, along with the % operator below:

select
  show_trgm('postgras') as tri1, -- {"  p"," po","as ",gra,ost,pos,ras,stg,tgr}
  show_trgm('postgres') as tri2, -- {"  p"," po","es ",gre,ost,pos,res,stg,tgr}
  similarity('postgras','postgres'), -- 0.5
  'postgras' % 'postgres' -- TRUE

The show_trgm function isn't one you'd necessarily use day-to-day, but it's good to see how Postgres breaks a string down into trigrams. You'll notice something interesting here, that two spaces are added to the beginning of the string, and a single space is added to the end.

This is done for a couple of reasons:

The first reason is that it allows trigram calculations on words with less than three characters, such as Hi.

Secondly, it ensures the first and last characters are not overly de-emphasized for comparisons. If we used only strict triplets, the first and last letters in longer words would each occur in only a single group: with padding they occur in three (for the first letter) and two (for the last). The last letter is less important for matching, which means that postgres and postgrez are more similar than postgres and postgras, even though they are both off by a single character.

The similarity function compares the trigrams from two strings and outputs a similarity number between 1 and 0. 1 means a perfect match, and 0 means no shared trigrams.

Lastly, we have the % operator, which gives you a boolean of whether two strings are similar. By default, Postgres uses the number 0.3 when making this decision, but you can always update this setting.

Ruby Trigram example

You don't need to know how to build a trigram in order to use them in Postgres, but it doesn't hurt to dive deeper and expand your knowledge. Let's take a look at how to implement something similar ourselves in Ruby.

The first method will take a string, and output an array of trigrams, adding two spaces to the front, and one to the back of the original string, just like Postgres does.

def trigram(word)
  return [] if word.strip == ""

  parts = []
  padded = "  #{word} ".downcase
  padded.chars.each_cons(3) { |w| parts << w.join }
  parts
end

p trigram("postgras")
# ["  p", " po", "pos", "ost", "stg", "tgr", "gra", "ras", "as "]
p trigram("postgres")
# ["  p", " po", "pos", "ost", "stg", "tgr", "gre", "res", "es "]

Next up, we'll compare the trigrams from our two words together, giving a ratio of how similar they are:

def similarity(word1, word2)
  tri1 = trigram(word1)
  tri2 = trigram(word2)

  return 0.0 if [tri1, tri2].any? { |arr| arr.size == 0 }

  # Find number of trigrams shared between them
  same_size = (tri1 & tri2).size
  # Find unique total trigrams in both arrays
  all_size = (tri1 | tri2).size

  same_size.to_f / all_size
end

p similarity("postgras", "postgres")
# 0.5

Now that we have our similarity calculator, we can implement a simple similar? method, which checks if the similarity is above the threshold of 0.3:

def similar?(word1, word2)
  similarity(word1, word2) >= 0.3
end

p similar?("postgras", "postgres")
# true

Using Trigrams in Rails

There aren't too many gotchas in order to use these similarity functions and operators within your Rails app, but there are a couple!

Below we have a migration to create a cities table. When indexing the name column, to ensure that querying with the similarity operator stays fast, we'll need to ensure that we use either a gin or gist index. We do this by indicating using: :gin. In addition to that, we have to pass the opclass option opclass: :gin_trgm_ops, so it knows which type of gin index to create.

Unless you have already enabled the pg_trgm extension, you will most likely receive an error, but this is easily fixed by adding enable_extension :pg_trgm to your migration.

class CreateCities < ActiveRecord::Migration[6.0]
  def change
    enable_extension :pg_trgm

    create_table :cities do |t|
      t.string :name, null: false
      t.timestamps
      t.index :name, opclass: :gin_trgm_ops, using: :gin
    end
  end
end

Now that we have the pg_trgm extension enabled, and have correctly indexed the table, we can use the similarity operator % inside of our where clauses, such as in the scope below:

class City < ApplicationRecord
  scope :name_similar, ->(name) { where("name % :name", name: name) }
end

City.name_similar("Torono").count
# SELECT COUNT(*) FROM "cities" WHERE (name % 'Torono')

Showing the closest matches for a term based on its similarity

We may not want to only limit by similarity using the % operator, but also order the results from most similar to least similar. Take the example query and its result below:

select name, similarity(name, 'Dease Lake')
from cities
where name % 'Dease Lake'
order by 2 desc

This query finds cities which have a name similar to Dease Lake, but you can see that we actually get seven results back, though we can clearly see that there was an exact match. Ideally then, we wouldn't just limit our query by similarity, but put it in the correct order as well.

Dease Lake  1
Deer Lake   0.5
Lake Louise 0.375
Lynn Lake   0.33333334
Red Lake    0.33333334
Cat Lake    0.33333334
Baker Lake  0.3125

We can do this by updating our scope to order by similarity. We have to be careful about this, because in order to use the similarity function, we need to pass in the user input of 'Dease Lake'. To avoid SQL injection attacks and to ensure safe string quoting, we'll use the quote_string method from ActiveRecord::Base.

class City < ApplicationRecord
  scope :name_similar, ->(name) {
    quoted_name = ActiveRecord::Base.connection.quote_string(name)
    where("name % :name", name: name).
      order(Arel.sql("similarity(name, '#{quoted_name}') DESC"))
  }
end

Now when we use the name_similar scope, the result will be ordered with the most similar city first, allowing us to find Dease Lake:

City.name_similar("Dease Lake").first.name
# => Dease Lake

And the SQL produced looks like:

SELECT "cities".*
FROM "cities"
WHERE (name % 'Dease Lake')
ORDER BY similarity(name, 'Dease Lake') DESC
LIMIT $1

Conclusion

In this article, we took a dive into the pg_trgm extension, seeing first what trigrams actually are, and then how we can practically use similarity functions and operators in our Rails apps. This allows us to improve keyword searching, by finding similar, rather than exact matches. We also managed to accomplish all of this without adding an additional backend service, or too much additional complexity to our application.

Share this article: If you liked this article we'd appreciate it if you'd tweet it to your peers.

About the Author

]]>

Efficient GraphQL queries in Ruby on Rails & Postgres

Leigh Halliday — Tue, 24 Sep 2019 12:00:00 GMT

GraphQL puts the user in control of their own destiny. Yes, they are confined to your schema, but beyond that they can access the data in any which way. Will they ask only for the "events", or also for the "category" of each event? We don't really know! In REST based APIs we know ahead of time what will be rendered, and can plan ahead by generating the required data efficiently, often by eager-loading the data we know we'll need.

In this article, we will discuss what N+1 queries are, how they are easily produced in GraphQL, and how to solve them using the graphql-batch gem along with a few custom batch loaders.

The source code for this article is available on GitHub.

What are N+1 queries?
N+1 queries in GraphQL
Optimizing GraphQL queries
Batch loading single records
Batch loading many records
Batch loading many records more efficiently
Batch loading active storage attachments
Conclusion
About the Author

What are N+1 queries?

N+1 queries can occur when you have one-to-many relationships in your models. Each Event belongs to a Category. Let's say that you find the last five events and you want to get the category name for each of them.

Event.last(5).each { |event| puts event.category.name }

Seems simple enough! We unfortunately just produced six queries. The first query to find the events, and another query to find each category's name. This is an easy problem to solve in Rails by using eager-loading:

Event.includes(:category).last(5).each { |event| puts event.category.name }

By using the includes method we've been able to knock our queries down from six to two: The first to find the events, and the second to find the categories for those events.

N+1 queries in GraphQL

As we mentioned earlier, in GraphQL, the user is in charge of their own destiny. They may or may not ask for the category name for each event. The query below will produce N+1 SQL queries as it finds the category for each event:

{
  events {
    id
    name
    category {
      id
      name
    }
  }
}

Optimizing GraphQL queries

Yes, we could solve the N+1 query in the previous example by eager-loading the category relationship, but if the user didn't actually want the category, why load it? We don't know what the user will ask for. There just so happens to be a better way, by lazy-loading data only as its needed using the graphql-batch gem.

Marc-André Giroux has written a very thorough article about the GraphQL Dataloader Pattern which I highly recommend reading before continuing.

Batch loading single records

The simplest case for batch loading data is the example of each event belonging to a category. Inside of our EventType class, there is a field called category which allows the user to access the category of an event.

class Types::EventType < Types::BaseObject
  field :category, Types::CategoryType, null: false

  def category
    # avoid `object.category`
    RecordLoader.for(Category).load(object.category_id)
  end
end

By using the RecordLoader class to load the category, we actually avoid loading the category right away, and instead load all of the required categories with a single query. The query it ends up producing may end up looking like:

SELECT "categories".* FROM "categories" WHERE "categories"."id" IN ($1, $2, $3, $4, $5)

Looking at the RecordLoader class we can see how it works. The perform method will receive all of the ids for a single model (Category in this case), load the records in a single SQL query, and then call the fulfill method for each of them. The fulfill method resolves the promise, which is basically like putting a face to a name... you gave me an ID, and I've fulfilled my promise to provide you with the corresponding record.

class RecordLoader < GraphQL::Batch::Loader
  def initialize(model)
    @model = model
  end

  def perform(ids)
    # Find all ids for this model and fulfill their promises
    @model.where(id: ids).each { |record| fulfill(record.id, record) }
    # Handle cases where a record was not found and fulfill the value as nil
    ids.each { |id| fulfill(id, nil) unless fulfilled?(id) }
  end
end

We can write a test for this class to ensure that it finds the records correctly, keeping in mind that in order for the lazy/promise based code to function correctly it needs to be wrapped inside something called an executor.

describe RecordLoader do
  it 'loads' do
    event = create(:event)
    result = GraphQL::Batch.batch do
      RecordLoader.for(Event).load(event.id)
    end
    expect(result).to eq(event)
  end
end

Batch loading many records

We've covered the case where we are batch loading a single record at a time, but how do we handle the reverse scenario? We are displaying categories along with the first five events for each category, which would also produce an N+1 query, so let's see how we can solve it using a batch loader. The query we're discussing would look something like this:

{
  categories {
    id
    name
    events(first: 5) {
      id
      name
    }
  }
}

I have created a custom loader called ForeignKeyLoader for this purpose. It will load the events using the foreign key category_id. I also added the ability to pass a lambda to merge in additional scopes into the query that will be run.

class Types::CategoryType < Types::BaseObject
  field :events, [Types::EventType], null: false do
    argument :first, Int, required: false, default_value: 5
  end

  def events(first:)
    ForeignKeyLoader.for(Event, :category_id, merge: -> { order(id: :asc) }).
      load(object.id).then do |records|
        records.first(first)
      end
  end
end

The query that gets produced looks something like:

SELECT "events".*
FROM "events"
WHERE "events"."category_id" IN ($1, $2, $3, $4, $5)
ORDER BY "events"."id" ASC

Notice in this case that we call the then method to execute some code after the promise has been resolved. Here we see the first issue with this method... we only wanted five events for each category, but our query will load ALL events for each category, and then, using the first method on the resulting Array, narrow it down to only the first five events. If there are thousands of events, we could run into some serious issues.

class ForeignKeyLoader < GraphQL::Batch::Loader
  attr_reader :model, :foreign_key, :merge

  def self.loader_key_for(*group_args)
    # avoiding including the `merge` lambda in loader key
    # each lambda is unique which defeats the purpose of
    # grouping queries together
    [self].concat(group_args.slice(0,2))
  end

  def initialize(model, foreign_key, merge: nil)
    @model = model
    @foreign_key = foreign_key
    @merge = merge
  end

  def perform(foreign_ids)
    # find all the records
    scope = model.where(foreign_key => foreign_ids)
    scope = scope.merge(merge) if merge.present?
    records = scope.to_a

    foreign_ids.each do |foreign_id|
      # find the records required to fulfill each promise
      matching_records = records.select do |r|
        foreign_id == r.send(foreign_key)
      end
      fulfill(foreign_id, matching_records)
    end
  end
end

Batch loading many records more efficiently

It turns out that there is a way to perform a query that says "find me the first N records for each X" (find me the first 5 records for each category), and that involves using Postgres Window Functions. While researching this concept, this article about window functions was useful along with this article about bringing window functions into Rails.

The following query produces the data that we want... we just need to figure out how to write a batch loader that generates the same result.

SELECT "events".*
FROM (
  SELECT
    *,
    row_number() OVER (
      PARTITION BY category_id ORDER BY start_time desc
    ) as rank
  FROM "events"
  WHERE "events"."category_id" IN (1, 2, 3, 4, 5)
) as events
WHERE rank <= 5

For this we'll create a batch loader called WindowKeyLoader which is used like:

class Types::CategoryType < Types::BaseObject
  field :events, [Types::EventType], null: false do
    argument :first, Int, required: false, default_value: 5
  end

  def events(first:)
    WindowKeyLoader.for(Event, :category_id,
      limit: first,
      order_col: :start_time,
      order_dir: :desc
    ).load(object.id)
  end
end

You can see the difference already. I am no longer required to slice the first N array elements in the then block of the resolved promise. The actual batch loader class looks like:

class WindowKeyLoader < GraphQL::Batch::Loader
  attr_reader :model, :foreign_key, :limit, :order_col, :order_dir

  def initialize(model, foreign_key, limit:, order_col: :id, order_dir: :asc)
    @model = model
    @foreign_key = foreign_key
    @limit = limit
    @order_col = order_col
    @order_dir = order_dir
  end

  def perform(foreign_ids)
    # build the sub-query, limiting results by foreign key at this point
    # we don't want to execute this query but get its SQL to be used later
    ranked_from = model.
      select("*,
        row_number() OVER (
          PARTITION BY #{foreign_key} ORDER BY #{order_col} #{order_dir}
        ) as rank").
      where(foreign_key => foreign_ids).
      to_sql

    # use the sub-query from above to query records which have a rank
    # value less than or equal to our limit
    records = model.
      from("(#{ranked_from}) as #{model.table_name}").
      where("rank <= #{limit}").
      to_a

    # match records and fulfill promises
    foreign_ids.each do |foreign_id|
      matching_records = records.select do |r|
        foreign_id == r.send(foreign_key)
      end
      fulfill(foreign_id, matching_records)
    end
  end
end

We're able to test the WindowKeyLoader by creating three events for a category but only asking for the first two of them:

describe WindowKeyLoader do
  it 'loads' do
    category = create(:category)
    events = (1..3).to_a.map do |n|
      create(:event, name: "Event #{n}", category: category)
    end

    result = GraphQL::Batch.batch do
      WindowKeyLoader.for(
        Event,
        :category_id,
        limit: 2, order_col: :id, order_dir: :asc
      ).load(category.id)
    end

    expect(result).to eq(events.first(2))
  end
end

Batch loading active storage attachments

You may run into situations where you're loading polymorphic data, or other types of relationships which don't exactly fit into the mold of your standard has-many or belongs-to relationships. One case is with ActiveStorage. In the code below we'll load an image URL for an event:

class Types::EventType < Types::BaseObject
  field :image, String, null: true

  def image
    # produces 2N + 1 queries... yikes!
    # url_for(object.image.variant({ quality: 75 }))

    AttachmentLoader.for(:Event, :image).load(object.id).then do |image|
      url_for(image.variant({ quality: 75 }))
    end
  end
end

This data is stored using a polymorphic relationship that loads an ActiveStorage::Attachment record, which then needs to load an ActiveStorage::Blob record in order to produce the image url. It ends up producing a 2N + 1 query... yikes! Our AttachmentLoader is able to completely optimize this field by cutting it down to just two queries to load as many images as you'd like.

class AttachmentLoader < GraphQL::Batch::Loader
  attr_reader :record_type, :attachment_name

  def initialize(record_type, attachment_name)
    @record_type = record_type
    @attachment_name = attachment_name
  end

  def perform(record_ids)
    # find records and fulfill promises
    ActiveStorage::Attachment.
      includes(:blob).
      where(record_type: record_type, record_id: record_ids, name: attachment_name).
      each { |record| fulfill(record.record_id, record) }

    # fulfill unfound records
    record_ids.each { |id| fulfill(id, nil) unless fulfilled?(id) }
  end
end

In this case we are taking advantage of eager-loading, because for each attachment we will need its corresponding blob record.

Conclusion

GraphQL can be as efficient as REST, but requires approaching optimizations from a different angle. Instead of upfront optimizations, we lazy-load data only when required, loading it in batches to avoid excess trips to the database. In this article, we covered techniques to load single records, multiple records, and records with different types of relationships, as is the case with Active Storage which has a polymorphic relationship.

Share this article: If you liked this article we'd appreciate it if you'd tweet it to your peers.

About the Author

]]>

pganalyze Blog - Articles by Leigh Halliday

PostGIS vs. Geocoder in Rails

Installing PostGIS

ActiveRecord PostGIS Adapter

Our Example Data

Building a Geo Helper Class with PostGIS

Finding Nearby Records with PostGIS and Geocoder

Finding Records Within a Bounding Box with PostGIS and Geocoder

Finding Records Within a Polygon with PostGIS and Geocoder

Finding Nearby Related Records with PostGIS and Geocoder

Conclusion

About the Author

Advanced Active Record: Using Subqueries in Rails

Working with Active Record in Rails

What are Subqueries in Rails

An Overview of our Data

The Where Subquery

Where Not Exists

The Select Subquery

The From Subquery

The Having Subquery

Conclusion

You might also be interested in

About the author

Full Text Search in Milliseconds with Rails and PostgreSQL

The Foundations of Full Text Search

Implementing Postgres Full Text Search in Rails

Configuring pg_search

Optimizing Full Text Search Queries in Rails

Conclusion

You might also be interested in

About the author

Effectively Using Materialized Views in Ruby on Rails

What is a view?

What makes a view materialized?

Creating a materialized view

Utilizing a materialized view

Refreshing a materialized view

When to use views vs. materialized views?

Migrating views

Testing with materialized views

Conclusion

About the Author

Similarity in Postgres and Rails using Trigrams

What are Trigrams?

Postgres Trigram example

Ruby Trigram example

Using Trigrams in Rails

Showing the closest matches for a term based on its similarity

Conclusion

About the Author

Efficient GraphQL queries in Ruby on Rails & Postgres

What are N+1 queries?

N+1 queries in GraphQL

Optimizing GraphQL queries

Batch loading single records

Batch loading many records

Batch loading many records more efficiently

Batch loading active storage attachments

Conclusion

About the Author