Order/Group by Column Numbers

When quickly iterating on a query its a lot less typing to put the column number as the thing you want to order by. Here's a quick lightweight example:

SELECT
  email,
  created_at
FROM 
  users
ORDER BY 2 DESC
LIMIT 5;

This gives me my last 5 users that have signed up for my site. Of course as soon as I have this I may want to add some data to it, like their first name so I can send them a welcome email. I quickly alter the query to:

SELECT
  email,
  first_name,
  created_at
FROM 
  users
ORDER BY 2 DESC
LIMIT 5;

And now I have 5 users that have signed up ordered by their first name. Sure its obvious when you have 1 column you're ordering by, but when you have GROUP BY 1, 2, 3, 4, 5, 6 which is actually open in one of my tabs currently its a bit more confusing....

Though if you really want to have some fun, share a query with someone that looks something like this:

SELECT
  email as "3",
  first_name "2",
  created_at "1"
FROM 
  users
ORDER BY "1", "3" DESC
LIMIT 5;

Implicit Joins

I seldom use the syntax INNER JOIN. Instead I simply put the two tables in my where clause and ensure I have a where condition. The problem with ensuring I have a where condition is sometimes I don't, especially when you're dealing with 3 tables.

SELECT 
  email,
  product.name,
  product.price
FROM 
  users,
  orders,
  items
WHERE users.id = orders.user_id
  AND orders.id = items.order_id

Is less clear (especially when dealing with 5-6 tables) than the alternative:

SELECT 
  email,
  product.name,
  product.price
FROM users
INNER JOIN orders on users.id = orders.user_id
INNER JOIN items on orders.id = items.order_id

Lack of comments

I comment my SQL far less than I comment my code, yet it can be done just as easily. For example I have this in one of my queries:

SELECT convert_from(CAST(E'\\x' || array_to_string(ARRAY(
   SELECT 
     CASE 
       WHEN length(r.m[1]) = 1 
     THEN encode(convert_to(r.m[1], 'SQL_ASCII'), 'hex') 
     ELSE substring(r.m[1] from 2 for 2) 
     END
  FROM regexp_matches(url_here, '%[0-9a-f][0-9a-f]|.', 'gi') AS r(m)
), '') AS bytea), 'UTF8');

While this has its own issues theres no documentation around what this actually does, in contrast:

--- DECODES url ---
SELECT convert_from(CAST(E'\\x' || array_to_string(ARRAY(
   SELECT 
     CASE 
       WHEN length(r.m[1]) = 1 
     THEN encode(convert_to(r.m[1], 'SQL_ASCII'), 'hex') 
     ELSE substring(r.m[1] from 2 for 2) 
     END
  FROM regexp_matches(url_here, '%[0-9a-f][0-9a-f]|.', 'gi') AS r(m)
), '') AS bytea), 'UTF8');

Comments also work well inline at the end of a line.

Large Manually Generated Lists

A lot of times in working with some specific data set I'll manually or automatically generate a list that I want to filter. A common example is filtering out staging/dev environments. I'll often manually search and prune the list, then save that result for the queries I'm going to build going forward. This is a bit of effort but still feels reasonable the downside is it results in something like:

SELECT 
  foo
FROM 
  bar
WHERE 
  bar.id NOT IN (34723, 42735, 32321, 47205, 20375, 30261, 26194, 109371, 9313, 6351, 20184, 50273, 34735, 39854, 23954, 25323, 23405, 30528, 50182, 29340, 47659, ... and the list goes on)

SQL is meant to be reasonable for containing some level of logic. Data changes, hard coding keys is going to bite you at some point, spend the extra effort and re-use something thats clear.

What else

I'm sure theres plenty more; I suspect within a few minutes of sitting down with someone they could point out some other bad habits. While I know mine at least some of mine I still often know the trade-off. What are yours? I'd love to hear to document them for others so hopefully they can prevent developing the same bad habits. Let me know; craig.kerstiens@gmail.com

Postgres has a flexible and robust array datatype that comes with a variety of functions. Even without taking advantage of the array datatype in your application, you can still take advantage of some of the functions to get the functionality you need. Lets take a look at an example schema and use case.

An example

Given a project management application, you may have users who have projects that have tasks. An example piece of functionality might be to send an email with a list of all projects that have tasks that are past their due dates of completion. Your schema might look something like this:

 # \d users
             Table "public.users"
    Column   |            Type             | Modifiers
 ------------+-----------------------------+-----------
  id         | integer                     | not null
  email      | character varying(255)      |
  ...

# \d projects
             Table "public.projects"
    Column   |            Type             | Modifiers
 ------------+-----------------------------+-----------
  id         | integer                     | not null
  user_id    | integer                     | not null
  name       | character varying(255)      | not null
  ...

# \d tasks
             Table "public.tasks"
    Column     |            Type             | Modifiers
 --------------+-----------------------------+-----------
  id           | integer                     | not null
  project_id   | integer                     | not null
  completed_at | timestamp without time zone | 
  due_at       | timestamp without time zone | 
  ...

To get a list of all projects that have tasks that haven't been completed, you would start with something like:

SELECT 
  projects.name
FROM
  projects,
  tasks
WHERE projects.id = tasks.project_id
  AND tasks.due_at > tasks.completed_at
  AND tasks.due_at > now()

This would give you a list of projects which you could then easily join this with users:

SELECT 
  users.email
  projects.name
FROM
  projects,
  tasks,
  users
WHERE projects.id = tasks.project_id
  AND tasks.due_at > tasks.completed_at
  AND tasks.due_at > now()
  AND users.id = projects.user_id

At this point you've got everything you need to pull this up into Ruby, Python, or other language of your choice and then build the full set. However if this is thousands or even hundreds of results you'll be spending more time than necessary, grouping this data for a sensible email. With 3 other small changes you can have this already formatted for you to immediately send of in an email. The first is using a handy function called array_agg which will aggregate items and then you can format them how you wish. The second is just ensuring you're grouping correctly. Finally you'll want to unnest the array so it formats the data in a clean way for you.

Looking at it all put together:

SELECT 
  users.email,
  array_to_string(array_agg(projects.name), ',')) as projects
FROM
  projects,
  tasks,
  users
WHERE projects.id = tasks.project_id
  AND tasks.due_at > tasks.completed_at
  AND tasks.due_at > now()
  AND users.id = projects.user_id
GROUP BY 
  users.email

This would give you a nice clean result of projects that have overdue tasks that you could then send to the user in an email:

           email            |     projects       
 ---------------------------+-------------------
 craig.kerstiens@gmail.com  | blog, timetracker      
 craig@heroku.com           | foo, bar, baz    

Get the most of psql

Psql, the CLI postgreSQL client, is a powerful tool. Sadly, lot of developers are not aware of the features and instead look for a GUI to provide what they need. Let's fly over what can psql do for you.

Feel yourself at home

One of the most common misconception people have about CLI is «They are a poor user interface». C'mon, the CLI is the most efficient user interface ever. There is nothing to disturb you from what you are doing and you are by far fastest without switching to your mouse all the time. Let's see how we can configure psql at our convenience.

First, you'll have managed to choose a nice and fancy terminal font like monofur or inconsolata. Do not underestimate the power of the font

The nice line style shown above can be set with \pset linestyle unicode and \pset border 2. This is just an example of the many environment variables you can play with to get your preferred style of working out of psql.

For example, I found the character ¤ the most accurate to express nullity (instead of default NULL). Let's just \pset null ¤ and here it is:

SELECT * FROM very_interesting_stat;
┌──────┬──────┬──────┬──────┬──────┐
│  a   │  b   │  c   │  d   │  e   │
├──────┼──────┼──────┼──────┼──────┤
│ 9.06 │    ¤ │    ¤ │    ¤ │    ¤ │
│ 7.30 │ 3.55 │ 7.57 │ 3.31 │    ¤ │
│ 7.20 │ 5.08 │    ¤ │ 6.58 │ 5.90 │
...

Another hugely value to get environment variables is colors in the prompt. Colors in the prompt are important because it makes easier to spot where output starts and ends between two interactions at the console. The PROMPT1 environment variable will even let you set an indicator to notify you are inside a transaction or not, give this a try for a sweet surprise...

\set PROMPT1 '%[%033[33;1m%]%x%[%033[0m%]%[%033[1m%]%/%[%033[0m%]%R%# '

I also like to disable the pager by default \pset pager off and display the time every issued query takes \timing. If you are used to psql, you may notice in the picture above, some content is wrapped. This is \pset format wrapped option.

Of course, writing all that on every connection would be a pain, so just write them in a ~/.psqlrc file, it will be sourced every time psql is launched.

If you are familiar with bash or other recent unix shells, you might also declare aliases in your configuration file. You can do the same with psql. For example if you want to have a query for slow queries such as from this earlier post but not have to remember the query every time you can set it up as:

\set show_slow_queries 
'SELECT 
  (total_time / 1000 / 60) as total_minutes, 
  (total_time/calls) as average_time, query 
FROM pg_stat_statements 
ORDER BY 1 DESC 
LIMIT 100;'

Now, just entering :show_slow_queries in your psql client will launch this query and give you the results:

    total_time    |     avg_time     |                                                                                                                                                              query
------------------+------------------+------------------------------------------------------------
 295.761165833319 | 10.1374053278061 | SELECT id FROM users WHERE email LIKE ?
 219.138564283326 | 80.24530822355305 | SELECT * FROM address WHERE user_id = ? AND current = True

Psql at your fingertips

Now you have got a fancy prompt, here is the real question you ask, what can psql do for me ? and \? has all of the answers. It has built-in queries to describe almost all database objects from tables to operators, indexes, triggers etc... with clever auto-completion. Not only completion on tables and columns -- but also on aliases (sweet), SQL commands (w00t) and database objects.

Now we can enter some SQL commands. As usual, you need to check in the documentation how the heck to write this damn ALTER TABLE. Relax, psql proposes inline documentation. Just enter \h alter table (auto complete w00t) and you ll be ok.

Interacting with your editor

psql provides two very handy commands: \e and \i. This last command sources a sql file in the client's current session. \e edits the last command using the editor defined in the EDITOR shell environment variables (aka vim). This grant you with real editor feature when it comes to writing long queries. What psql does, it saves the buffer in a temporary file and fires up the editor with that file. Once the editor is terminated, psql sources the file. Of course, you can use your editor to save queries in other places where they would be under version control, but the \e has a serious limitation: it spawns only the last query. Even if you sent several queries on the same line. (Note that \r clears psql's last query buffer).

Note: \ef my_function opens stored function source code (With auto completion, I know, it's awesome).

Vim users can here benefit from Vim's server mode. If you launch a vim specifying a server name (let' say "PSQL") somewhere, and set the EDITOR variable as is export EDITOR="vim --servername PSQL --remote-tab-wait then psql will open a new tab on the running vim with the last query and run it as soon as you close this tab. Tmux or gnu/screen users will split their screen to have Vim and psql running on the same terminal window.

Call a friend

Vim power users know it is possible to pipe a buffer (or selection) directly in a program that can be ... psql (Using the :w !psql syntax). Even from the shell, you might want to take advantage of the fantastic \copy feature that loads formated file in the database (I use it to load apache logs). But always having to specify connection parameters are a hassle. Let's use shell environment instead. Psql is sensitive to the following variables:

• PGDATABASE
• PGHOST
• PGPORT
• PGUSER
• PGCLUSTER (debian wrapper).

Set them once for all in you shell environment and call psql to connect to the database. In case you want to skip password prompt, you can store your pass in a 600 mode access file named .pgpass in your home (do not do that on shared or exposed computers). Although this is nice for development database servers, I do NOT recommend this for production servers since it should not be easy to mess with them.

Resource for additional information is ... the man page and Postgres Docs. All PostgreSQL documentation is an example of what software reference documentation should be. Enjoy!

I've been hunting for a nice PG interface that works within other things. PGAdmin kinda works, except the SQL editor is a piece of shit

Sometimes it leans more to, what is the Sequel Pro equivilant for Postgres. My default answer is I just use psql, though I do have to then go on to explain how I use it. For those just interested you can read more below or just get the highlights here:

• Set your default EDITOR then use \e
• On postgres 9.2 and up \x auto is your friend
• Set history to unlimited
• \d all the things

Before going into detail on why psql works perfectly fine as an interface I want to rant for a minute about what the problems with current editors are and where I expect them to go in the future. First this is not a knock on the work thats been done on previous ones, for their time PgAdmin, phpPgAdmin, and others were valuable tools, but we're coming to a point where theres a broader set of users of databases than ever before and empowering them is becoming ever more important.

Empowering developers, DBA's, product people, marketers and others to be comfortable with their database will lead to more people taking advantage of whats in their data. pg_stat_statements was a great start to this laying a great foundation for valuable information being captured. Even with all of the powerful stats being captured in the statistics of PostgreSQL so many are still terrified when they see something like:

                                                   QUERY PLAN
----------------------------------------------------------------------------------------------------------------
 Hash Join  (cost=4.25..8.62 rows=100 width=107) (actual time=0.126..0.230 rows=100 loops=1)
   Hash Cond: (purchases.user_id = users.id)
   ->  Seq Scan on purchases  (cost=0.00..3.00 rows=100 width=84) (actual time=0.012..0.035 rows=100 loops=1)
   ->  Hash  (cost=3.00..3.00 rows=100 width=27) (actual time=0.097..0.097 rows=100 loops=1)
         Buckets: 1024  Batches: 1  Memory Usage: 6kB
         ->  Seq Scan on users  (cost=0.00..3.00 rows=100 width=27) (actual time=0.007..0.042 rows=100 loops=1)
 Total runtime: 0.799 ms
(7 rows)

Empowering more developers by surfacing this information in a digestable form, such as building on top of pg_stat_statements tools such as datascope by @leinweber and getting this to be part of the default admin we will truly begin empowering a new set of user.

But enough of a detour, those tools aren't available today. If you're interested in helping build those to make the community better please reach out. For now I live in a work where I'm quite content with simple ole psql here's how:

Editor

Ensuring you've exported your preferred editor to the environment variable EDITOR when you run \e it will allow you to view and edit your last run query in your editor of choice. This works for vim, emacs, or even sublime text.

export EDITOR=subl
psql
\e 

Gives me:

Note you need to make sure you connect with psql and have your editor set, once you do that saving and exiting the file will then execute the query

\x auto

psql has long had a method of formatting output. You can toggle this on and off easily by just running the \x command. Running a basic query you get the output:

SELECT * 
FROM users 
LIMIt 1;
 id | first_name | last_name |           email            |    data    |     created_at      |     updated_at      |     last_login
 ----+------------+-----------+----------------------------+------------+---------------------+---------------------+---------------------
   1 | Rosemary   | Wassink   | Rosemary.Wassink@yahoo.com | "sex"=>"F" | 2010-07-01 18:16:00 | 2011-05-14 11:47:00 | 2011-06-07 23:04:00

With toggling the output and re-running the same query we can see how its now formatted:

\x
Expanded display is on.
craig=# SELECT * from users limit 1;
-[ RECORD 1 ]--------------------------
id         | 1
first_name | Rosemary
last_name  | Wassink
email      | Rosemary.Wassink@yahoo.com
data       | "sex"=>"F"
created_at | 2010-07-01 18:16:00
updated_at | 2011-05-14 11:47:00
last_login | 2011-06-07 23:04:00

Using \x auto will automatically put this in what Postgres believes is the most intelligible format to read it in.

psql history

Hopefully this needs no justification... having an unlimited history of all your queries is incredibly handy. Ensuring you set the following environment variables will ensure you never lose that query you ran several months ago again:

export HISTFILESIZE=
export HISTSIZE=

\d

And while the last on the list one of the first things I do when connecting to any database is check out whats in it. I don't do this by running a bunch of queries but rather checking out the schema and then poking at definitions of specific tables. \d and variations on it are incredibly handy for this. Here's a few highlights below:

Listing all relations with simply \d:

\d
                 List of relations
 Schema |       Name       |     Type      | Owner
--------+------------------+---------------+-------
 public | products         | table         | craig
 public | products_id_seq  | sequence      | craig
 public | purchases        | table         | craig
 public | purchases_id_seq | sequence      | craig
 public | redis_db0        | foreign table | craig
 public | users            | table         | craig
 public | users_id_seq     | sequence      | craig
(7 rows)

List only all tables with dt:

\dt
         List of relations
 Schema |   Name    | Type  | Owner
--------+-----------+-------+-------
 public | products  | table | craig
 public | purchases | table | craig
 public | users     | table | craig
(3 rows)

Describe a specific relation with \d RELATIONNAMEHERE:

\d users
                                     Table "public.users"
   Column   |            Type             |                     Modifiers
------------+-----------------------------+----------------------------------------------------
 id         | integer                     | not null default nextval('users_id_seq'::regclass)
 first_name | character varying(50)       |
 last_name  | character varying(50)       |
 email      | character varying(255)      |
 data       | hstore                      |
 created_at | timestamp without time zone |
 updated_at | timestamp without time zone |
 last_login | timestamp without time zone |

One more pro-tip if you're running a transaction with many tables and forget which are involved in it you can run '\d *transaction*' and it'll display tables curently affected.

Have a tool you prefer, have something you use daily in psql that I missed, or interested in helping create a new admin experience please reach out and lets talk craig.kerstiens at gmail.com

It all starts with stats

I wrote about some of the great new features in Postgres 9.2 in the recent announcement on support of Postgres 9.2 on Heroku. One of those awesome features, is pg_stat_statements. Its not commonly known how much information Postgres keeps about your database (beyond the data of course), but in reality it keeps a great deal. Ranging from basic stuff like table size to cardinality of joins to distribution of indexes, and with pg_stat_statments it keeps a normalized record of when queries are run.

First you'll want to turn on pg_stat_statments:

CREATE extension pg_stat_statements;

What this means it would record both:

SELECT id 
FROM users
WHERE email LIKE 'craig@heroku.com';

and

SELECT id 
FROM users
WHERE email LIKE 'craig.kerstiens@gmail.com';

To a normalized form which looks like this:

SELECT id 
FROM users
WHERE email LIKE ?;

Understanding them from afar

While Postgres collects a great deal of this information dissecting it to something useful is sometimes more mystery than it should be. This simple query will show a few very key pieces of information that allow you to begin optimizing:

SELECT 
  (total_time / 1000 / 60) as total_minutes, 
  (total_time/calls) as average_time, 
  query 
FROM pg_stat_statements 
ORDER BY 1 DESC 
LIMIT 100;

The above query shows three key things:

1. The total time a query has occupied against your system in minutes
2. The average time it takes to run in milliseconds
3. The query itself

Giving an output something like:

    total_time    |     avg_time     |                                                                                                                                                              query
------------------+------------------+------------------------------------------------------------
 295.761165833319 | 10.1374053278061 | SELECT id FROM users WHERE email LIKE ?
 219.138564283326 | 80.24530822355305 | SELECT * FROM address WHERE user_id = ? AND current = True
(2 rows)

What to optimize

A general rule of thumb is that most of your very common queries that return 1 or a small set of records should return in ~ 1 ms. In some cases there may be queries that regularly run in 4-5 ms, but in most cases ~ 1 ms or less is an ideal.

To pick where to begin I usually attempt to strike some balance between total time and long average time. In this case I'd start with the second probably, as on the first one I could likely shave an order of magnitude off, on the second I'm hopeful to shave two order of magnitudes off thus reducing the time spent on that query from a cumulative 220 minutes down to 2 minutes.

Optimizing

From here you probably want to first example my other detail on understanding the explain plan. I want to highlight some of this with a more specific case based on the second query above. The above second query on an example data set does contain an index on user_id and yet there's still high query times. To start to get an idea of why I would run:

EXPLAIN ANALYZE
SELECT * 
FROM address 
WHERE user_id = 245 
  AND current = True

This would yield results:

                                                                                   QUERY PLAN
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 Aggregate  (cost=4690.88..4690.88 rows=1 width=0) (actual time=519.288..519.289 rows=1 loops=1)
   ->  Nested Loop  (cost=0.00..4690.66 rows=433 width=0) (actual time=15.302..519.076 rows=213 loops=1)
         ->  Index Scan using idx_address_userid on address  (cost=0.00..232.52 rows=23 width=4) (actual time=10.143..62.822 rows=1 loops=8)
               Index Cond: (user_id = 245)
               Filter: current
               Rows Removed by Filter: 14
 Total runtime: 219.428 ms
(1 rows)

Hopefully not being too overwhelmed by this due to having read the other detail on query plans we can see that it is using an index as expected. The difference is its having to fetch 15 different rows from the index then discard the bulk of them. The number of rows discarded is showcased by the line:

Rows Removed by Filter: 14

This is just one more of the many improvements in Postgres 9.2 alongside pg_stat_statements.

To further optimize this we would great a conditional OR composite index. A conditional would be where only current = true, where as the composite would index both values. A conditional is commonly more valuable when you have a smaller set of what the values may be, meanwhile the composite is when you have a high variability of values. Creating the conditional index:

CREATE INDEX CONCURRENTLY idx_address_userid_current ON address(user_id) WHERE current = True;

We can then see the query plan is now even further improved as we'd hope:

EXPLAIN ANALYZE
SELECT * 
FROM address 
WHERE user_id = 245 
  AND current = True

                                                                                   QUERY PLAN
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 Aggregate  (cost=4690.88..4690.88 rows=1 width=0) (actual time=519.288..519.289 rows=1 loops=1)
     ->  Index Scan using idx_address_userid_current on address  (cost=0.00..232.52 rows=23 width=4) (actual time=10.143..62.822 rows=1 loops=8)
           Index Cond: ((user_id = 245) AND (current = True))
 Total runtime: .728 ms
(1 rows)

For further reading, give Greg Smith's Postgres High Performance a read