Branching Out Into Hierarchy Data

2020-07-15T00:00:00+00:00

ltree is a Postgres extension that can give you and your application a lot of options around complex hierarchal relationships.

At some point in an applications development lifecycle, you can end up having to figure out how to represent hierarchal data. In hierarchal data, a root or 'parent' node can have 'child' nodes attached to it. For example, A (parent) -> B, C (children) or grilled cheese (parent) -> bread, cheese, mayo (children).

While these relationships can be stored in a variety of ways, such as raw strings, arrays, lookup tables, bitmasks, and so on... Postgres has a dedicated type and extension for this type of data called ltree.

To get started with ltree, check to see if it is available in your setup:

select * from pg_available_extensions where name = 'ltree';

If you see it listed as an available extension, you can install it:

create extension if not exists ltree;

Confirm installation:

select * from pg_extension where extname='ltree';

Note: Magistrate makes your available extension list just a click away.

Now we can build our first hierarchy:

create table menu (id serial, title text, ingredients ltree);
insert into menu (title, ingredients)
values ('basic', 'tortilla.cheese'), ('starter', 'tortilla.cheese.carne'), ('extra', 'tortilla.cheese.carne.pollo.guac'), ('setup', 'chips.salsa');

If I am curious what on the menu has a tortilla as the parent ingredient:

select * from menu where 'tortilla' @> ingredients;

If want something that has a tortilla and cheese:

select * from menu where 'tortilla*.cheese.*' ~ ingredients;

If I want something that has a tortilla and cheese, but not guac because of an allergy or preference:

select * from menu where 'tortilla.cheese.!guac' ~ ingredients;

One of the reasons you might lean towards a dedicated type for this problem is all the supporting functions and query options that go along with ltree.
With the simple examples above and tiny data set, we could any solution would likely work fine. However, with more complex relationships and datasets, using ltree can give you a lot more options including more intelligent indexing for performance, recursion on your queries, and more confident precision with lquery stanzas. Using ltree can also get you out of the trap of thinking you need to build your own handler for when you leave the trivial examples.

While Magistrate can't solve all your hierarchal data needs, it can certainly get you started in making sure the extension is available and helping you determine when you have a performance issue with your lquery work.

Using pg_repack in AWS RDS

2020-10-13T00:00:00+00:00

As your database keeps growing, there is a good chance you're going to have to address database bloat.
While Postgres 13 has launched with some exciting features with built-in methods to rebuild indexes concurrently, many people still end up having to use pg_repack to do an online rebuild of the tables to remove the bloat. Customers on AWS RDS struggle figuring out how to do this. Ready to learn how?

Since you have no server to access the local binaries, and because AWS RDS provides no binaries for the versions they are using, you're going to have to build your own. This isn't as hard as one might think because the official pg repos have an installer (ie: sudo apt install postgresql-10-pg_repack). If you don't use the repos, the project itself, is an open source project with directions: http://reorg.github.io/pg_repack/

While you were getting up to speed above, I was spinning up a postgres 10.9 db on RDS. I started it yesterday so that it would be ready by the time you got to this part of the post. Lets create some data:

-- let's create the table
CREATE TABLE burritos (
id SERIAL UNIQUE NOT NULL primary key,
title VARCHAR(10) NOT NULL,
toppings TEXT NOT NULL,
thoughts TEXT,
code VARCHAR(4) NOT NULL,
UNIQUE (title, toppings)
);

--disable auto vacuum
ALTER TABLE burritos SET (autovacuum_enabled = false, toast.autovacuum_enabled = false);


-- orders up
INSERT INTO  burritos (title, toppings, thoughts, code)
SELECT
  left(md5(i::text), 10),
  md5(random()::text),
  md5(random()::text),
  left(md5(random()::text), 4)
FROM GENERATE_SERIES(1, 1000000) s(i);


UPDATE burritos SET toppings = md5(random()::text) WHERE id < 250;
UPDATE burritos SET toppings = md5(random()::text) WHERE id between 250 and 500;
UPDATE burritos SET code = left(md5(random()::text), 4) WHERE id between 2050 and 5000;
UPDATE burritos SET thoughts = md5(random()::text) WHERE id between 10000 and 20000;
UPDATE burritos SET thoughts = md5(random()::text) WHERE id between 800000 and 900000;
UPDATE burritos SET toppings = md5(random()::text) WHERE id between 600000 and 700000;

(If you are curious how Magistrate presents bloat, here is a clip of the screen:)

Much like a human that has had that much interaction with burritos... our database has quite a bit of bloat. Assuming we already have the pg_repack binaries in place, either though compilation or installing the package on the OS, we now need to enable the extension. We've put together a handy reference for installing extensions to get you going.

pg_repack has a lot of options. Feel free to check them out, but I'm going to start packing:

/usr/local/bin/pg_repack -U greataccounthere -h bloatsy.csbv99zxhbsh.us-east-2.rds.amazonaws.com -d important -t burritos -j 4
NOTICE: Setting up workers.conns
ERROR: pg_repack failed with error: You must be a superuser to use pg_repack

This might feel like game over because of the implementation of superuser on RDS, but the trick is to take a leap of faith and add another flag (-k) that skips the superuser check:

/usr/local/bin/pg_repack-1.4.3/pg_repack -U greataccounthere -h bloatsy.csbv99zxhbsh.us-east-2.rds.amazonaws.com -k -d important -t burritos -j 4
NOTICE: Setting up workers.conns
INFO: repacking table "public.burritos"
LOG: Initial worker 0 to build index: CREATE UNIQUE INDEX index_16449 ON repack.table_16442 USING btree (id) TABLESPACE pg_default
LOG: Initial worker 1 to build index: CREATE UNIQUE INDEX index_16451 ON repack.table_16442 USING btree (title, toppings) TABLESPACE pg_default
LOG: Command finished in worker 0: CREATE UNIQUE INDEX index_16449 ON repack.table_16442 USING btree (id) TABLESPACE pg_default
LOG: Command finished in worker 1: CREATE UNIQUE INDEX index_16451 ON repack.table_16442 USING btree (title, toppings) TABLESPACE pg_default

It works! The table is feeling fresh and tidy and your application has a little more pep in its step. When using Magistrate our platform matrix also knows when you have pg_repack installed and gives you the commands to run for tables it detects with high bloat percentage.

Magistrate Postgres Blog by Rob Sullivan

Branching Out Into Hierarchy Data

Using pg_repack in AWS RDS