Home on Philipp Steinrötter

Postgres Async Triggers

philipp@steinroetter.com (psteinroe) — Mon, 24 Feb 2025 00:00:00 +0000

## Postgres Async Triggers

Run asynchronous side-effects on database events.

This blog post is accompanied by a demo repository on GitHub. Check it out here.

At hellomateo, we rely heavily on Supabase. And like any SaaS, we need to execute side-effects like sending a Webhook after a record changed. At first, we manually created database triggers that inserted jobs into the Postgres-based queue Graphile Worker after insert/update/delete. Not only was this a bad DX, we also hit scalability issues: the fetch job query from Graphile Worker dominated our database load, and the maxed out workers were not able to process jobs fast enough during peak times.

As an intermediary solution, we forwarded the critical jobs from Graphile Worker to QStash from Upstash. This helped with the maxed out workers at the cost of higher latency. As a messaging solution, higher latency is bad though. But still, things stabilised for the time being which gave us time to find a better solution.

Our goal was not only to solve the scalability issues, but also to improve DX. We wanted to reduce the surface area to write business logic. It should be easy to implement and test, and promote functional programming.

# Exploring Change Data Capture

First, we looked into the scalability issue, and explored how to get database events into workers efficiently. The obvious option is Change Data Capture (CDC). CDC is a mechanism based on Write-Ahead-Logging provided by Postgres, more specifically logical decoding. It allows us to subscribe to any change happening in the database. It’s almost instant, and without overhead. But it’s a complex beast. To better understand the protocol, I created a quick proof of concept for a pipeline that batches the events before sending them to to a HTTP endpoint. While the PoC was fun and the approach is feasible, there are a lot of things to take care of, and we did not want to take on that extra complexity.

Next, we looked for tools that do the job for us. There is WalEx, an Elixir module to implement async triggers. Thats great, but we want to keep using Typescript. During that time, I also got in contact with Sequin. Their product is great, but its designed more as an infrastructure piece, and not towards application developers. We want to define our triggers in the code, and Sequin requires you to use their Dashboard or Terraform. There are other tools like Debezium, but their operational complexity is way too high. We decided that while CDC can be a great solution further down the road, its complexity is not justified yet.

# The Simple Solution is Good Enough

We took a step back, and began tinkering on improving what we already have (and know well): How can we reduce the database load and reduce latency without compromising on horizontal scalability?

First, QStash had to be replaced. A HTTP based queue will always introduce latency we cannot afford. We quickly decided on BullMq, because we were already using Redis for caching and it’s a battle-proven queuing system with low double digit latency.

Now to the fun part: how to reduce the database load from Graphile Worker while increasing throughput? The only option is to start batching jobs. We have tables that trigger a lot of side effects, and each job was inserted from its own database trigger similar to this:

-- first side effect on `my_table`
create or replace function public.first_side_effect() returns trigger as $$
begin
 perform private.add_graphile_worker_job (
 'first_side_effect',
 to_json(new),
 priority := 'high'::private.graphile_worker_priority
 );

 return new;
end
$$ language plpgsql volatile set search_path = '' security definer;

create trigger first_side_effect
after insert on my_table for each row
when (new.my_column is true)
execute procedure first_side_effect();

-- second side effect on `my_table`
create or replace function public.second_side_effect() returns trigger as $$
begin
 perform private.add_graphile_worker_job (
 'second_side_effect',
 to_json(new),
 priority := 'high'::private.graphile_worker_priority
 );

 return new;
end
$$ language plpgsql volatile set search_path = '' security definer;

create trigger second_side_effect
after insert on my_table for each row
when (new.another_column is true)
execute procedure second_side_effect();

To use batching, we would need to have a single trigger that inserts a single batched job for each table and operation (AFTER [INSERT|UPDATE|DELETE]).

create or replace function public.run_side_effects() returns trigger as $$
declare
 v_payload json := '[]'::json
begin
 if new.my_column is true then
 v_payload := v_payload || jsonb_build_object('tg_name', 'first_side_effect', 'new', to_json(new));
 end if;

 if new.another_column is true then
 v_payload := v_payload || jsonb_build_object('tg_name', 'second_side_effect', 'new', to_json(new));
 end if;

 perform private.add_graphile_worker_job (
 'side_effects',
 v_payload,
 priority := 'high'::private.graphile_worker_priority
 );

 return new;
end
$$ language plpgsql volatile set search_path = '' security definer;

create trigger first_side_effect
after insert on my_table for each row
when (new.my_column is true or new.another_column is true)
execute procedure first_side_effect();

Here, we move the when clauses into if clauses, and add the job to the list of jobs if it evaluates to true. The Graphile Worker then picks up the batched job, and forwards the payloads based on the tg_name. Note that this sample is missing a few important details, but more on that later.

We did a quick proof of concept and the impact was significant.

In our very first iteration, we had a single trigger function that used a lot of execute format to dynamically build the payloads based on tg_name and tg_op, but that was even worse than not batching. Turns out, the execution context switch Postgres has to perform with execute format is very costly.

Now, how do we fix the DX? The database triggers are boilerplate, and we do not want to require a new migration every time we implement a side-effect anymore.

# Crafting the DX: Implementing Async Triggers

Our goal with “async triggers” is to have a single place where a developer implements, tests and registers side-effects. For each trigger, we want to bootstrap the DDL for the actual database trigger that checks the conditions for each subscription and inserts a batch job.

The DSL we came up with is very similar to its equivalent in SQL:

import { builder } from '../../builder';

export default builder
 .createTrigger('my-side-effect')
 .on('my_table')
 .withColumns('column_a,column_b')
 .afterInsert("new.column_a is true")
 .afterUpdate("old.column_b is not true")
 .afterDelete()
 .execute(
 async (payload, dependencies) => {
 // full typed:
 // payload.new.column_a
 // dependencies.supabaseClient.from()
 },
 { concurrency: 300 }
 );

Any async trigger declares up to three subscriptions on a table: insert, update and delete. Unlike database triggers, each subscription has its own optional when clause to filter on old and new. We also enforce the selection of columns to reduce the payload size. The execute function receives a typed event payload as well as globally declared dependencies.

The payload types are the only integration with Supabase. Everything else is just plain Postgres and Node.

All functions are registered on the server and collected on startup of the service. Before we start processing jobs, a set_subscriptions rpc is called that merges all subscriptions into the async_trigger.subscription table:

create table if not exists async_trigger.subscription (
 id uuid not null default gen_random_uuid() primary key,
 app_name text not null,
 name text not null,
 operation async_trigger.operation_type not null, -- INSERT, UPDATE, DELETE
 schema_name text not null,
 table_name text not null,
 when_clause text,
 column_names text[],
 unique (name, operation, schema_name, table_name, app_name)
);

The subscription table itself has a trigger that fires after insert, update and delete to sync the database triggers. It dynamically generates and executes the create function and create trigger statements to create one trigger for every operation and table. The generated DDL for the async trigger defined above looks somewhat like this:

create or replace function async_trigger._publish_events_after_update_on_my_table ()
 returns trigger
 as $inner$
declare
 v_jsonb_output jsonb := '[]'::jsonb;

 -- shared payload
 v_base_payload jsonb := jsonb_build_object(
 'tg_op', tg_op,
 'tg_table_name', tg_table_name,
 'tg_table_schema', tg_table_schema,
 'auth', jsonb_build_object(
 'role', auth.role(),
 'user_id', auth.uid()
 ),
 'timestamp', (extract(epoch from now()) * 1000)::bigint
 );
begin
 if old.column_b is not true then
 v_jsonb_output := v_jsonb_output || (jsonb_build_object(
 'tg_name', 'my-side-effect',
 'new', case when tg_op is distinct from 'DELETE' then jsonb_build_object(
 'column_a', new.column_a, 'column_b', new.column_b
 ) else null end,
 'old', case when tg_op is distinct from 'INSERT' then jsonb_build_object(
 'column_a', new.column_a, 'column_b', new.column_b
 ) else null end,
 ) || v_base_payload);
 end if;

 -- if ...some other clause then
 -- v_jsonb_output := ...
 -- end if;

 if jsonb_array_length(v_jsonb_output) > 0 then
 perform private.add_graphile_worker_job(
 'publish_events',
 v_jsonb_output::json,
 priority := 'high'::private.graphile_worker_priority
 );
 end if;

 if tg_op = 'DELETE' then
 return old;
 end if;

 return new;
end
$inner$
language plpgsql
security definer;

create trigger async_trigger_after_update
after update on my_table for each row
when (new.my_column is true or new.another_column is true)
execute procedure async_trigger._publish_events_after_update_on_my_table ();

In the trigger function, we add an if block for every “subscription” and populate the payload with all relevant data. There is also payload shared by all subscriptions like the current user.

The final job payload looks like this:

{
 "auth":{
 "role":"authenticated",
 "user_id":"xx"
 },
 "new":{
 "column_a":"value",
 "column_b":"value"
 },
 "tg_name":"my-trigger",
 "tg_op":"INSERT",
 "tg_table_name":"event",
 "tg_table_schema":"public",
 "timestamp":1739550115638
}

Now, lets take a look at the “trigger that creates the triggers”. Its a bit more involved, so I added comments to help you go through it.

See the trigger in its full glory

create or replace function async_trigger.sync_trigger()
 returns trigger as
$$
declare
 v_app_name text := coalesce(new.app_name, old.app_name);
 v_table_name text := coalesce(new.table_name, old.table_name);
 v_schema_name text := coalesce(new.schema_name, old.schema_name);

 v_when_clause text;

 v_if_blocks text;

 v_op async_trigger.operation_type;
begin
 -- every subscription can have all operations, so we need to iterate through them
 foreach v_op in array array['INSERT', 'UPDATE', 'DELETE']::async_trigger.operation_type[] loop

 -- clean up any existing trigger and function
 execute format(
 $sql$drop trigger if exists async_trigger_after_%s%s on %I.%I;$sql$,
 lower(v_op::text), v_test_postfix, v_schema_name, v_table_name
 );

 execute format(
 $sql$drop function if exists async_trigger._publish_events_after_%s_on_%s%s;$sql$,
 lower(v_op::text), v_table_name, v_test_postfix
 );

 -- check if we need to create a new trigger for the current operation
 if exists (select 1 from async_trigger.subscription where table_name = v_table_name and schema_name = v_schema_name and operation = v_op) then

 -- build the `WHEN` clause for the trigger
 -- if there is at least one subscription for current operation without a when_clause, we do not add the when clause at all
 -- otherwise, we combine them with OR, e.g. (old.col is true) OR (new.col is true)
 v_when_clause := (
 case when exists (
 select 1
 from async_trigger.subscription
 where table_name = v_table_name and schema_name = v_schema_name and operation = v_op and (when_clause is null or when_clause = '') and app_name = v_app_name
 ) then null
 else (
 select string_agg(when_clause, ') or (')
 from async_trigger.subscription
 where table_name = v_table_name and schema_name = v_schema_name and operation = v_op and when_clause is not null and when_clause != '' and app_name = v_app_name
 )
 end
 );

 -- build the if block for every subscription
 v_if_blocks := (
 select string_agg(format(
 $sql$
 -- the WHEN clause is put here
 if %s then
 v_jsonb_output := v_jsonb_output || (jsonb_build_object(
 -- the trigger name
 'tg_name', %L,
 -- dynamically build the new and the old data from the selected `column_names`
 'new', case when tg_op is distinct from 'DELETE' then jsonb_build_object(
 %s
 ) else null end,
 'old', case when tg_op is distinct from 'INSERT' then jsonb_build_object(
 %s
 ) else null end
 -- merge with the shared payload from below
 ) || v_base_payload);
 end if;
 $sql$,
 coalesce(nullif(subscription.when_clause, ''), 'true'),
 subscription.destination,
 subscription.name,
 (select string_agg(format($s$%L, new.%I$s$, column_name, column_name), ', ') from unnest(subscription.column_names) as column_name),
 (select string_agg(format($s$%L, old.%I$s$, column_name, column_name), ', ') from unnest(subscription.column_names) as column_name)
 ), e'\n') from async_trigger.subscription where table_name = v_table_name and schema_name = v_schema_name and operation = v_op and (v_is_test_app is false or app_name = v_app_name)
 );

 -- bootstrap the `CREATE FUNCTION` statement and execute it
 execute format(
 $sql$
 create or replace function async_trigger._publish_events_after_%s_on_%s ()
 returns trigger
 as $inner$
 declare
 v_jsonb_output jsonb := '[]'::jsonb;

 v_base_payload jsonb := jsonb_build_object(
 'tg_op', tg_op,
 'tg_table_name', tg_table_name,
 'tg_table_schema', tg_table_schema,
 'auth', jsonb_build_object(
 'role', auth.role(),
 'user_id', auth.uid()
 ),
 'timestamp', (extract(epoch from now()) * 1000)::bigint
 );
 begin
 -- this is where we put all the if blocks
 %s

 -- if at least one subscription evaluates to true, insert the job
 if jsonb_array_length(v_jsonb_output) > 0 then
 perform private.add_graphile_worker_job(
 'publish_events',
 v_jsonb_output::json,
 priority := 'high'::private.graphile_worker_priority
 );
 end if;
p
 if tg_op = 'DELETE' then
 return old;
 end if;

 return new;
 end
 $inner$
 language plpgsql
 security definer;
 $sql$,
 lower(v_op::text),
 v_table_name,
 v_if_blocks
 );

 -- bootstrap the `CREATE TRIGGER` statement and execute it
 execute format(
 $sql$
 create trigger async_trigger_after_%s
 after %s on %I.%I
 for each row
 %s
 -- execute the function we created above
 execute procedure async_trigger._publish_events_after_%s_on_%s()
 $sql$,
 lower(v_op::text),
 lower(v_op::text),
 v_schema_name,
 v_table_name,
 case when v_when_clause is not null and length(v_when_clause) > 0
 then 'when ((' || v_when_clause || '))'
 else ''
 end,
 lower(v_op::text),
 v_table_name
 );
 end if;
 end loop;

 return new;
end
$$ language plpgsql security definer;

Graphile Worker now simply iterates the jobs and forwards them to BullMQ. We use this canary release of Graphile Worker that implements local queues to maximise throughput and reduce database load at a theoretical latency cost. But we found that due to our single-digit job processing time and with at least two parallel workers, the downsides are negligible.

# Summary

And that’s it for the async triggers! We successfully reduced the load on our database and increased throughput by batching the export jobs and thanks to the local queue mode. The latency is very low too: the BullMQ worker is executed in less than 100ms after the transaction completes on the database. We are also happy with the DX. A new developer should be able to implement business logic quickly. And the best part: we are still using only the simple tools.

# Future Work

Right now, the framework is published as a shadcn-style demo. It is meant as a starting point for others to build their own. At some point, I hope to make this a more generic solution, where Graphile worker and BullMQ are just adapters for a generic „transport layer“. I would also love to see a variant that uses only Supabase infrastructure:

pg_net to get the jobs out of the database
pg_mq as the queue
Edge Functions to execute the jobs

We did not go this path because at that time queues were not released yet, we don’t use edge functions, and this architecture would not fulfil our throughput and latency requirements. Still, it is a viable option for many, and I hope PgFlow can evolve into this.

# Additional Goodies

This blog post focusses on the async trigger implementation only. But the framework you can find in the demo repository is capable of more. We implemented an abstraction layer on top of BullMQ that allows for a boilerplate-free and type-safe implementation of business logic in a very functional manner. Functions can call, trigger and schedule other functions, and every handler receives globally defined dependencies that are created from the environment in a type-safe way. There is also a test framework to test both triggers and functions. Although a lot of things can still be polished, we are happy with its current status and hope it’s already of value for the community.

If you want to help build this, we are hiring in Berlin! You can reach me on philipp@hellomateo.de.

Build A Public API with Supabase in 10 Minutes

philipp@steinroetter.com (psteinroe) — Sat, 20 Apr 2024 00:00:00 +0000

Building with Supabase means most likely you are not buildig your API yourself thanks to PostgREST and the GraphQL extension. But what if your customers want a public API to integrate with your product? Instead of building your own API server from scratch, you can leverage Postgres + PostgREST to build a public API in minutes. This blog post walks you through the required steps. The target architecture is very simple: A custom database role and Row-Level-Security, your PostgREST server, and a proxy that adds rate-limiting and observability, and a few additional security measures.

# The Database Schema

First we create a sample schema. In this case, we create an organisation table. Each organisation has employees. We also have a contact table that we want to expose in our public API.

create table organisation
(
 id uuid primary key not null default gen_random_uuid(),
 name text unique not null
);

create table employee
(
 id uuid primary key not null default gen_random_uuid(),
 organisation_id uuid not null references organisation on update restrict on delete cascade default private.organisation_id(),
 user_id uuid unique references auth.users on update restrict on delete cascade
);

-- we want to expose this in our public api!
create table contact
(
 id uuid primary key not null default gen_random_uuid(),
 organisation_id uuid not null references organisation on delete cascade default private.organisation_id(),
 full_name text
);

We use Row-Level-Security to secure our data and make it accessible only if the user is authenticated and employee of the organisation. The private.organisation_id() function is a little helper function that returns the organisation_id of the authenticated user, if any.

create or replace function private.organisation_id() returns uuid as
$sql$
select organisation_id from employee where user_id = auth.uid()
$sql$ language sql stable security definer;

create policy employee_all on organisation to authenticated using (
 (select private.organisation_id()) = id
);

create policy employee_all on employee to authenticated using (
 (select private.organisation_id()) = organisation_id
);

create policy employee_all on contact to authenticated using (
 (select private.organisation_id()) = organisation_id
);

## Setup API Tokens

We will manage and create api tokens right within our database, and use Row-Level-Security to define what the api token can do. The first step is to create a role tokenauthed that can be used to authenticate with the database. It should behave like the authenticated role.

create role tokenauthed;
grant tokenauthed to authenticator;
grant anon to tokenauthed;

## Create the API Token Table

We store references to api tokens in a table. Note that the key itself is not stored. Employees should be able to manage tokens.

create table if not exists api_token
(
 id uuid primary key not null default gen_random_uuid(),
 organisation_id uuid not null references organisation on delete cascade default private.organisation_id(),
 created_at timestamp with time zone not null default now(),
 name text not null
);

alter table api_token enable row level security;

create policy employee_all on api_token to authenticated using (
 (select private.organisation_id()) = organisation_id
);

## Allow users to actually create a token

To actually create the api tokens we need to mint our own jwt tokens. We will use the pgjwt extension for this. Luckily, its ready to install on our Supabase instance.

create extension if not exists "pgjwt" with schema "extensions";

The create_api_token function inserts an api token into the database, and then signs a jwt token with id of the token and the organisation id in the payload. There are a few notable things here. First, live supabase instances store the jwt_secret in the app.settings.jwt_secret variable. >ou can run show app.settings.jwt_secret; in the Supabase SQL editor for your project to prove this for yourself. The jwt secret for local development with the Supabase CLI has the constant value “super-secret-jwt-token-with-at-least-32-characters-long” as an undocumented “feature”. You might notice that the token is always valid forever for now. We will fix that at a later stage. And we use HS256, because its the same algorithm Supabase auth uses.

create or replace function create_api_token (organisation_id uuid, name text)
returns text as $token$
declare
 token_id uuid;
 jwt_secret text := coalesce(
 nullif(current_setting('app.settings.jwt_secret', true), ''),
 'super-secret-jwt-token-with-at-least-32-characters-long'
 );
begin
 insert into api_token (organisation_id, name) values (organisation_id, name)
 returning id into token_id;
 return extensions.sign(json_build_object(
 'tid', token_id, 'iss', 'supabase', 'sub', organisation_id,
 'role', 'tokenauthed', 'iat', trunc((extract(epoch from Now()))::numeric, 0)
 ),
 jwt_secret::text,
 'HS256'::text
 );
end;
$token$ security invoker language plpgsql;

## Update Row-Level-Security Policies

First, we add a function to extract the tid from the jwt token, similar to auth.uid().

create or replace function private.tid ()
 returns uuid language 'sql' stable
 as $body$
 select coalesce(
 nullif (current_setting('request.jwt.claim.tid', true), ''),
 (nullif (current_setting('request.jwt.claims', true), '')::jsonb ->> 'tid')
 )::uuid
$body$;

We then update private.organisation_id() to return the organisation_id from the jwt payload when the role is tokenauthed.

create or replace function private.organisation_id() returns uuid as
$sql$
 select (
 case
 when auth.role() = 'authenticated'
 then (select organisation_id from employee where user_id = auth.uid())
 when auth.role() = 'tokenauthed'
 then (
 select coalesce(
 nullif (current_setting('request.jwt.claim.sub', true), ''),
 (nullif (current_setting('request.jwt.claims', true), '')::jsonb ->> 'sub')
 )::uuid
 )
 end
);
$sql$ language sql stable security definer;

Finally, we re-create the Row-Level-Security policy on contact to include tokenauthed.

drop policy employee_all on contact;

create policy employee_tokenauthed_all on contact to authenticated, tokenauthed using (
 (select private.organisation_id()) = organisation_id
);

## Create the API Schema

To control what tables and columns should be exposed, we create a separate api schema. This separates our internal data model from the data model exposed in our public api.

create schema api;

grant usage on schema api to postgres, anon, authenticated, service_role, tokenauthed;
alter default privileges in schema api grant all on tables to postgres, anon, authenticated, service_role, tokenauthed;
alter default privileges in schema api grant all on functions to postgres, anon, authenticated, service_role, tokenauthed;
alter default privileges in schema api grant all on sequences to postgres, anon, authenticated, service_role, tokenauthed;

We now use “updateable views” and the security_invoker config to expose the contact. An updatable view in postgres is a view which allows mutations, because the underlying table can be derived. Thanks to the security_invoker config, all operations will respect the Row-Level-Security policy we set on the contact table. For better performance, we always filter on the organisation_id and therefore mimick the security policy clause. With the view, we can also define what columns should be exposed. This is pretty neat, since most apps will have columns that are internal to the app logic.

create view api.contact with (security_invoker) as
select id, full_name
from contact
where organisation_id = private.organisation_id();

And thats it for the database part. As of now, we have:

a custom database role
a function to generate tokens
helper functions to easily define Row-Level-Security Policies
a table that is exposed via a custom api schema

We could just share our Supabase Rest API endpoint and be done. I would not recommend that. Instead, lets build a little proxy that will give us the possibility to add required security measures such as rate-limiting and observability.

# The Proxy

Our little proxy will be a very simple Cloudflare Worker using Hono. You can choose any framework or platform you want for this though. First, we set up the project.

bunx create-hono api-proxy

First, setup a JWT middleware to verify the incoming request. You can find the JWT secret in your Supabase Dashboard.

const app = new Hono<Env>();

app.use((c, next) => {
 const jwtMiddleware = jwt({
 secret: c.env.SUPABASE_JWT_SECRET,
 alg: "HS256",
 });
 return jwtMiddleware(c, next);
});

Now, add a single “catch-all” route that forwards the request to the upstream PostgREST api.

app.all("/:table", async (c) => {
 const searchParams =
 c.req.url.indexOf("?") > -1 ? c.req.url.slice(c.req.url.indexOf("?")) : "";

 const upstreamUrl = `${c.env.SUPABASE_URL}/rest/v1/${c.req.param("table")}${searchParams}`;

 const newRequest = new Request(upstreamUrl, c.req.raw);
 const res = await fetch(newRequest);
 const newResponse = new Response(res.body, res);
 return newResponse;
});

To fail early, you might want to validate the path parameter.

import { object, picklist } from "valibot";
import { vValidator } from "@hono/valibot-validator";

const TABLES = ["contact"];

const paramsSchema = object({
 table: picklist(TABLES),
});

app.all(
 "/:table",
 vValidator<typeof paramsSchema, "param", Env, any>("param", paramsSchema),
 async (c) => {
 // ...
 },
);

For Kong, the API gateway that Supabase uses, to accept your proxied request, we need to set the apiKey header to the anon key. To target the api schema, set the Accept-Profile and Content-Profile headers.

app.all(
 "/:table",
 vValidator<typeof paramsSchema, "param", Env, any>("param", paramsSchema),
 async (c) => {
 // ...
 const newRequest = new Request(upstreamUrl, c.req.raw);
 newRequest.headers.set("apiKey", c.env.SUPABASE_ANON_KEY);
 if (["GET", "HEAD"].includes(c.req.method)) {
 newRequest.headers.set("Accept-Profile", "api");
 } else {
 newRequest.headers.set("Content-Profile", "api");
 }
 // ...
 },
);

This is also a great opportunity to change the default behavior of PostgREST. For example, your users might expect that any mutation always returns the mutated row. This can be enabled by default by always adding return=representation to the Prefer header.

app.all(
 "/:table",
 vValidator<typeof paramsSchema, "param", Env, any>("param", paramsSchema),
 async (c) => {
 // ...
 const newRequest = new Request(upstreamUrl, c.req.raw);
 // ...
 if (c.req.query("select")) {
 newRequest.headers.set(
 "Prefer",
 [c.req.header("Prefer"), "return=representation"]
 .filter(Boolean)
 .join(","),
 );
 }
 // ...
 },
);

And thats it! From here, you should be able to complete this on your own. To be ready for production, you should add rate-limiting as well as logging and monitoring to your worker.

# Recommendations

Control the statement timeout for your tokenauthed role. You are giving your users a lot of power with all the query features that PostgREST offers. To make sure they spend some time optimizing their requests, set the limit to a relatively low value depending on your use case.

alter role tokenauthed set statement_timeout = '2s';

Next, you should make sure that we can invalidate tokens. For now, a token can be invalidated if its reference in the api_token table is deleted. We can leverage the db_pre_request hook that PostgREST exposes for this. To read more about the Pre-Request feature check out their documentation. Its basically a middleware within Postgres which we can use to ensure that the token reference in our database still exists.

create or replace function private.validate_token ()
 returns void
 language 'plpgsql'
 security definer
 as $body$
begin
 -- 1. we should only verify api tokens.
 -- skip check if auth is not done using an api token
 if auth.role () <> 'tokenauthed' then
 return;
 end if;
 -- 2. make sure the token has not been revoked
 if (case when (
 select
 id
 from api_token t
 where
 id = private.tid ()
 ) is not null then false else true end) is true then
 raise sqlstate 'pt401'
 using message = 'unauthorized';
 end if;
end
$body$;

alter role authenticator set pgrst.db_pre_request to 'private.validate_token';
notify pgrst, 'reload config';

Finally, we only want to allow tokenauthed requests if they were routed through our proxy, so that rate-limiting applies. The simplest way to achieve this is to add a custom secret value to the header in the request.

app.all(
 "/:table",
 vValidator<typeof paramsSchema, "param", Env, any>("param", paramsSchema),
 async (c) => {
 // ...
 const newRequest = new Request(upstreamUrl, c.req.raw);
 // ...
 if (c.req.query("select")) {
 newRequest.headers.set("x-proxy-secret", c.env.PROXY_SECRET);
 }
 // ...
 },
);

We can check for the existence of this token within the validate_token function. Luckily, our Supabase project comes with a built-in Vault, so we do not have to worry about storing our secret key in the database.

create or replace function private.validate_token ()
 returns void
 language 'plpgsql'
 security definer
 as $body$
begin
 -- 1. we should only verify api tokens.
 -- skip check if auth is not done using an api token
 if auth.role () <> 'tokenauthed' then
 return;
 end if;

 -- 2. Make sure that the request is going through our proxy
 if (select current_setting('request.headers', true)::json ->> 'x-proxy-secret') <> (select decrypted_secret from vault.decrypted_secrets where name = 'proxy_secret') then
 raise sqlstate 'pt401'
 using message = 'unauthorized';
 end if;

 -- 3. make sure the token has not been revoked
 if (case when (
 select
 id
 from api_token t
 where
 id = private.tid ()
 ) is not null then false else true end) is true then
 raise sqlstate 'pt401'
 using message = 'unauthorized';
 end if;
end
$body$;

And thats it!

You can find the source code and a demo on GitHub.

Announcing Supabase Cache Helpers 1.0

philipp@steinroetter.com (psteinroe) — Tue, 25 Apr 2023 00:00:00 +0000

Supabase Cache Helpers is a collection of framework specific cache utilities for working with Supabase. It bridges the gap between data-fetching libraries such as SWR and React Query, and the Supabase client libraries.

# Motivation

Supabase offers a bunch of SDKs to interact with their services, such as the Postgres database, Realtime and Storage. Depending on your requirements, these might be sufficient to build your application. At some point however, you will want to use data-fetching libraries such as SWR and React Query to reduce boilerplate, cache data, dedupe requests, and much more. When starting to use these libraries, you might find that its quite straightforward to fetch data from Supabase: define a cache key, write a simple fetcher functions and off you go. Here is an example of a simple hook that fetches a note by id using a SWR hook:

// fetcher function to get contact by id
export function getContactById(client: SupabaseClient, contactId: string) {
 return client
 .from('contact')
 .select(`id,username,ticket_number`)
 .eq('id', contactId)
 .throwOnError()
 .single()
}

// wrapper hook
function useContactQuery(contactId: string) {
 const client = useSupabaseClient()
 // cache key for this query
 const key = ['contact', contactId]

 return useSWR(key, async () => {
 return getContactById(client, contactId).then((result) => result.data)
 })
}

export default useContactQuery

Easy enough. Now import the hook into your component and display the contact.

function ContactCard({ contactId }: { contactId: string }) {
 const { data: contact, isLoading, isError } = useContactQuery(contactId)

 if (isLoading) {
 return <div>Loading...</div>
 }

 if (isError) {
 return <div>Error</div>
 }

 return (
 <div>
 <h1>{contact.username}</h1>
 </div>
 )
}

But its a lot of boilerplate, right? And as a developer, you have little transparency on the actual query being executed when looking at the ContactNote component. In our organisation, we found this intransparency to be the source of performance issues and bugs, because people tend to avoid writing a new hook, and just overload that one query with all the columns, filters and joins they need. This becomes worse when the size and complexity of your app grows. Imagine a complex table view, that allows the user to apply a bunch of filters and ordering. For proper caching, you will need to pass filter object into your hook, and encode the query settings into the key. The maintainability will decrease to zero at some point.

The same is true for mutations. It is simple at first. Just define your fetcher, and pass it to the useMutation hook:

// fetcher function to update a note by id
export async function updateContact(
 client: SupabaseClient,
 id: string,
 data: Partial<Contact>
) {
 return client
 .from('contact')
 .update(data)
 .eq('id', id)
 .throwOnError()
 .select('id,username')
 .throwOnError()
 .single()
}

// wrapper hook
function useUpdateContactMutation() {
 const client = useSupabase()

 return useMutation(async (contact: Contact) => {
 return updateContactById(client, note).then((result) => result.data)
 })
}

Easy. When updating the username of the contact, you might want the change to be reflected both in the ContactCard component, as well as in the table view. Oversimplified, such cache update code might look like this:

{
 onSuccess: (updatedContact) => {
 // update query that selects a single contact by id
 mutate(['contact', updatedContact.id], (existingContact) => {
 return {
 ...existingContact,
 ...updatedContact,
 }
 })

 // you might have a contact list query somewhere, too.
 mutate(['contacts', 'filters=has_golden_ticket.is.true'], (data) => {
 if (data) {
 return data.map((contact) => {
 if (updatedContact.id === contact.id) {
 return { ...contact, ...updatedContact }
 }

 return contact
 })
 }
 })
 }
}

This is a very simple example, and does not nearly cover all required edge cases. For example, what if the table query is filtered or ordered on the username of the contact? You will need to write a lot of repetitive code, just to keep your apps cache up-to-date. But what if we could implement a generalizable solution?

With Supabase Cache Helpers, you can archieve all of the above and more with just a single line of code.

# Encoding Queries

The useContactQuery hook can be simplified to:

const { data, error } = useQuery(
 client
 .from('contact')
 .select(`id,username,ticket_number`)
 .eq('id', contactId)
 .single()
)

Thats it. Supabase Cache Helpers will encode the client object into a definite cache key, and pass it to a generic fetcher function that makes the request. The key itself will contain all important information about the query:

schema and table name
selected columns
applied filters
body of the request (for encoding .rpc() queries)
count settings
whether it is a head query
what ordering is applied

For the (simple) query above, the query key in SWR is:

;[
 'postgrest', // key prefix
 'null', // for swr, this is set to a constant if its an infinite query
 'public', // schema
 'contact', // table
 'select=id%2Cusername%2Cticket_number&username=eq.psteinroe', // sorted search params including selected columns and filters
 'null', // body (for encoding `.rpc()` queries)
 'count=exact', // count settings
 'head=false', // whether it is a head query
 '', // no ordering
]

This key contains all information of the query it belongs to, and will help us to automatically populate the cache after mutations, or when subscription data is arriving. All information is extracted from the PostgrestClient instance. Internally, the client simply sets a few protected properties, and builds an URL object. Luckily, only TypeScript knows what a protected property is, so we can access them via client['url'] and extract the query details. For pagination and infinite scroll query keys, the encoding works the same.

# Mutating Data

Similar to queries, we can replace the mutation fetcher and hook with a one-liner:

const { trigger } = useUpsertMutation(
 client.from('contact'),
 ['id'], // primary keys of `contact`
 'id,username' // will be passed to .select(`id,username`)
)

To auto-populate the data the same way the custom code above does, knowledge about the required paths, and applied filters of the query is leveraged to ensure that the mutated data is still a valid member of the query. For example, a contact should only be inserted into the cache of the following query, if its ticket number is less than 100, and if it defines values for id and username.

const { data, error } = useQuery(
 client.from('contact').select(`id,username`).lt('ticket_number', 100)
)

Note that the exemplary upsert mutation above only selects id and username, and there is no way to know whether it would be a valid member of the query that filters on ticket_number. To ensure that every mutation returns all data required to make relevant updates to the cache, cache helpers, by default, auto-expands the query. Before the request is being made, all cache keys are scanned for queries on the same table, and the .select() statement is expanded to include all paths that any query selects or filters on. Note that this works for any valid PostgREST query, no matter the complexity. For the example above, the actual select statement of the request will be id,username,ticket_number. Since this can have a performance impact, you can opt-out of this feature on a per-mutation level.

After the mutation request returns the data, the entire query cache is searched for keys that target the same schema and table as the mutation itself. For each key, the input is checked against a set of conditions that depend on the type of operation to decide whether to mutate the cache for that key. During mutation, all relevant information about the query is observed, including .limit() filters and .order() modifiers. If you have a paginated table with contacts ordered by username, the new contact will be inserted in the same position as it is when the query is fetched from the server.

The fine-grained cache update is enabled by PostgrestFilter, which takes the parsed query key and creates filter functions to evaluate whether an object is valid given the query filters (applyFilters) and select statements (hasPaths). For the above example, hasPaths returns true if the input object returned by the mutation defines values for id, username, and ticket_number. For applyFilters to return true, the input object must define a ticket_number which is less than 100. If the input has no value for the path that the query filters for, false is returned. Note that both methods work with queries of any complexity:

const filter = PostgrestFilter.fromFilterBuilder(
 supabase
 .from('contact')
 .select(
 'id,username,ticket_number,golden_ticket,tags,country!inner(code,name,full_name)'
 )
 .or(`username.eq.unknown,and(ticket_number.eq.2,golden_ticket.is.true)`)
 .is('golden_ticket', true)
 .in('username', ['thorwebdev'])
 .contains('tags', ['supateam'])
 .or('name.eq.unknown,and(name.eq.Singapore,code.eq.SG)', {
 foreignTable: 'country',
 })
)
console.log(
 filter.apply({
 id: '68d2e5ef-d117-4f0c-abc7-60891a643571',
 username: 'thorwebdev',
 ticket_number: 2,
 golden_ticket: false,
 tags: ['supateam', 'investor'],
 country: {
 code: 'SG',
 name: 'Singapore',
 full_name: 'Republic of Singapore',
 },
 })
) // --> false
console.log(
 filter.apply({
 id: '68d2e5ef-d117-4f0c-abc7-60891a643571',
 created_at: '2022-08-19T15:30:33.072441+00:00',
 username: 'thorwebdev',
 ticket_number: 2,
 golden_ticket: true,
 tags: ['supateam', 'investor'],
 country: {
 code: 'SG',
 name: 'Singapore',
 full_name: 'Republic of Singapore',
 },
 })
) // --> true

Internally, PostgrestFilter implements all operators supported by PostgREST in JavaScript. Undoubtedly, it will not be able to perfectly replicate the behavior of Postgres. However, it seems to be good enough for this use case and we have been using the Cache Helpers successfully in production for months now.

# What’s next

At the moment, only SWR and React Query are supported. However, all the logic is split into framework-independent TypeScript packages, and I’d love to add adapters for other libraries and frameworks. If you are interested in collaborating, please file an issue or contact me directly!

Set up a monorepo with Supabase and Turborepo

philipp@steinroetter.com (psteinroe) — Tue, 06 Dec 2022 00:00:00 +0000

At my current company, we recently migrated our codebase to a turborepo-powered pnpm monorepo, and are now shipping three NextJS apps, a Preact widget, a React Native app and two Fastify servers out of a single repository, all speaking to the same Supabase instance. A monorepo setup not only allows us to share code and configurations, but also decreases ci time and thereby saves $$. It took me a few attempts to properly integrate Supabase with Turborepo, and here is a quick rundown of what I’ve learned setting up the monorepo.

tl;dr; check out the supasample repository if you prefer to read code.

This guide assumes you have basic knowledge about Turborepo and setup a project using pnpm and turborepo already. If that is not the case, I highly recommend skimming through the turborepo documentation. It is one the best I have ever read.

# Declare Supabase as a Workspace

Workspaces are the building blocks of every monorepo, and are managed by the package manager. To properly integrate Supabase into the monorepo, the supabase/ directory has to be declared as a workspace itself. First, create supabase/package.json and give the workspace a name. I like to prefix all local workspaces with the organisation name, but you can choose any name you want.

{
 "name": "@supasample/supabase"
}

Next, to make pnpm aware of the @supasample/supabase workspace, add its path to the pnpm-workspace.yaml file.

packages:
 - 'apps/*'
 - 'packages/*'
 - 'supabase'

# Add Supabase CLI Commands to the Supabase Workspace

To integrate Supabase in the default development workflow, you need to add the common CLI commands to supabase/package.json, namely start, db reset, db test, db lint and deploy.

{
 "name": "@supasample/supabase",
 "scripts": {
 "start": "supabase status || supabase start",
 "reset": "supabase db reset || supabase start",
 "test": "supabase db test",
 "lint": "supabase db lint",
 "deploy": "supabase link --project-ref $SUPABASE_PROJECT_REF && supabase db push"
 }
}

Note that both start and reset use a try-catch mechanism to make sure that the desired result is achieved, no matter in what state your local Supabase instance currently is.

Next, all scripts declared within the Supabase workspace have to be configured in turbo.json.

{
 "$schema": "https://turborepo.org/schema.json",
 "pipeline": {
 // ...
 "test": {
 "dependsOn": ["^build"],
 "outputs": ["coverage/**"]
 },
 "start": {
 "outputs": []
 },
 "lint": {
 "dependsOn": ["format:check"],
 "outputs": []
 },
 "deploy": {
 "outputs": []
 }
 // ...
 },
 "globalEnv": [
 "NODE_ENV",
 "SUPABASE_URL",
 "SUPABASE_ANON_KEY",
 "SUPABASE_SERVICE_KEY"
 ]
}

Note that the snippet contains only the few configurations necessary for the supabase workspace, you will most likely have a few more in yours.

To test that everything works, run pnpm run start --filter=supabase from the repository root. Make sure that your local Docker host is up.

# Configure workspace-task to workspace-task dependencies

Withe the current configuration, you have to run pnpm run start --filter=supabase or supabase start expliclity before executing any task that requires a running Supabase instance. A GitHub Action workflow might look like this:

 ...
 - name: 🏗 Setup Supabase
 uses: supabase/setup-cli@v1
 with:
 version: latest

 - name: 🚀 Start Supabase
 run: supabase db start

 - name: 🦺 Check Format, Lint, Typecheck and Test
 run: pnpm run turbo format:check lint:report typecheck test
 ...

Although the team is making continous improvements, starting Supabase takes around 60-90 seconds of your costly GitHub Actions time. In most projects, your database will change the least, and the test and lint tasks will be replayed from the Turborepo cache 90% of the time. Hence, you are throwing away money here. Fortunately, Turborepo supports workspace-task to workspace-task dependencies, that can be expressed using the <workspace>#<task> syntax. You can leverage this feature to have Turobrepo decide whether it is required to start Supabase.

{
 "pipeline": {
 // ...
 "@supasample/supabase#start": {
 "outputs": [],
 "cache": false
 },
 "@supasample/supabase#test": {
 "dependsOn": ["@supasample/supabase#start", "^build"],
 "outputs": []
 },
 "@supasample/supabase#lint": {
 "dependsOn": ["@supasample/supabase#start", "format:check"],
 "outputs": []
 }
 // ...
 }
}

Although it might appear counter intuitive, you want to disable caching for supabase start. Next, to ensure that the supabase instance is running before the database is tested or linted, you the respective task configurations, and add the start script as a dependency.

If you now run pnpm run test --filter=supabase, the supabase instance will be started if not already up, and your GitHub Action can be simplified to

 ...
 - name: 🏗 Setup Supabase
 uses: supabase/setup-cli@v1
 with:
 version: latest

 - name: 🦺 Check Format, Lint, Typecheck and Test
 run: pnpm run turbo format:check lint:report typecheck test
 ...

# Add `types` Package

The same configuration can be used for other workspaces too. For example, you might have a shared types package in your monorepo, from which all packages and apps import the database types generated by the Supabase CLI.

First, add a generate script to the respective package.json.

{
 "scripts": {
 // ...
 "generate": "(cd ../../ && supabase gen types typescript --local > ./packages/types/src/database.ts) && prettier --write \"src/**/*.{ts,tsx,md}\""
 // ...
 }
}

The script first changes into the root directory before running the type generation. All types are written into a database.ts file of the same package. Finally, the generated types are formatted using prettier.

You might have noticed that the --local flag is used. To ensure you have a Supabase instance running, configure the generate task similar to the test and lint tasks within the turbo.json file:

{
 "pipeline": {
 // ...
 "generate": {
 "dependsOn": ["^generate"],
 "outputs": []
 },
 "@supasample/types#generate": {
 "dependsOn": ["@supasample/supabase#start", "^generate"],
 "outputs": []
 }
 // ...
 }
}

If you now run pnpm run generate, and Turborepo will make sure you have a Supabase instance running before the types are generated.

Thanks for reading!