PostgreSQL Mage
Ruohang Feng @Vonng / Pigsty
PostgreSQL ecosystem news, plus development, operations, internals and tuning.
AI Large Models and Vector Database PGVector
This article focuses on vector databases hyped by AI, introduces the basic principles of AI embeddings and vector storage/retrieval, and demonstrates the functionality, performance, acquisition, and …
How Powerful is PostgreSQL Really?
Let performance data speak: Why PostgreSQL is the world's most advanced open-source relational database, aka the world's most successful database. MySQL vs PostgreSQL performance showdown and …
Why PostgreSQL is the Most Successful Database?
Database users are developers, but what about developers' preferences, likes, and choices? Looking at StackOverflow survey results over the past six years, it's clear that in 2022, PostgreSQL has won …
Ready-to-Use PostgreSQL Distribution: Pigsty
Yesterday I gave a live presentation in the PostgreSQL Chinese community, introducing the open-source PostgreSQL full-stack solution — Pigsty
Why Does PostgreSQL Have a Bright Future?
Databases are the core component of information systems, relational databases are the absolute backbone of databases, and PostgreSQL is the world's most advanced open source relational database. With …
Localization and Collation Rules in PostgreSQL
What? Don't know what COLLATION is? Remember one thing: using C COLLATE is always the right choice!
Implementing Advanced Fuzzy Search
How to implement relatively complex fuzzy search logic in PostgreSQL?
PostgreSQL Logical Replication Deep Dive
This article introduces the principles and best practices of logical replication in PostgreSQL 13.
PG Replica Identity Explained
Replica identity is important - it determines the success or failure of logical replication
A Methodology for Diagnosing PostgreSQL Slow Queries
Slow queries are the sworn enemy of OLTP databases. Here’s how to identify, analyze, and fix them using metrics (Pigsty dashboards), pg_stat_statements, and logs.
Incident-Report: Patroni Failure Due to Time Travel
Machine restarted due to failure, NTP service corrected PG time after PG startup, causing Patroni to fail to start.
Online Primary Key Column Type Change
How to change column types online, such as upgrading from INT to BIGINT?
Golden Monitoring Metrics: Errors, Latency, Throughput, Saturation
Understanding the golden monitoring metrics in PostgreSQL
Database Cluster Management Concepts and Entity Naming Conventions
Concepts and their naming are very important. Naming style reflects an engineer's understanding of system architecture. Poorly defined concepts lead to communication confusion, while carelessly set …
PostgreSQL's KPI
Managing databases is similar to managing people - both need KPIs (Key Performance Indicators). So what are database KPIs? This article introduces a way to measure PostgreSQL load: using a single …
Online PostgreSQL Column Type Migration
How to modify PostgreSQL column types online? A general approach
Transaction Isolation Level Considerations
PostgreSQL actually has only two transaction isolation levels: Read Committed and Serializable.
Frontend-Backend Communication Wire Protocol
Understanding the TCP protocol used for communication between PostgreSQL server and client, and printing messages using Go
Incident: PostgreSQL Extension Installation Causes Connection Failure
Today encountered an interesting case where a customer reported database connection issues caused by extensions.
CDC Change Data Capture Mechanisms
Change Data Capture is an interesting ETL alternative solution.
Locks in PostgreSQL
Snapshot isolation does most of the heavy lifting in PG, but locks still matter. Here’s a practical guide to table locks, row locks, intention locks, and pg_locks.
O(n2) Complexity of GIN Search
When GIN indexes are used to search with very long keyword lists, performance degrades significantly. This article explains why GIN index keyword search has O(n^2) time complexity.
PostgreSQL Common Replication Topology Plans
Replication is one of the core issues in system architecture.
Warm Standby: Using pg_receivewal
There are various backup strategies. Physical backups can usually be divided into four types.
AI Large Models and Vector Database PGVector
New AI applications have shown exponential growth over the past year, and these applications face a common challenge: how to store and query AI embeddings represented as vectors at scale. This article focuses on vector databases hyped by AI, …
New AI applications have shown exponential growth over the past year, and these applications face a common challenge: how to store and query AI embeddings represented as vectors at scale. This article focuses on vector databases hyped by AI, …
How Powerful is PostgreSQL Really?
Previously, we analyzed StackOverflow survey data to explain “Why PostgreSQL is the Most Successful Database”. This time we’ll let performance data speak for itself, discussing just how powerful the most successful PostgreSQL really is, helping you …
Previously, we analyzed StackOverflow survey data to explain “Why PostgreSQL is the Most Successful Database”. This time we’ll let performance data speak for itself, discussing just how powerful the most successful PostgreSQL really is, helping you …
Why PostgreSQL is the Most Successful Database?
When we say a database is “successful,” what exactly do we mean? Is it about features, performance, usability, or cost, ecosystem, complexity? There are many metrics, but ultimately users decide. Database users are developers, so what about …
When we say a database is “successful,” what exactly do we mean? Is it about features, performance, usability, or cost, ecosystem, complexity? There are many metrics, but ultimately users decide. Database users are developers, so what about …
Ready-to-Use PostgreSQL Distribution: Pigsty
What is Pigsty Pigsty is a ready-to-use production-grade open-source PostgreSQL distribution. A distribution refers to a complete database solution consisting of a database kernel and its suite of software packages. For example, Linux is an operating …
What is Pigsty Pigsty is a ready-to-use production-grade open-source PostgreSQL distribution. A distribution refers to a complete database solution consisting of a database kernel and its suite of software packages. For example, Linux is an operating …
Why Does PostgreSQL Have a Bright Future?
Recently, everything I’ve been working on revolves around the PostgreSQL ecosystem, because I’ve always felt this is a direction with unlimited potential. Why do I say this? Because databases are the core component of information systems, relational …
Recently, everything I’ve been working on revolves around the PostgreSQL ecosystem, because I’ve always felt this is a direction with unlimited potential. Why do I say this? Because databases are the core component of information systems, relational …
Localization and Collation Rules in PostgreSQL
Why does Pigsty default to locale=C and encoding=UTF8 when initializing PostgreSQL databases? The answer is simple: Unless you explicitly know you need LOCALE-related functionality, you should never configure anything other than C.UTF8 for character …
Why does Pigsty default to locale=C and encoding=UTF8 when initializing PostgreSQL databases? The answer is simple: Unless you explicitly know you need LOCALE-related functionality, you should never configure anything other than C.UTF8 for character …
Implementing Advanced Fuzzy Search
In daily development, we often encounter requirements for fuzzy search. Today, let’s briefly discuss how to implement some advanced fuzzy search using PostgreSQL. Of course, the fuzzy search I’m talking about here isn’t the old-fashioned LIKE …
In daily development, we often encounter requirements for fuzzy search. Today, let’s briefly discuss how to implement some advanced fuzzy search using PostgreSQL. Of course, the fuzzy search I’m talking about here isn’t the old-fashioned LIKE …
PostgreSQL Logical Replication Deep Dive
Logical Replication Logical Replication is a method of replicating data objects and their changes based on their replica identity (usually the primary key). The term logical replication contrasts with physical replication. Physical replication uses …
Logical Replication Logical Replication is a method of replicating data objects and their changes based on their replica identity (usually the primary key). The term logical replication contrasts with physical replication. Physical replication uses …
PG Replica Identity Explained
Introduction: DIY Logical Replication The concept of replica identity serves logical replication. The basic working principle of logical replication is to decode row-level INSERT/UPDATE/DELETE events from logical publication-related tables and …
Introduction: DIY Logical Replication The concept of replica identity serves logical replication. The basic working principle of logical replication is to decode row-level INSERT/UPDATE/DELETE events from logical publication-related tables and …
A Methodology for Diagnosing PostgreSQL Slow Queries
“You can’t optimize what you can’t measure.” Slow queries hog connections, hold locks, block replication, trigger deadlocks, and waste resources. Every DBA must know how to find and fix them quickly. Traditional tools pg_stat_statements – essential …
“You can’t optimize what you can’t measure.” Slow queries hog connections, hold locks, block replication, trigger deadlocks, and waste resources. Every DBA must know how to find and fix them quickly. Traditional tools pg_stat_statements – essential …
Incident-Report: Patroni Failure Due to Time Travel
Summary: Machine restarted due to failure, NTP service corrected PG time after PG startup, causing Patroni to fail to start. The failure information in Patroni is shown as follows: Process %s is not postmaster, too much difference between PID file …
Summary: Machine restarted due to failure, NTP service corrected PG time after PG startup, causing Patroni to fail to start. The failure information in Patroni is shown as follows: Process %s is not postmaster, too much difference between PID file …
Online Primary Key Column Type Change
Author: Vonng (@Vonng) How to change primary key column types online, such as upgrading from INT to BIGINT, without affecting business operations? Suppose you have a table in PostgreSQL where you initially chose an INT primary key without much …
Author: Vonng (@Vonng) How to change primary key column types online, such as upgrading from INT to BIGINT, without affecting business operations? Suppose you have a table in PostgreSQL where you initially chose an INT primary key without much …
Golden Monitoring Metrics: Errors, Latency, Throughput, Saturation
Preface Playing with databases and playing with cars have something in common - they both require frequently checking the dashboard. What are you doing staring at the dashboard? Looking at metrics. Why look at metrics? You need to understand the …
Preface Playing with databases and playing with cars have something in common - they both require frequently checking the dashboard. What are you doing staring at the dashboard? Looking at metrics. Why look at metrics? You need to understand the …
Database Cluster Management Concepts and Entity Naming Conventions
Author: Vonng (@Vonng) “Once named, it can be spoken; once spoken, it can be acted upon.” Concepts and their naming are very important. Naming style reflects an engineer’s understanding of system architecture. Poorly defined concepts lead to …
Author: Vonng (@Vonng) “Once named, it can be spoken; once spoken, it can be acted upon.” Concepts and their naming are very important. Naming style reflects an engineer’s understanding of system architecture. Poorly defined concepts lead to …
PostgreSQL's KPI
Managing databases is similar to managing people - both need KPIs (Key Performance Indicators). So what are database KPIs? This article introduces a way to measure PostgreSQL load: using a single horizontally comparable metric that is basically …
Managing databases is similar to managing people - both need KPIs (Key Performance Indicators). So what are database KPIs? This article introduces a way to measure PostgreSQL load: using a single horizontally comparable metric that is basically …
Online PostgreSQL Column Type Migration
Scenario In the lifecycle of a database, there’s a common type of requirement: modifying column types. For example: Using INT as a primary key, only to discover that business is booming and the 2.1 billion limit of INT32 isn’t enough, wanting to …
Scenario In the lifecycle of a database, there’s a common type of requirement: modifying column types. For example: Using INT as a primary key, only to discover that business is booming and the 2.1 billion limit of INT32 isn’t enough, wanting to …
Transaction Isolation Level Considerations
PostgreSQL actually has only two transaction isolation levels: Read Committed and Serializable Basics The SQL standard defines four isolation levels, but PostgreSQL actually has only two transaction isolation levels: Read Committed and Serializable …
PostgreSQL actually has only two transaction isolation levels: Read Committed and Serializable Basics The SQL standard defines four isolation levels, but PostgreSQL actually has only two transaction isolation levels: Read Committed and Serializable …
Frontend-Backend Communication Wire Protocol
Understanding the TCP protocol used for communication between PostgreSQL server and client Startup Phase The basic flow of the startup phase is as follows: Client sends a StartupMessage (F) to initiate connection request to server Payload includes …
Understanding the TCP protocol used for communication between PostgreSQL server and client Startup Phase The basic flow of the startup phase is as follows: Client sends a StartupMessage (F) to initiate connection request to server Payload includes …
Incident: PostgreSQL Extension Installation Causes Connection Failure
Author: Vonng (@Vonng) Today encountered an interesting case where a customer reported database connection issues. The error was: psql: FATAL: could not load library "/export/servers/pgsql/lib/pg_hint_plan.so": …
Author: Vonng (@Vonng) Today encountered an interesting case where a customer reported database connection issues. The error was: psql: FATAL: could not load library "/export/servers/pgsql/lib/pg_hint_plan.so": …
CDC Change Data Capture Mechanisms
In actual production, we often need to synchronize database states to other places, such as synchronizing to data warehouses for analysis, to message queues for downstream consumption, or to caches to accelerate queries. Generally speaking, there are …
In actual production, we often need to synchronize database states to other places, such as synchronizing to data warehouses for analysis, to message queues for downstream consumption, or to caches to accelerate queries. Generally speaking, there are …
Locks in PostgreSQL
PostgreSQL relies on snapshot isolation (SI) for concurrency and two-phase locking (2PL) as a supporting act. DML (SELECT/INSERT/UPDATE/DELETE) uses SSI; DDL (CREATE TABLE etc.) still uses 2PL. Understanding locks is essential when diagnosing …
PostgreSQL relies on snapshot isolation (SI) for concurrency and two-phase locking (2PL) as a supporting act. DML (SELECT/INSERT/UPDATE/DELETE) uses SSI; DDL (CREATE TABLE etc.) still uses 2PL. Understanding locks is essential when diagnosing …
O(n2) Complexity of GIN Search
When GIN indexes are used to search with very long keyword lists, performance degrades significantly. This article explains why GIN index keyword search has O(n^2) time complexity. Here is the detail of why that query have O(N^2) inside GIN …
When GIN indexes are used to search with very long keyword lists, performance degrades significantly. This article explains why GIN index keyword search has O(n^2) time complexity. Here is the detail of why that query have O(N^2) inside GIN …
PostgreSQL Common Replication Topology Plans
Replication is one of the core issues in system architecture. Cluster Topology Suppose we use a standard 4-unit configuration: primary, synchronous replica, delayed backup, and remote replica, identified by letters M, S, O, R respectively. M: Master, …
Replication is one of the core issues in system architecture. Cluster Topology Suppose we use a standard 4-unit configuration: primary, synchronous replica, delayed backup, and remote replica, identified by letters M, S, O, R respectively. M: Master, …
Warm Standby: Using pg_receivewal
Author: Vonng (@Vonng) Backup is the foundation of a DBA’s livelihood and one of the most critical tasks in database management. There are various types of backups, but the backups discussed here are all physical backups. Physical backups can …
Author: Vonng (@Vonng) Backup is the foundation of a DBA’s livelihood and one of the most critical tasks in database management. There are various types of backups, but the backups discussed here are all physical backups. Physical backups can …